DEV Community: btkcodedev

I Just Wanted a WhatsApp Channel That Teaches Me One German Word a Day. It Took a Week and a Git History Rewrite.

btkcodedev — Sun, 16 Aug 2026 04:42:09 +0000

Here's the thing nobody tells you about WhatsApp automation. The hard part was never the AI, and it was never the German. It was WhatsApp itself.

I wanted something dumb simple: a WhatsApp Channel that posts one German word every morning, A1 level, with an example sentence and a mnemonic. Gemini can write that in a second. The actual project turned into an argument with WhatsApp's session model, two hosting platforms, and, at one point, my own repo's git history.

The 43MB problem

whatsapp-web.js is the library most people reach for first. It drives a real headless Chrome instance that logs into web.whatsapp.com like your browser would, and it works, locally. The moment you try to deploy it, you hit the same wall everyone hits: the session it needs to stay logged in is a full Chromium profile. Cookies, IndexedDB, cache, the works. Mine was 43MB.

GitHub Secrets cap out in the kilobytes. So the "obvious" fix, the one a dozen tutorials suggest, is: tar it, base64-encode it, shove it into a secret anyway, decode it back into place on every CI run. I actually did this. I had a script called setup.ts whose entire job was printing a giant base64 blob for me to paste into GitHub's secret editor by hand.

It's exactly as bad as it sounds, and it doesn't even solve the real problem, because that session expires. Not on a schedule you control. Whenever WhatsApp feels like it. Phone offline too long, and it's gone. Which means eventually you're back to scanning a QR code, except now you're doing it on a server that has no screen.

Where do you even scan the QR on a headless box?

This is the question I actually came in asking, because I genuinely didn't know. Turns out there are two real answers.

Run the linking step once, locally, pointed at the same database your deployed job uses. Scan it on your own laptop, and the session persists wherever you told it to.
Or just watch your CI provider's live log stream. The QR prints as ASCII art, and you can scan a phone camera off a terminal on a screen, no problem.

Neither needs a screen physically attached to the server. I'd been overcomplicating this in my head for no reason.

Railway said no, and I didn't know why yet

I had this deployed in three places at once: Railway, Render, GitHub, and none of them worked, and I couldn't figure out why. Turns out Railway's fair-use policy explicitly bans "userbots," anything that logs into a personal account via an unofficial, reverse-engineered protocol instead of a real bot API. whatsapp-web.js is precisely that. It's not a bug I could fix. It's a platform I had to stop using for this.

Render fared a little better, but I kept hitting a different, more annoying issue. RemoteAuth with a Mongo-backed session store has open, unresolved GitHub issues where the session saves fine but doesn't reliably restore, silently forcing a fresh QR scan on restart. Not something I broke. Just a library with rough edges.

Baileys, and why it's what most of these channels actually run on

The fix that actually stuck was switching libraries entirely, to Baileys. It speaks WhatsApp's multi-device protocol directly over a WebSocket. No Chromium, no Puppeteer, nothing to render. The session is a handful of small JSON credential files, kilobytes, not tens of megabytes.

That one change collapsed two separate problems at once. The session fits anywhere now (I put it in a free MongoDB Atlas cluster instead of a secret), and without a browser to spin up, the whole thing runs fine on a constrained free-tier box instead of choking on headless-Chrome memory pressure.

It also made the hosting question moot. My actual requirement was "post one message a day," which doesn't need an always-on server at all. It needs about sixty seconds of compute, once every 24 hours. That's a GitHub Actions cron job, for free. I deleted the Dockerfile, deleted Render, deleted Railway, and the whole deployment surface became one YAML file with a cron: line in it.

The security scare I almost shipped

Somewhere in the earlier, messier phase, I had committed a full .wwebjs_auth session folder straight into git: 204 files, plus a couple of standalone tar/base64 dumps of the same thing, at 27MB and 41MB. That's not hypothetical risk. That folder is an active login session. Anyone who could read that repo could have hijacked my WhatsApp account with it.

Deleting the files from the working tree isn't enough. They're still sitting in every earlier commit, recoverable by anyone who clones the repo. Fixing it for real meant git filter-repo to strip those paths out of the entire history, followed by a force-push. My .git folder went from 84MB to 284KB in about a second. If you've ever committed something you regret, this is the actual fix. Not git rm. The history itself has to go.

I only caught this because I asked for a full audit before making the repo public. Worth doing that even if you're sure you didn't leak anything. I was sure. I was wrong.

The model chain

Gemini deprecates old models too quickly, and the solution isn't "use the newest model." It's "never hardcode one." I wrote a small fallback chain: try the latest Flash model, and on a 404, 429, or 503, fall through to the next one down the list automatically. I know it works because I watched it happen live. The newest model hit a genuine 503 Service Unavailable mid-run, fell through to the one behind it, and posted anyway without me touching anything.

What's actually running now

One GitHub Actions workflow, triggered on a cron schedule. It pulls a WhatsApp session out of a free MongoDB cluster, asks Gemini for a word it hasn't used yet, posts it to the channel, and writes the updated state back to Mongo. No server. No Docker image. No monthly bill. The entire compute cost is about a minute of free CI time a day.

The message itself ended up plainer than I expected too. I started with a version stuffed with emoji and flag icons and a line advertising DM commands, and once I actually deleted the interactive bot half of the project, that line was just false advertising to a channel with nobody listening on the other end. It reads like a dictionary entry now: word, pronunciation, meaning, an example sentence, a note. That turned out to be the right call anyway. It's a vocabulary channel, not a chat app.

If you're building anything that automates a personal WhatsApp account, expect the session/auth layer to be the actual project, and the thing you originally wanted to build to be the easy 10%.

Developer Activity and Collaboration Analysis with Airbyte Quickstarts ft. Dagster, BigQuery, Google Colab, dbt and Terraform

btkcodedev — Thu, 11 Jul 2024 06:33:19 +0000

Airbyte could be used as a wonderful tool in-order to leverage the power of data with useful transformations

Those transformed data could be further used for training AI models (Examples at the end)

In this tutorial, GitHub source API is used as source and transformed with trends in developer activity, which could be feed for an AI modal for image training purposes for enhancing its prediction capabilities

I've made a full code walk-through at colab reference
You could either download it as ipynb and run with local jupyter or run step-by-step with local cmd
(If hyperlink is broken, try: https://colab.research.google.com/drive/14U7NYK4dy5fBN3891Tbkl3SJYEqxkMYr?usp=sharing)

The initial part of setting up Airbyte for pulling data from GitHub source to BigQuery and SQL transformations are already given precisely at quickstarts directory

Architecture

Explanation:
Tech stacks: Dagster, dbt, Airbyte, GitHub API, BigQuery, Terraform

We are intended to pull data from GitHub source via Airbyte User Interface towards BigQuery dataset, The Airbyte User Interface is automated via Terraform Provider.

After the dataset creation, data build tool (dbt) is used for transforming data with SQL queries for various metric findings viz. average time per PR, mean of total commits etc...

Part 1.1: Setting Up the Data Pipeline

Screenshots are attached to the colab notebook,

Airbyte and GitHub API:

The GitHub source connector requires three credentials:

Repository name
GitHub personal access token
Workspace ID

Airbyte and BigQuery:

The BigQuery destination connector requires three credentials:

IAM & Admin service account JSON key
Google cloud project ID
BigQuery Dataset ID

Ref: Configuration steps

In case you are wondering about behind the scenes, refer to GitHub, where you could see the sync between GitHub and BigQuery

After which the Terraform jobs are finished, Airbyte UI would be set ready with all the config which are provided and streams are ready to be pulled

You could find the reference of number of streams at GitHub

After running terraform apply, the Airbyte UI is configured with all the setup and the streams are ready to be pulled

Part 1.2: Transformations with dbt

Setup the environment variables used for dbt setup, Currently three

Either setup those env variables or hard code in the files for next steps.
Run dbt debug for confirming the setup.

The schema for table population for each stream could be seen at GitHub

After running dbt run --full-refresh, You could find the transformed tables populated in the BigQuery dataset

The dbt marts are very useful where insights are extracted from the pulled data and could be further utilized for AI training purposes (*Provided large dataset)

Part 1.3: Orchestration using Dagster

Dagster and BigQuery:

Dagster is a modern data orchestrator designed to help you build, test, and monitor your data workflows.

After running dagster dev, localport would be opened or dagster where the workflow could be seen and syncs could be monitored

Part 1.4: Future Reference AI Model Creation in Colab ft. Tensorflow

Export BigQuery data to Colab:

Use the BigQuery connector in Colab to load the desired data from your analysis tables.
Preprocess the data by cleaning, filtering, and transforming it for your specific model inputs.
Build a Tensorflow Model for Team Dynamics and Productivity:

Choose a suitable architecture like LSTM or RNN for time series analysis of developer activity, or use scikit-learn for quantitative analysis.
Train the model on historical data, using features like time to merge PRs, commits per day, code review frequency, etc.
Evaluate the model performance on validation data.

Specific code example is provided at Colab

Another example of Tensorflow Model:

import tensorflow as tf
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

# Load data from BigQuery
client = bigquery.Client()
query = """
SELECT
  author.email,
  author.time_sec AS time_sec,
  committer.email,
  committer.time_sec AS time_sec_1,
  committer.time_sec - author.time_sec AS time_difference
FROM
  `micro-company-task-367016.transformed_data.stg_commits`,
  UNNEST(difference) AS difference
WHERE
  TIMESTAMP_SECONDS(author.time_sec) BETWEEN TIMESTAMP("2023-01-01") AND TIMESTAMP("2023-12-31")
LIMIT 1000
"""
data = client.query(query).to_dataframe()

# Preprocess data
features = ['time_sec', 'time_sec_1']
target = 'time_difference'

# Drop rows with missing values
data = data.dropna(subset=features + [target])

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(data[features], data[target], test_size=0.2, random_state=42)

# Standardize features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Define TensorFlow model architecture
model = Sequential([
    Dense(32, activation='relu', input_shape=(len(features),)),
    Dense(16, activation='relu'),
    Dense(1)  # Output layer, no activation for regression
])

# Compile the model
model.compile(optimizer='adam', loss='mean_squared_error')

# Convert y_train to a NumPy array with a compatible dtype
y_train_np = y_train.values.astype('float32')

scaler_y = StandardScaler()
y_train_scaled = scaler_y.fit_transform(y_train_np.reshape(-1, 1))
y_test_scaled = scaler_y.transform(y_test.values.reshape(-1, 1))

# Train the model
model.fit(X_train_scaled, y_train_scaled, epochs=50, batch_size=32, validation_data=(X_test_scaled, y_test_scaled))


# Convert y_test to a NumPy array with a compatible dtype
y_test_np = y_test.values.astype('float32')

# Evaluate the model
mse = model.evaluate(X_test_scaled, y_test_np)
print(f'Mean Squared Error on Test Data: {mse}')

# Generate synthetic new data
new_data = pd.DataFrame({
    'time_sec': np.random.rand(10) * 1000,  # Adjust the range as needed
    'time_sec_1': np.random.rand(10) * 1000  # Adjust the range as needed
})

# Preprocess new data
new_data_scaled = scaler.transform(new_data[features])

# Make predictions on new data
predictions = model.predict(new_data_scaled)
predictions_inverse = scaler_y.inverse_transform(predictions)

# Display predictions 
print(predictions_inverse)


# Plot model predictions
plt.plot(predictions, label="Predicted Time Difference")
plt.plot(y_test.values, label="Actual Time Difference")
plt.xlabel("Data Point")
plt.ylabel("Time Difference")
plt.title("Predicted vs. Actual Time Difference")
plt.legend()
plt.show()

Results of model:

Other use cases

To predict future trends in team dynamics and productivity.

Identify factors that influence collaboration and individual developer performance.

Visualise the results using charts, graphs, and network visualisations.

Final thoughts

350+ source connectors and huge data warehouse destinations are definitely a plus at Airbyte. We could use this power of transformed data as training data for many AI modals specifically tailored for prediction especially in developer behavioural analysis , stock prediction etc...

Links:

Colab Notebook: https://colab.research.google.com/drive/14U7NYK4dy5fBN3891Tbkl3SJYEqxkMYr?usp=sharing
Airbyte Quickstarts Repository: https://github.com/airbytehq/quickstarts
Airbyte Main Repository: https://github.com/airbytehq/airbyte