DEV Community: D_SECURITY

Building a zero-infra job queue with SQLite (and stress-testing it to 14 jobs/sec)

D_SECURITY — Sun, 24 May 2026 17:56:14 +0000

I needed to trigger automation scripts on an old Android phone from a cloud VPS, without opening ports or spinning up Redis. So I built a job queue out of Flask and SQLite.

Here is a deep dive into what it is, how SQLite handles the concurrency, and what happened when I tried to stress-test it in production.

What it is (and what it isn't)

Intent Bus is designed to be the missing link between simple cron jobs and heavy enterprise message brokers.

What it aims for:

Giving indie hackers and home-lab enthusiasts actual execution guarantees (retries, dead-letter queues) without needing to deploy or maintain broker infrastructure.
Running anywhere. You can deploy the server on a free cloud tier, and run workers on a Raspberry Pi, a Termux phone, or a cheap VPS.

What it is not:

It is not a Kafka replacement. It is not meant for microservices processing 10,000 transactions per second. If you have enterprise scale, use enterprise tools.

The Core Features

Cron is strictly fire-and-forget, with no coordination and silent failures. To fix that, Intent Bus provides:

Feature	Description
Atomic Locking	Prevents double-claiming. A cryptographic lease token is issued to the claiming worker.
Reliable Delivery	If a worker crashes or drops offline, the lease expires and the job requeues with exponential backoff.
Dead-Letter Queue	Failed jobs (e.g., after 3 attempts) are archived for inspection rather than deleted.
Capability Routing	Workers advertise what they can do; jobs require what they need (e.g., require a worker with `ffmpeg`).
Priority Queues	High-priority intents are always claimed and processed first.

The SQLite Atomic Lock

The biggest challenge with using SQLite for a queue is concurrent writes. If 40 workers poll the server at the exact same millisecond, they can't all claim the same job.

Intent Bus solves this by strictly enforcing SQLite's WAL (Write-Ahead Logging) mode and relying on the UPDATE ... RETURNING clause introduced in SQLite 3.35.0.

When a worker asks for a job, the server executes a single atomic transaction:

UPDATE intents
SET
    status = 'claimed',
    claimed_at = :now,
    claim_expires_at = :lease_exp,
    claim_token = :token,
    claim_attempts = claim_attempts + 1
WHERE id = (
    SELECT id FROM intents 
    WHERE status = 'open' AND run_at <= :now
    ORDER BY priority DESC, created_at ASC
    LIMIT 1
)
RETURNING id, payload, claim_token;

Because SQLite serializes writes, this guarantees that only one worker will ever receive the ephemeral claim_token required to fulfill or fail that specific job.

The Stress Test: A Tale of Two Deployments

I wanted to see exactly where SQLite would choke under concurrent worker load, so I wrote an enterprise-style stress harness.

Attempt 1: The PythonAnywhere Disaster
I originally deployed the architecture to PythonAnywhere's free tier. The results were catastrophic.

When I launched the high-intensity concurrent test, it didn't even manage to start. I stepped it down to the medium-intensity test (15 concurrent workers fighting for 500 jobs), and it threw 100% network errors.

Why? Because PythonAnywhere's free tier routes requests through a single-threaded Gunicorn worker. The concurrent polling requests bottlenecked at the WSGI layer, queued up, and timed out before SQLite even knew what hit it. It wasn't a database lock issue; it was a server concurrency issue.

Attempt 2: Docker on Render
I moved the exact same SQLite file and Flask app to a basic Docker container on Render, configured with multiple application threads (--threads 4).

Render's free tier aggressively puts your container to sleep when idle, meaning the very first request takes about 30 seconds to wake the server up (the dreaded cold start). But once it's awake? It handles the load beautifully.

Here is the benchmark data from the Render Docker deployment:

Configuration	Workers	Jobs	Success	P99 Latency	Throughput
Light	5	50	100%	0.594s	3.72 j/s
Medium	15	500	98.75%	0.517s	13.27 j/s
Heavy	40	2000	99.01%	2.586s	13.62 j/s

Even with 40 concurrent workers furiously hammering the single SQLite file for write locks, there were 0 network errors, 0 lost leases, and 0 rate limit crashes. It comfortably sustained over 13 jobs per second.

Limits & The Theoretical Future

Right now, the system is tuned for safety. Payloads are strictly limited to 8KB to prevent database bloat, and busy_timeout PRAGMAs are set to allow workers to wait gracefully for locks.

Could it go further? In theory, yes. If you moved off a free cloud container onto a dedicated VPS with NVMe storage, and tuned SQLite for pure speed (e.g., PRAGMA synchronous=OFF), you could probably push 50-100 jobs a second. Furthermore, the architecture is simple enough that you could easily swap the get_db() call to point to PostgreSQL if you needed horizontal scaling.

But honestly, if you hit that scale, you should just bite the bullet and deploy Redis. The entire philosophy of Intent Bus is avoiding that infrastructure jump until it is absolutely necessary.

The server runs on Docker, Raspberry Pi, or any free cloud tier. Workers run anywhere that speaks HTTP — including Termux.

Core Server: GitHub - Intent Bus
Python SDK: GitHub - Intent Bus SDK

I would love to hear your feedback. Has anyone else experimented with using SQLite for job queueing? What edge cases did you run into?

How I Coordinate Scripts Across Devices Without Open Ports, Firebase, or a VPS

D_SECURITY — Mon, 04 May 2026 12:07:27 +0000

My scrapers run on PythonAnywhere. My phone runs Termux. I wanted them to talk to each other.

The standard options all had the same problem: they required infrastructure I didn't want to maintain.

Firebase — cloud lock-in, SDK overhead, costs money at scale
Ngrok — exposes a port on my phone, dies when the tunnel resets
A VPS with Redis — another server to maintain, SSH into, keep alive
Webhook to phone — my phone is behind NAT, no open ports

The fundamental issue: cloud scripts push, but phones can't receive pushes without an exposed port or a cloud intermediary.

So I flipped the model. Instead of the cloud pushing to the phone, the phone pulls from a neutral server. Workers poll outbound. No open ports needed. Works behind any firewall.

That's Intent Bus.

How It Works

A lightweight Flask server sits between your scripts and your workers. Scripts post jobs ("intents") to it. Workers anywhere poll for jobs matching their goal, claim them atomically, execute them, and report back.

┌─────────────────────┐
│  Cloud Script                     │ (PythonAnywhere, VPS, Lambda)
│  POST /intent                     │
└──────────┬──────────┘
                  │
                  ▼
┌─────────────────────┐
│  Intent Bus                       │  (Flask + SQLite — free tier works)
│  /intent                          │
│  /claim                           │
│  /fulfill                         │
└──────────┬──────────┘
                  │
        ┌─────┴──────┐
        ▼                    ▼
┌─────────┐  ┌─────────┐
│ Termux        │  │ Linux         │
│ Worker        │  │ Worker        │
│ (phone)       │  │ (Pi, PC).     │
└─────────┘  └─────────┘

The phone worker polls every 10 seconds. When there's a job, it claims it, runs it, and marks it fulfilled. If the worker crashes mid-job, the 60-second visibility timeout auto-requeues it.

Setup in 3 Steps

Step 1 — Deploy the server (free)

Fork the repo and deploy flask_app.py to PythonAnywhere. Set one environment variable:

os.environ["BUS_SECRET"] = "your_secret_key"

That's it. The server is live.

Step 2 — Start a worker on your phone

In Termux:

echo "your_api_key" > ~/.apikey
pkg install jq curl termux-api
bash worker.sh

The worker boots up and starts polling for send_notification intents.

Step 3 — Trigger it from anywhere

curl -X POST https://your-bus.pythonanywhere.com/intent \
  -H "X-API-Key: your_key" \
  -H "Content-Type: application/json" \
  -d '{"goal":"send_notification","payload":{"message":"Scrape complete"}}'

Your phone buzzes.

Real Use Cases

Because workers are just scripts, any device that can run bash or Python can be a worker:

Phone notifications from cloud scripts

# worker.sh on Termux
termux-notification --title "Alert" --content "$MESSAGE"

Log events from multiple machines to one file

# logger.sh on any Linux box
echo "$(date): $TEXT" >> bus.log

Relay alerts to Discord

# discord_worker.sh
curl -X POST "$DISCORD_WEBHOOK" \
  -H "Content-Type: application/json" \
  -d "{\"content\": \"$MESSAGE\"}"

Trigger a deploy on a home Raspberry Pi behind a firewall

# python_worker.py on Pi
if goal == "deploy":
    subprocess.run(["git", "pull"])
    subprocess.run(["systemctl", "restart", "myapp"])

The Pi never needs an open port. It polls out, picks up the deploy job, and runs it.

The Python SDK

pip install intent-bus

from intent_bus import IntentClient

bus = IntentClient()  # reads key from ~/.apikey automatically

# From your cloud script
bus.publish(goal="send_notification", payload={"message": "Done"})

# On your phone/Pi/worker machine
def handle(payload):
    os.system(f"termux-notification --content '{payload['message']}'")

bus.listen(goal="send_notification", handler=handle)

What This Doesn't Do

Be honest about limitations:

Not for high-frequency workloads — SQLite serializes writes, ceiling around 50-100/sec
Not exactly-once delivery — at-least-once, workers should be idempotent
Not multi-region — single SQLite node, no replication
Not a Firebase replacement — no real-time push, no client SDKs

It's designed for the gap between cron scripts and full distributed systems. Scripts, bots, scrapers, phone notifications, home automation — workloads where Redis is overkill and cron is too dumb.

Try It

Server + workers: https://github.com/dsecurity49/Intent-Bus
Python SDK: https://github.com/dsecurity49/intent-bus-sdk

pip install intent-bus

Live demo instance available — open a GitHub Discussion or join the Discord to request a free tester key.

What cross-device automation have you built with Termux? I'm curious what use cases I haven't thought of yet.

Why I Built a Job Queue With SQLite Instead of Redis (And What I Learned)

D_SECURITY — Sun, 03 May 2026 16:10:20 +0000

I needed to coordinate background scripts running across different machines.

The obvious answer was Redis. Everyone uses Redis for this. The tutorials all use Redis. The Stack Overflow answers all say "just use Redis."

So I looked at what deploying Redis would actually cost me:

A running Redis server I had to maintain
A broker to connect workers to it
Celery or RQ on top of that
Memory-based storage with no persistence by default
A whole new failure mode: what happens when Redis goes down?

I'm running scripts on PythonAnywhere and a phone running Termux. I don't want to maintain infrastructure. I just want my cloud script to say "hey, do this" and have a worker somewhere execute it.

So I built Intent Bus — a pure HTTP job queue backed by SQLite. No Redis. No broker. No open ports required.

Here's what I learned.

The Core Idea

Any script POSTs a job to a Flask server. Workers anywhere poll for jobs matching their goal, claim them atomically, execute them, and mark them fulfilled.

Cloud Script
    │
    │ POST /intent  {"goal": "send_notification", "payload": {...}}
    ▼
Intent Bus (Flask + SQLite)
    │
    │ POST /claim?goal=send_notification
    ▼
Worker (Termux / Linux / anywhere)
    │
    │ executes job → POST /fulfill/<id>
    ▼
Done

Workers poll outbound. They don't need an open port. They work behind any firewall.

The SQLite Locking Question

The part everyone pushes back on: can SQLite actually handle concurrent workers without race conditions?

Yes — with the right setup. Here's the exact claim query:

BEGIN IMMEDIATE;

WITH candidate AS (
    SELECT id
    FROM intents
    WHERE expires_at > :now
      AND status != 'failed'
      AND claim_attempts < :max_attempts
      AND (status='open' OR (status='claimed' AND claimed_at < :stale))
      AND (publisher=:worker_key OR visibility='public')
      AND goal=:target_goal
    ORDER BY claim_attempts ASC, created_at ASC, id ASC
    LIMIT 1
)
UPDATE intents
SET status='claimed',
    claimed_at=:now,
    claimed_by=:claimer,
    claim_attempts=claim_attempts+1
WHERE id = (SELECT id FROM candidate)
  AND claim_attempts < :max_attempts
  AND (status='open' OR (status='claimed' AND claimed_at < :stale))
RETURNING id, goal, payload, claim_attempts;

BEGIN IMMEDIATE acquires a write lock before the CTE evaluates. The entire block — candidate selection, UPDATE, and RETURNING — is a single atomic statement. No other writer can start a transaction until this one commits. One worker gets the job; every other concurrent claim hits the claim_attempts < :max_attempts guard or finds no candidate and gets a 204 back.

With PRAGMA journal_mode=WAL and PRAGMA busy_timeout=30000, readers never block writers and the system handles short bursts cleanly.

In practice on PythonAnywhere's free tier: tested with 3 concurrent workers, zero double-claims.

What SQLite Gets You That Redis Doesn't

Aspect	SQLite (Intent Bus)	Redis + Celery
Infrastructure	Zero — just a file	Redis server + broker
Persistence	ACID by default	Requires AOF/RDB config
Inspect queue	Any SQL tool	Redis CLI or dashboard
Self-hostable	Yes, free tier works	Needs paid hosting or VPS
Mobile workers	Yes, just HTTP	Complex from Termux
Failure recovery	60s visibility timeout, auto-requeue	Ack-based, more complex
Throughput ceiling	~50-100 writes/sec	Tens of thousands/sec

The throughput ceiling is the honest limitation. SQLite WAL mode serializes writes. At high concurrency (>10 req/sec sustained) you'll see queue buildup. For that use case, use Redis — it's the right tool.

For low-to-medium frequency workloads (scripts, bots, scrapers, phone notifications), SQLite is not just adequate — it's better, because you get persistence, inspectability, and zero infrastructure.

Delivery Guarantees

Intent Bus provides at-least-once delivery. If a worker crashes after claiming but before fulfilling, the 60-second visibility timeout auto-requeues the job. The next available worker picks it up.

This means workers should be idempotent where possible. For most script automation use cases (send a notification, write a log, trigger a webhook), this is fine.

The Scale Ceiling

I get asked: "what happens when this gets popular?"

The upgrade path is straightforward:

Swap SQLite for PostgreSQL — minimal code changes, the query logic is identical
Move to a paid VPS for higher CPU allocation
Split read/write paths if needed

But the honest answer is: if you need to worry about that upgrade path, Intent Bus isn't the right tool for you today. It's designed for the tier below that problem.

Try It

pip install intent-bus

from intent_bus import IntentClient

bus = IntentClient(api_key="your_key")

# Publish a job
bus.publish(goal="send_notification", payload={"message": "Scrape complete"})

# Listen for jobs
def handle(payload):
    print(payload["message"])

bus.listen(goal="send_notification", handler=handle)

Full server code, spec, and worker examples: https://github.com/dsecurity49/Intent-Bus

The server runs free on PythonAnywhere. Workers run anywhere that can make HTTP requests — including a phone.

Have you used SQLite as a message broker in production? I'd genuinely like to know where it held up and where it broke down — drop a comment.

How I Control My Android Phone From a Cloud Server Using 100 Lines of Flask

D_SECURITY — Fri, 24 Apr 2026 08:29:05 +0000

My background scrapers run on PythonAnywhere. My phone runs Termux. I wanted the scrapers to ping my phone when something interesting happened — without Firebase, without Ngrok, without paying for anything.

So I built Intent Bus.

The Problem

Most cross-device automation forces you to choose between:

Too simple: hardcoded cron jobs with no coordination
Too complex: Redis, RabbitMQ, Firebase — full infrastructure for a simple notification

I just wanted my cloud script to say "hey, buzz my phone." That's it.

How It Works

Intent Bus is a tiny Flask app backed by SQLite. It has one job: let scripts post work, and let workers claim and execute that work.

Step 1 — Cloud script posts an intent:

curl -X POST https://your-bus.pythonanywhere.com/intent \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your_key_here" \
  -d '{"goal":"send_notification","payload":{"message":"Scrape complete"}}'

Step 2 — Termux worker claims it:

A bash script running on my phone polls the bus every 10 seconds. When it sees a job, it claims it with an atomic lock, executes it, and marks it fulfilled.

Step 3 — Phone buzzes:

termux-notification --title "System Update" --content "$MESSAGE"

That's the entire flow.

What Makes It Reliable

Three things I had to get right:

Atomic locking — SQLite's UPDATE with rowcount check ensures only one worker ever claims a job, even with multiple workers running simultaneously.

Visibility timeout — if a worker crashes mid-job, the lock expires after 60 seconds and the job goes back into the queue automatically.

Topic routing — workers only claim jobs matching their goal via ?goal=. A notification worker never accidentally picks up a logging job.

The Architecture

Cloud Scraper (PythonAnywhere)
        |
        | POST /intent
        ↓
Intent Bus (Flask + SQLite)
        |
        | claim + fulfill
        ↓
Termux Worker (Android Phone)
        |
        | termux-notification
        ↓
📱 Phone Notification

What It Can Do

Because the bus is just HTTP, any script anywhere can post or claim jobs:

Scraper finishes → notify phone
Website goes down → alert Discord
GitHub push → trigger deploy on home Raspberry Pi behind a firewall
Old Android phone → free Twilio replacement via termux-sms-send

Security

After posting this publicly, people immediately stress-tested the open endpoint. Fair. I added API key auth — key stored in PythonAnywhere's WSGI environment, never in the repo. Workers read from a local .apikey file.

Try It

The full code, spec, and examples are on GitHub:

https://github.com/dsecurity49/Intent-Bus

It runs free on PythonAnywhere + any Android phone with Termux. No Docker, no dependencies beyond Flask.

If you build a worker script for it, PRs are welcome.

Update: v7 is Live

Since the original post, Intent Bus has been hardened significantly based on real-world feedback:

Rate limiting (60 req/min per key+IP)
Tester key system — request a free key to try the live instance
Intent expiry after 24 hours
Python worker for non-bash users
Security headers and payload size limits

DM me on Dev.to, open a GitHub Issue, or join the Discord to request a free tester key: https://discord.gg/bzAneAQzGX