DEV Community: El Housseine Jaafari

Building an Engineering & Security News Aggregator (10 Sources, No APIs)

El Housseine Jaafari — Wed, 01 Apr 2026 18:10:32 +0000

We built a curated engineering and security news aggregator that pulls from 10 high-signal sources, deduplicates content, and updates every 6 hours.

No paid APIs. No scraping. No login. Just clean, structured news for developers.

This post breaks down exactly how it works.

What This Is

A lightweight news wire combining:

Hacker News
Lobsters
InfoQ
Cloudflare Blog
Krebs on Security
The Hacker News (Security)
NIST NVD (vulnerabilities)
GitHub Blog
OpenAI Blog
Anthropic Research

The goal: high-quality signal, zero noise, zero cost.

Why Build This?

Most engineering/news aggregators fail in one of these ways:

Too noisy (no curation)
Too expensive (paid APIs)
Too slow (manual updates)
Too fragmented (you check 10 sites anyway)

We wanted:

A single feed
Fresh updates (but not real-time obsession)
No operational cost
No lock-in (no accounts, no tracking)

Stack

Hono (API layer)
Drizzle ORM
Postgres
Next.js (frontend)
RSS feeds + Hacker News Firebase API

High-Level Architecture

           ┌───────────────┐
           │   RSS Feeds   │
           │ (9 sources)   │
           └──────┬────────┘
                  │
                  ▼
           ┌───────────────┐
           │ Fetch Workers │
           │ (every 6 hrs) │
           └──────┬────────┘
                  │
                  ▼
        ┌──────────────────────┐
        │ Normalize Articles   │
        │ title, url, date     │
        └─────────┬────────────┘
                  │
                  ▼
        ┌──────────────────────┐
        │ SHA-256 Deduplication│
        │ (based on URL)       │
        └─────────┬────────────┘
                  │
                  ▼
           ┌───────────────┐
           │   Postgres    │
           └──────┬────────┘
                  │
                  ▼
           ┌───────────────┐
           │   Hono API    │
           └──────┬────────┘
                  │
                  ▼
           ┌───────────────┐
           │   Next.js UI  │
           └───────────────┘

Data Sources

We deliberately chose sources with:

High editorial quality
Low duplication between each other
Stable RSS feeds or APIs

Breakdown

Source	Type	Why It Matters
Hacker News	API	Real-time dev signal
Lobsters	RSS	More technical discussions
InfoQ	RSS	Deep engineering content
Cloudflare Blog	RSS	Infra + performance insights
Krebs on Security	RSS	Trusted security reporting
The Hacker News	RSS	Security news (broader)
NIST NVD	RSS/API	Verified vulnerabilities
GitHub Blog	RSS	Platform + ecosystem updates
OpenAI Blog	RSS	AI developments
Anthropic Research	RSS	AI + safety research

Fetching Strategy

We run a simple scheduled job:

// every 6 hours
cron.schedule("0 */6 * * *", async () => {
  await fetchAllSources();
});

Why every 6 hours?

Keeps content fresh
Avoids unnecessary load
Works well with RSS update frequencies

Deduplication (Key Part)

Different sources often post the same story.

We solve this using SHA-256 hashing of URLs.

import { createHash } from "crypto";

function hashUrl(url: string) {
  return createHash("sha256").update(url).digest("hex");
}

Why URL hashing?

Fast
Deterministic
No fuzzy matching complexity
Works across sources

Tradeoff

Won’t catch rewritten articles with different URLs
But avoids false positives (important for trust)

Normalization

Each source has its own format. We normalize into a single shape:

type Article = {
  title: string;
  url: string;
  source: string;
  publishedAt: Date;
};

This keeps the frontend simple and predictable.

API Layer (Hono)

Example endpoint:

app.get("/articles", async (c) => {
  const articles = await db.query.articles.findMany({
    orderBy: (a, { desc }) => [desc(a.publishedAt)],
    limit: 100,
  });

  return c.json(articles);
});

Minimal, fast, no overengineering.

Frontend (Next.js)

Server-rendered list
No login required
No personalization
Just chronological, deduplicated news

Limitations

Not real-time (by design)
No personalization
Deduplication is URL-based only
Dependent on RSS availability

What We’d Improve

Smarter clustering (same story, different URLs)
Tagging (infra, AI, security, etc.)
Optional filters (without accounts)

Try It

The news wire is open to everyone:

👉 https://clawship.app/blog/engineering-security-news-wire

Connect with Us

Discord: https://discord.gg/
Twitter: https://twitter.com/
News Wire: https://clawship.app/blog/engineering-security-news-wire

Introducing MCCP (Managed Capabilities Control Plane): MCP + Browser Automation (Alpha) on Clawship

El Housseine Jaafari — Fri, 13 Mar 2026 15:54:15 +0000

MCCP in one line

MCCP (Managed Capabilities Control Plane) is the layer in Clawship that connects your AI assistant to real-world capabilities—MCP (Model Context Protocol) servers, integrations, and Browser Automation (Alpha)—with safe, reviewable workflows.

Try it here: https://clawship.app

Screenshot: MCCP Browser (Alpha) — launch authenticated browser sessions to automate platforms through your AI assistant.

Why MCCP matters (beyond tool-calling demos)

Shipping an AI agent to production usually breaks on the “boring” parts:

Integrations drift (schemas, tokens, rate limits)
Different agents need different tools and permissions
Teams want draft-first outputs and explicit confirmation before publishing
Some workflows are still UI-only (no reliable API)

MCCP exists to make those operational edges predictable.

What you can do with MCCP

1) Connect MCP servers (Model Context Protocol)

MCP gives you a standardized way to expose tools (SaaS apps, internal services, custom servers) to your agent.

Example MCP config (conceptual):

{
  "agents": {
    "list": [
      {
        "id": "main",
        "mcp": {
          "servers": [
            { "name": "notion", "command": "npx", "args": ["-y", "@notionhq/mcp"] },
            { "name": "linear", "command": "npx", "args": ["-y", "@linear/mcp"] }
          ]
        }
      }
    ]
  }
}

2) Automate UI workflows with Browser Automation (Alpha)

When the UI is the source of truth (admin consoles, partner portals, legacy tools), MCCP Browser sessions let your assistant execute the workflow directly in a real browser session.

3) Stay safe with draft-first workflows

For posts, announcements, and user-facing updates, the best default is:

Generate content
Save as draft
Human review
Publish explicitly

Use cases (where readers will feel the value)

Developer productivity: pull context from docs/tickets via MCP, generate changelogs and release notes
Content operations: draft DEV.to/Hashnode posts from real product updates, keep them reviewable
Ops & support: create structured summaries and next steps from your tool stack
Growth automation: repeatable, safe workflows across platforms

Keywords (optional, but good for SEO)

MCCP
Managed Capabilities Control Plane
MCP (Model Context Protocol)
MCP server integrations
browser automation for AI agents
AI agent tools / agent ops
developer automation platform

Call to action

If you’re exploring MCP + browser automation and want a managed way to ship it with safer defaults, start here:

https://clawship.app

Host Your Own Matrix + Element on a Server (Without Losing Your Weekend)

El Housseine Jaafari — Sat, 21 Feb 2026 17:05:22 +0000

Why Matrix, why now?

Matrix is an open protocol for real-time messaging. The two pieces you’ll run are:

Synapse (the Matrix homeserver): users, rooms, federation, auth, storage.
Element (the client): the web app people actually use to chat.

You can absolutely self-host this. You can also absolutely spend a weekend in TLS/DNS/Reverse-proxy purgatory if you’re not careful. Let’s do it the sane way.

What you’ll set up

A VPS/server with Linux
A domain, with:
- matrix.example.com → Synapse
- element.example.com → Element (web)
Nginx as reverse proxy
Let’s Encrypt TLS certificates
Synapse running in Docker

This tutorial assumes:

Ubuntu 22.04+
You have SSH access and a domain name

Step 0: Pick your domain + DNS

You need two DNS records:

A/AAAA record: matrix.example.com → your server IP
A/AAAA record: element.example.com → your server IP

Give DNS a minute to propagate.

Step 1: Install prerequisites

sudo apt update
sudo apt install -y docker.io docker-compose-plugin nginx certbot python3-certbot-nginx
sudo systemctl enable --now docker nginx

Step 2: Generate Synapse config (Docker)

Create a working directory:

mkdir -p ~/matrix/synapse
cd ~/matrix/synapse

Generate Synapse configuration (replace example.com):

docker run -it --rm \
  -v "$(pwd)":/data \
  -e SYNAPSE_SERVER_NAME=example.com \
  -e SYNAPSE_REPORT_STATS=no \
  matrixdotorg/synapse:latest generate

This creates homeserver.yaml and keys in ~/matrix/synapse.

Step 3: Create a docker-compose.yml for Synapse

Create docker-compose.yml:

services:
  synapse:
    image: matrixdotorg/synapse:latest
    container_name: synapse
    restart: unless-stopped
    volumes:
      - ./data:/data
    environment:
      - SYNAPSE_CONFIG_PATH=/data/homeserver.yaml
    ports:
      - "8008:8008"

Now, move your generated config into ./data (Synapse expects /data inside the container):

mkdir -p data
# If generate already wrote into ./data, skip this.
# Otherwise, move the generated files:
shopt -s dotglob
mv homeserver.yaml *signing.key data/ 2>/dev/null || true

Start Synapse:

docker compose up -d

Synapse is now listening on http://127.0.0.1:8008 (we’ll put Nginx in front).

Step 4: Nginx reverse proxy for Synapse

Create an Nginx site for matrix.example.com:

sudo nano /etc/nginx/sites-available/matrix

Paste:

server {
  listen 80;
  server_name matrix.example.com;

  location / {
    proxy_pass http://127.0.0.1:8008;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}

Enable it:

sudo ln -s /etc/nginx/sites-available/matrix /etc/nginx/sites-enabled/matrix
sudo nginx -t
sudo systemctl reload nginx

Step 5: Add HTTPS (Let’s Encrypt)

sudo certbot --nginx -d matrix.example.com

Follow prompts. Certbot updates Nginx to serve HTTPS.

Step 6: Set up Element Web

Element Web is just static files served by a web server. Easiest: run it in Docker and proxy it.

Create a directory:

mkdir -p ~/matrix/element
cd ~/matrix/element

Create docker-compose.yml:

services:
  element:
    image: vectorim/element-web:latest
    container_name: element
    restart: unless-stopped
    ports:
      - "8080:80"
    volumes:
      - ./config.json:/app/config.json:ro

Create config.json (replace domains):

{
  "default_server_config": {
    "m.homeserver": {
      "base_url": "https://matrix.example.com",
      "server_name": "example.com"
    }
  },
  "disable_custom_urls": false,
  "disable_guests": true,
  "default_theme": "light"
}

Start Element:

docker compose up -d

Element is now at http://127.0.0.1:8080.

Step 7: Nginx reverse proxy + HTTPS for Element

Create:

sudo nano /etc/nginx/sites-available/element

Paste:

server {
  listen 80;
  server_name element.example.com;

  location / {
    proxy_pass http://127.0.0.1:8080;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
}

Enable + reload:

sudo ln -s /etc/nginx/sites-available/element /etc/nginx/sites-enabled/element
sudo nginx -t
sudo systemctl reload nginx

Add HTTPS:

sudo certbot --nginx -d element.example.com

Now you should be able to open:

https://element.example.com

…and point it at your homeserver.

Step 8: Create your first Matrix user

By default, Synapse disables open registration (good). Create an admin user:

docker exec -it synapse register_new_matrix_user \
  -c /data/homeserver.yaml \
  http://localhost:8008

Follow prompts. You can create additional non-admin users the same way.

Step 9 (Important): Don’t forget the boring-but-deadly parts

Self-hosting isn’t hard, it’s just annoying in exactly the places you can’t afford to mess up:

Backups (Synapse data + database if you add Postgres)
Updates (Synapse and Element)
Monitoring logs
Firewalling and SSH hardening
Rate-limits and anti-spam settings

If you’re doing this for a community, these are not optional.

Want to save yourself the headache?

If you read the steps above and thought “I’d rather eat glass than debug Nginx + TLS at 2am,” here’s the spicy truth: you can skip most of this pain.

Using https://clawship.app, you can spin up Matrix + Element fast and also get your own bot interaction without babysitting the plumbing. Same end goal, less yak-shaving.

(If you like yak-shaving, carry on. No judgment. Mild concern, maybe.)

Quick troubleshooting

Element loads but can’t connect: confirm config.json points to https://matrix.example.com.
502 from Nginx: verify containers are running:

  docker ps

Certbot fails: DNS record is wrong or not propagated.
Synapse feels slow: consider switching to Postgres (recommended for real use).

Wrap-up

You now have:

A working Matrix homeserver (Synapse)
Element Web served securely over HTTPS
A foundation you can harden and scale

If you want, tell me your domain + whether you want Postgres and I’ll provide a production-grade compose setup (Synapse + Postgres + Redis, plus safer Nginx headers).

Self‑Hosting Matrix + Element on Your Own Server (The Friendly, Spicy Tutorial)

El Housseine Jaafari — Sat, 21 Feb 2026 17:02:07 +0000

Why you’d do this (and why it’s a bit of a rite of passage)

Matrix is an open, decentralized communication protocol. Element is the most common client for it. Self‑hosting them is appealing when you want:

Ownership over your comms stack
Your own domain + branding
Data residency / compliance control
The satisfaction of running real infrastructure

It’s also… a small pile of moving parts: DNS, TLS, reverse proxies, federation ports, well‑known files, and making sure your homeserver doesn’t melt the moment you enable registration.

This guide walks you through a practical, beginner‑friendly setup using Docker Compose and Synapse (the reference Matrix homeserver).

What you’ll build

By the end you’ll have:

A Matrix homeserver at: matrix.yourdomain.com
Element Web at: chat.yourdomain.com
HTTPS everywhere (Let’s Encrypt)
A working client login and room chat

Assumptions

You own a domain, e.g. yourdomain.com
You have a VPS/server (Ubuntu/Debian‑like)
You can open ports 80 and 443 to the server

Step 0: DNS (don’t skip this, future‑you will cry)

Create DNS records:

matrix.yourdomain.com → your server IP (A/AAAA)
chat.yourdomain.com → your server IP (A/AAAA)

You can put both behind the same reverse proxy; we’ll do that.

Step 1: Install Docker + Compose

On most modern distros:

sudo apt update
sudo apt install -y docker.io docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker $USER

Log out and back in so your user is in the docker group.

Step 2: Pick a reverse proxy (Caddy keeps your blood pressure lower)

You can do this with Nginx + Certbot, Traefik, etc. For a clean tutorial, Caddy is hard to beat: automatic TLS, simple config.

We’ll run:

caddy (reverse proxy + TLS)
synapse (Matrix homeserver)
element-web (web client)
postgres (recommended DB for Synapse)

Create a working directory:

mkdir -p ~/matrix-stack
cd ~/matrix-stack

Step 3: Generate Synapse config

Synapse needs a config file + keys.

Create a docker-compose.yml we’ll fill in (don’t run it yet):

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: synapse
      POSTGRES_PASSWORD: CHANGE_ME_STRONG
      POSTGRES_DB: synapse
    volumes:
      - ./data/postgres:/var/lib/postgresql/data
    restart: unless-stopped

  synapse:
    image: matrixdotorg/synapse:latest
    environment:
      SYNAPSE_SERVER_NAME: yourdomain.com
      SYNAPSE_REPORT_STATS: "no"
    volumes:
      - ./data/synapse:/data
    depends_on:
      - postgres
    restart: unless-stopped

  element-web:
    image: vectorim/element-web:latest
    volumes:
      - ./data/element/config.json:/app/config.json:ro
    restart: unless-stopped

  caddy:
    image: caddy:2
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./data/caddy:/data
      - ./data/caddy-config:/config
    restart: unless-stopped

Now generate Synapse config and keys:

docker run -it --rm \
  -v "$(pwd)/data/synapse:/data" \
  -e SYNAPSE_SERVER_NAME=yourdomain.com \
  -e SYNAPSE_REPORT_STATS=no \
  matrixdotorg/synapse:latest generate

This creates data/synapse/homeserver.yaml and signing keys.

Configure Synapse to use Postgres

Edit data/synapse/homeserver.yaml and set the database section (replace the default sqlite3):

database:
  name: psycopg2
  args:
    user: synapse
    password: "CHANGE_ME_STRONG"
    database: synapse
    host: postgres
    cp_min: 5
    cp_max: 10

Also set the public base URL:

public_baseurl: "https://matrix.yourdomain.com/"

Step 4: Configure Element Web

Create data/element/config.json:

{
  "default_server_config": {
    "m.homeserver": {
      "base_url": "https://matrix.yourdomain.com",
      "server_name": "yourdomain.com"
    }
  },
  "disable_custom_urls": true,
  "disable_guests": true,
  "brand": "Your Matrix"
}

Step 5: Reverse proxy + TLS with Caddy

Create Caddyfile:

matrix.yourdomain.com {
  reverse_proxy synapse:8008
}

chat.yourdomain.com {
  reverse_proxy element-web:80
}

Synapse listens on 8008 inside the container by default.

Start everything:

docker compose up -d

Check logs if needed:

docker compose logs -f --tail=200

Step 6: Create your first user

Exec into the synapse container and register a user.

docker compose exec synapse register_new_matrix_user \
  -c /data/homeserver.yaml \
  http://localhost:8008

Follow the prompts (username/password). Then open:

Element Web: https://chat.yourdomain.com

username: @youruser:yourdomain.com

Step 7 (important): Lock it down a bit

Self‑hosting chat is fun until it becomes a spam magnet.

Consider:

Disable open registration unless you truly want it
Enable reCAPTCHA / token‑based registration if you must allow signups
Keep Synapse updated
Monitor disk space (media can grow fast)
Back up Postgres + Synapse keys (data/synapse)

In homeserver.yaml, check:

enable_registration: false

Federation and “well‑known” (optional, but good to know)

If you want @user:yourdomain.com to work nicely and support federation cleanly, you’ll eventually run into:

.well-known/matrix/server
.well-known/matrix/client
Potentially opening port 8448 (depends on your setup)

This is where the “I just wanted to chat with friends” journey turns into “I am now operating a small internet service.” Congrats.

If you want a follow‑up post, I can provide a clean, battle‑tested .well-known + federation configuration.

The honest part: this is a headache you don’t have to keep

If you love tinkering, self‑hosting Matrix is rewarding.

If you’d rather skip the yak‑shaving and still get:

Matrix + Element on your own domain
A managed path to add a bot and interact with it
A smoother “it just works” experience

…you can save yourself the whole weekend by using https://clawship.app.

Clawship is built to help you spin up your stack faster and add your own bot interaction without reinventing the infrastructure wheel. You focus on the community and automation, not the reverse proxy sudoku.

Troubleshooting quick hits

Caddy won’t get certificates → check DNS points to the server and ports 80/443 are open
Synapse errors on DB → verify Postgres credentials match in homeserver.yaml
Element loads but can’t login → confirm Element config points at https://matrix.yourdomain.com
Media storage exploding → set retention policies and monitor disk usage

What’s next

If you want, I can write a follow‑up tutorial on:

Federation + .well-known done correctly
Adding a Matrix bot (and safe bot permissions)
Moderation + anti‑spam strategy
Backups and disaster recovery

And if you’d rather just ship: https://clawship.app.

From Prompt to Post: Secure Auto‑Publishing to DEV.to (and Medium) with Clawship

El Housseine Jaafari — Sat, 21 Feb 2026 02:48:00 +0000

Why this matters

Writing is one of the highest‑leverage ways to ship ideas: announcements, changelogs, tutorials, and design notes. The problem is the last mile—formatting, previews, tags, canonical URLs, and the “did it actually publish?” dance.

Clawship.app aims to make that last mile boring (in a good way): you write the content, your assistant turns it into a clean markdown article, and Clawship can save as a draft or publish to DEV.to and Medium through your connected accounts.

This post explains the workflow, the prototype value, and the security model at a practical level.

What Clawship gives you

At a high level, Clawship combines:

A hosted assistant you can chat with to draft, edit, and polish posts.
A publishing bridge that can create drafts or publish to:
- DEV.to
- Medium
A workflow that keeps you in control: you can require “draft only,” request a review, then publish when ready.

The result: fewer tools, fewer copy/paste errors, and a repeatable publishing pipeline.

The end‑to‑end workflow (prototype‑friendly)

Here’s the simplest “prove it works” flow your team can demo:

Ask for an article
- Example: “Write a DEV.to article about our feature X, 900–1200 words, include code samples.”
Iterate in chat
- Tighten the outline
- Add screenshots/links
- Adjust tone (marketing vs. technical)
Generate publish‑ready markdown
- Proper headings, lists, code fences, etc.
- A clear structure (intro → problem → solution → steps → FAQ)
Publish as a draft (recommended)
- Clawship creates a DEV.to draft under your connected account.
Review + publish
- You confirm formatting, tags, canonical URL, and call‑to‑action.
- Then publish to DEV.to and optionally mirror to Medium.

Even with a prototype, this flow demonstrates real user value: the assistant produces content in the exact format platforms expect, and the “last mile” becomes one click.

What the assistant produces (DEV.to‑ready)

DEV.to is markdown‑first. A Clawship‑generated post can include:

## and ### headings with consistent hierarchy
Bullet lists for scannability
Code blocks with language hints
Short, skimmable paragraphs (DEV.to readers love this)
A tight conclusion + CTA back to https://clawship.app

Tip: If you want maximum credibility in a demo, ask the assistant to include a “What I built / How it works / What’s next” section and keep claims testable.

Draft vs. publish: staying in control

A secure publishing workflow isn’t just about protecting accounts—it’s also about preventing accidental public posts.

Clawship supports a two‑step posture:

Default: Draft mode
- The assistant writes the article and saves it as a draft.
- You review on DEV.to/Medium before publishing.
Publish mode (explicit)
- You must clearly ask to publish.
- This reduces “oops” moments and helps teams keep approvals in the loop.

For prototypes and early teams, draft‑first is the safest and most demo‑friendly default.

Security model (what “secure” means in practice)

When you’re proving a prototype, “secure” should mean concrete, observable properties:

1) You control where content goes

Publishing only happens to the platforms you connect (DEV.to and/or Medium). If you don’t connect an account, there’s nothing to publish to.

2) Least‑surprise behavior

Good assistants should not take irreversible actions without an explicit request. A robust posture is:

Generate content → create draft → wait for confirmation → publish

3) Scope‑limited permissions (platform‑dependent)

DEV.to and Medium integrations typically rely on an API token or OAuth‑style authorization. The safest approach is to grant only what’s needed to create or update posts.

4) No “mystery steps”

A publishing pipeline should be auditable at the product level:

What platform is being targeted?
Is the action “draft” or “publish”?
What title/tags are set?

Clawship’s goal is to make these knobs obvious so teams can verify behavior during demos and tighten it later for production.

(If you’re doing an external security review, treat tokens like passwords: rotate them, limit access, and store them using your platform’s secret management.)

Medium support (same idea, different audience)

Once you have the markdown and a clear structure, Medium becomes a second distribution channel:

DEV.to tends to reward practical, tool‑driven posts.
Medium tends to reward narrative + perspective.

With Clawship you can:

Write once
Adapt the intro/outro for the audience
Save drafts in both places
Publish when the timing is right

A concrete demo script (copy/paste)

If you want to demonstrate the prototype end‑to‑end in under 5 minutes:

In chat: “Write a DEV.to article: From Prompt to Post with Clawship, include a 5‑step workflow and a security section. End with a CTA to clawship.app.”
Ask: “Now tighten it to ~900 words and make the tone more technical.”
Ask: “Create a DEV.to draft with tags: productivity, devtools, writing, ai.”
Open the DEV.to draft, show formatting is correct.
Publish (optional) or keep it as draft to emphasize safe defaults.

Call to action

If you want writing + publishing to feel like part of your developer workflow—not a separate chore—try Clawship.

Website: https://clawship.app
Use it to draft technical posts, product updates, or docs
Publish to DEV.to and Medium with a draft‑first workflow

If you’re evaluating the prototype, the best test is simple: pick a real post your team needs to publish this week and see how much time you save.

Fixing no space left on device for Docker when /run is full (Ubuntu)

El Housseine Jaafari — Tue, 07 Oct 2025 11:45:38 +0000

If Docker or containerd suddenly refuse to start containers with errors like:

Error response from daemon: failed to create task for container:
failed to start shim: symlink ... /run/containerd/...: no space left on device: unknown

…but df -h shows plenty of free disk on /, you’re likely hitting a full /run (tmpfs) or inode exhaustion. This guide explains why it happens and how to fix it quickly—and permanently.

TL;DR (Quick Unblock)

# Give /run more room (temporary until reboot)
sudo mount -o remount,size=512M /run   # or size=1G if you have RAM

# Restart services to clear stale state
sudo systemctl restart systemd-udevd
sudo systemctl restart containerd docker

If it still fails:

sudo systemctl stop docker containerd
sudo rm -rf /run/containerd/io.containerd.runtime.v2.task/*
sudo systemctl start containerd docker

Make it persistent:

echo 'tmpfs /run tmpfs defaults,size=512M 0 0' | sudo tee -a /etc/fstab
sudo mount -o remount /run

Why this happens

/run is a RAM-backed tmpfs used by systemd, udev, Docker/containerd shims, etc.
On some systems, /run defaults to a small size (e.g., ~200 MB).
When /run runs out of space or inodes, containerd can’t create its shim files/symlinks under /run/containerd/..., resulting in “no space left on device” even though your disk has free space.

How to confirm the root cause

Check Docker’s data roots and the /run mount:

docker info 2>/dev/null | sed -n 's/^ Docker Root Dir: //p'
df -hT /var/lib/docker /var/lib/containerd /run
df -i  /var/lib/docker /var/lib/containerd /run
sudo du -xhd1 /run | sort -h

What to look for:

/run type is tmpfs and Use% is ~100%; or
/run inodes are 100% used (df -i shows IUse% 100%); and
Large directories under /run (commonly udev, containerd, or docker).

Example output (problem case):

tmpfs  197M  197M  8.0K  100%  /run
tmpfs  251822 251816 6  100%   /run
sudo du -xhd1 /run | sort -h
...
3.7M  /run/containerd
193M  /run/udev
197M  /run

Step-by-step fixes

1) Temporarily grow `/run` (immediate relief)

sudo mount -o remount,size=512M /run      # or size=1G on bigger hosts

Then restart the relevant services:

sudo systemctl restart systemd-udevd
sudo systemctl restart containerd docker

Re-check:

df -hT /run && df -i /run

2) Clear stale containerd runtime state (if needed)

Stop services, remove leftover shim dirs, start again:

sudo systemctl stop docker containerd
sudo rm -rf /run/containerd/io.containerd.runtime.v2.task/*
sudo systemctl start containerd docker

Safe because those are runtime artifacts recreated on start.

3) Make the larger `/run` persistent (survives reboot)

Append to /etc/fstab:

echo 'tmpfs /run tmpfs defaults,size=512M 0 0' | sudo tee -a /etc/fstab
sudo mount -o remount /run

Pick a size appropriate to your RAM (e.g., 512 MB–1 GB for servers running containers).

Related issues & checks

A) Inode exhaustion (lots of tiny files)

If df -i /run shows 100% usage:

The remount with a larger size also increases available inodes on tmpfs.
Restarting systemd-udevd, containerd, and docker frees stale runtime entries.

For Docker’s persistent storage (not /run), prune if inodes are tight:

docker system df
docker system prune -f
docker image prune -a -f
docker volume prune -f
docker builder prune -a -f

B) Docker root on a different filesystem that’s full

If /var/lib/docker or /var/lib/containerd are on a small/near-full partition, move Docker’s data:

sudo systemctl stop docker containerd
sudo rsync -aHAXx --info=progress2 /var/lib/docker/ /mnt/docker/
printf '{\n  "data-root": "/mnt/docker"\n}\n' | sudo tee /etc/docker/daemon.json
sudo mv /var/lib/docker /var/lib/docker.bak
sudo mkdir /var/lib/docker
sudo mount --bind /mnt/docker /var/lib/docker
sudo systemctl start containerd docker
docker info | grep "Docker Root Dir"
# If all good, clean up:
sudo rm -rf /var/lib/docker.bak

Prevention tips

Right-size /run: Set size=512M–1G in /etc/fstab for hosts that run containers or have busy udev activity.
Prune regularly: Set up periodic pruning for dangling images/builders/volumes if CI/build-heavy.
Monitor /run: Watch Use% and inode use with a simple cron or systemd timer:

  df -h /run; df -i /run

Avoid runaway logs/artifacts in /run: Runtime data should be short-lived; if something keeps filling /run, investigate that service.

FAQ

Q: I have free disk space. Why “no space left on device”?
Because the error refers to the /run tmpfs, not your disk. /run can be full (or out of inodes) while / has gigabytes free.

Q: Is it safe to delete things under /run/containerd/...?
Yes—only after stopping containerd and Docker. Those are runtime artifacts and will be recreated.

Q: Can I just reboot?
Often yes, but the problem will likely return unless you increase /run or fix the service causing the pressure.

That’s it! With a right-sized /run and a quick service restart, Docker will stop tripping over “no space left on device” despite plenty of disk space.

Implementing your own Proof Of Authentication. (Part 1)

El Housseine Jaafari — Mon, 16 Dec 2024 17:56:07 +0000

In the world of IoT, companies and innovators often turn to blockchain solutions to securely automate tasks such as collecting, filtering, and identifying incoming data. While decentralized networks can be quite helpful in many respects, certain bottlenecks still hinder scalability. For example, some consensus mechanisms require extensive re-validation and computational effort, increasing both the time and energy consumed by the network. This heightened resource usage is particularly undesirable for factories and other industries that aim to reduce their environmental impact, making the exploration of more efficient blockchain protocols essential.

Thus comes the new consensus algorithm called Proof Of Authentication that will replace the existing one Proof of Work and introduce authentication in such environments to make the blockchain application-specific.

The algorithm is designed to ensure secure and authenticated block validation in a blockchain network. Each node in the network begins by combining transactions into a block, which is then signed using the node's private key. This signed block is broadcasted across the network. Trusted nodes verify the block's signature using the source node's public key to confirm its authenticity. If the block is authenticated, it is appended with a cryptographic Proof of Authentication (PoAh), ensuring integrity and traceability, and then added to the blockchain. Blocks that fail authentication are discarded, maintaining the network's reliability and security. PoAh emphasizes a lightweight yet robust validation process, making it ideal for systems prioritizing transparency, trust, and resource efficiency.

To simplify how the algorithm works, let us translate the theoretical concept into a practical Python example. The code below demonstrates the full process, from creating a block of transactions to signing, verifying, and appending it to the blockchain. By following this example, you can gain a clear understanding of the algorithm's steps and their implementation:

import hashlib
import rsa

def sha256_hash(data):
    return hashlib.sha256(data.encode()).hexdigest()

# Generate keys for demonstration
(pubkey, privkey) = rsa.newkeys(512)

# PoAh Procedure
def poah_procedure(transactions):
    # Step 1: Combine transactions to form a block
    block = "|".join(transactions)

    # Step 2: Sign the block with private key
    signed_block = rsa.sign(block.encode(), privkey, 'SHA-256')

    # Step 3: Broadcast (simulation)
    print("Broadcasting signed block...")

    # Step 4: Trusted node verifies signature
    try:
        rsa.verify(block.encode(), signed_block, pubkey)
        print("Block authenticated!")

        # Step 5: Append PoAh and add to blockchain
        poah = sha256_hash("Proof of Authentication")
        blockchain.append(f"{block} || {poah}")
        print("Block added to blockchain.")
    except rsa.VerificationError:
        print("Block verification failed. Dropping block.")

# Blockchain simulation
blockchain = []

# Example usage
transactions = ["Tx1: A->B", "Tx2: C->D"]
poah_procedure(transactions)

print("\nBlockchain:", blockchain)

You are the bottleneck

El Housseine Jaafari — Mon, 26 Feb 2024 03:49:00 +0000

In the spring of 2012, fresh out of the Technology Institute, I brimmed with ambition and an unyielding thirst for knowledge in the realm of technology. With grand aspirations of becoming one of the foremost software engineers, I set sail into the turbulent seas of the tech industry.

My journey was marked by an unwavering dedication to learning, an unquenchable hunger to master new tools and languages. I immersed myself in the intricacies of coding patterns and algorithms, relentlessly seeking out solutions to every problem that crossed my path. Doubt never clouded my mind; I held firm to the belief that every challenge bore a solution within its depths.

This relentless pursuit of excellence forged within me the essence of a perfectionist. Armed with mastery over the tools at my disposal, I refined my craft to the point where I wielded code like a virtuoso, crafting elegant solutions with the utmost efficiency.

But time marched on, and with each passing year, the hunger within me grew. A voracious beast, it drove me to incessantly refine and perfect, never allowing me to rest until every line of code was pristine, every algorithm optimized to perfection.

Then came the fateful year of 2018, a year of reckoning. Despite my expertise, a project of my own creation faltered in the face of customer expectations. In my pride, I cast blame upon those I was meant to serve, blind to the flaws in my own approach.

On February 20th, 2018, the CEO implored me to "trust your expertise," entrusting me with the task of assembling a new team to salvage our project. With the weight of expectations upon my shoulders, I ventured forth, drawing upon years of experience to hand-pick a team worthy of the challenge.

Four months later, we emerged victorious, our project launched and ready for the world. Yet, amidst the celebrations, a shadow loomed over my leadership. Though camaraderie thrived within the team, my relentless pursuit of perfection often hindered our progress.

"You are the bottleneck," the CEO's words pierced through my defences, shattering the illusion of infallibility I had crafted around myself. At that moment of vulnerability, I realized the folly of my ways.

With humility, I sought guidance, a departure from the solitary path I had trod for so long. Through books, articles, and podcasts, I embarked on a journey of self-discovery, striving to understand the intricacies of business and the art of customer satisfaction.

And so, my journey continues, guided by the realization that true success lies not in the perfection of code, but in the fulfilment of customer needs. In this ever-evolving landscape of technology, I embrace the challenges that lie ahead, armed not just with technical prowess, but with a newfound understanding of the human element that underpins it all.

Word Definitions

Brimmed: Overflowed or filled to the edge; abounded.
Unquenchable: Impossible to satisfy or extinguish.
Intricacies: Complex details or elements; the fine points.
Virtuoso: A person highly skilled in a particular art or field, especially in music or the fine arts.
Implored: Urgently requested or begged for.
Ventured: Undertook a risky or daring journey or course of action.
Loomed: Appeared as a threat or imminent danger; overshadowed.
Trod: Walked or stepped on; followed a particular path or course of action.

GraphQL Pagination

El Housseine Jaafari — Tue, 18 Feb 2020 09:04:27 +0000

Hey there! Glad you are here and welcome to GraphQL Pagination tutorial. The goal of this tutorial is to guide you through setting up your pagination using graphql.

In graphql there are two major approaches, the first one is offset style and the second one goes by cursor style.

As a Ninja once said "行きましょ" 🙈.

Offset Pagination

Now that you have your blog up and running, you notice that you are querying all your posts. This is not efficient, especially if you go over a 100 blog posts. Your SQL query will grow by time and spend then more time to execute.

To solve the problem, by default as someone who knows SQL well, you will probably think of pagination. The way you do it is by adding two arguments to your SQL query offset¹ and limit². You are asking right now about how to achieve this in your graphql server.

Allow me to show you an example of fetching 10 blog posts from all your posts starting from the 11th one. Your query will look like this:

query {
  posts(limit: 10, offset: 10) {
    title,
    content
  }
}

Until now everything seems to be working, this type of pagination works great if you have either a static or small data. It results into a good user experience for quickly fetching the next page data.

But, this approach have its downsides and issues that needs to be addressed.

Performance

As mentioned before, in small datasets, the offset approach is good for you and will not break the user experience.

But as the data grows in your dataset and you are trying to fetch the farthest of the results 😩 it will take longer than you can imagine (You may 😢).

But how? (you ask). Let's break it down easily, take this query as an example:

query {
  posts(limit: 10, offset: 10000) {
    title,
    content
  }
}

The query is fetching 10 posts with an offset of 10K, in a first glance you think that's it, your database engine will directly understand the offset and start querying from that row.

Well! That won't happen because the way a relational database engine works, is that it will still have to compute all the skipped rows inside the server; therefore a large OFFSET might be inefficient.

You see?! It's bad right, just like if you want to start reading a book from a certain page but you still have to go over every page and count it yourself without the use of an index.

Don't worry, we got your back with a better solution 😎

Cursor pagination

This approach tend to set a pointer to a specific record in your datatable. For it to work at its finest, the cursor must be unique and sequential. What this does, is that you will always fetch data after a certain row instead of relying in the positioning of rows inside your table.

Let's have an example to understand this more:

query {
  posts(limit: 10, publisherId: 1, cursor: 1) {
    title,
    content
  }
}

The query will skip all the 24th rows and returns the next 10 results in your dataset after the value of your cursor. The cursor here refers to the id of our posts.

Looking at your resolver, it will look like this:

P.S: We are using Sequelize as our ORM framework.

{
  async posts(parent, { limit, publisherId, cursor }) {
    return await Posts.findAll({
      where: {
        publisherId,
        id: {
          [Op.gt]: cursor
        }
      },
      limit
    })
  }
}

Cursor based pagination is more performant than offsets because we can leverage database indexes on the columns in the where clause that are being used as cursors.

There is more to this solution, and many libraries and frameworks out there are presenting you with their own cursor style pagination; But we can't go over each of if, we will stick to the one that is most used with React.

Relay style pagination

Relay is a framework for building data driven React apps. Relay provides a standard mechanism to slice data and paginate the results.

query {
  posts(first: 10, after: "cursor") {
    edges {
      cursor
      node {
        id
        title
        content
      }
    }
    pageInfo {
      hasNextPage
    }
  }
}

Here, slicing is done using the first argument. This is similar to the limit which we had applied in the earlier queries. Pagination is done using the after argument, which will pass a cursor expecting results to be after that cursor. It also asks for hasNextPage which tells the client whether there are more results, since there is no concept of total number of pages in Cursor pagination.

Recap

We learned two different pagination styles, and walked trough both of it. To recap our tutorial, let's list again what each approach offers.

Offset

Pros

It gives you the ability to see the total number of pages and their progress through that total.
It gives you the ability to jump to a specific page within the set.
It’s easy to implement as long as there is an explicit ordering of the results from a query.

Cons

Using LIMIT OFFSET doesn’t scale well for large datasets. As the offset increases the farther you go within the dataset, the database still has to read up to offset + count rows from disk, before discarding the offset and only returning count rows.
If items are being written to the dataset at a high frequency, the page window becomes unreliable, potentially skipping or returning duplicate results.

Cursor

Pros

This will scale well for large datasets. We’re using a WHERE clause to fetch rows with index values less than the last index from the previous page. This lets us leverage the index on the column and the database doesn’t have to read any rows that we’ve already seen. We’re also not returning the total number of pages or items in the set, so we avoid having to calculate the full result set on each request.
The pagination process is stabilized. Instead of calculating from scratch on each request based on the total number of items, we’re always fetching the next count rows after a specific reference point. If items are being written to the dataset at a high frequency, the overall position of the cursor in the set might change, but the pagination window adjusts accordingly.

Cons

The cursor must be based on a unique, sequential column (or columns) in the source table.
There is no concept of the total number of pages or results in the set.
The client can’t jump to a specific page.

P.S.: This image shows the differences between both approaches.

Dictionary

(1): OFFSET says to skip that many rows before beginning to return rows.
(2): LIMIT is an optional clause of the SELECT statement that returns a subset of rows returned by the query.

With that said, I hope you enjoyed reading the article and understood the differences between both approaches 😄

Brought to you with ❤️

=====================

Github repo: https://github.com/JefferyHus/graphql-pagination