DEV Community: instanceofGod

SwiftDeploy: Building a Self-Writing Infrastructure Tool with OPA Policy Gates and Prometheus Observability

instanceofGod — Wed, 06 May 2026 20:46:28 +0000

A complete walkthrough of building a declarative deployment CLI from scratch — one that generates its own config files, enforces policy before acting, and audits everything it does.

The Problem
Architecture Overview
Part A — The Engine
- The Manifest: Single Source of Truth
- The Tool That Writes Its Own Infrastructure
- The API Service
- The CLI Subcommands
Part B — The Eyes and the Brain
- Instrumentation: /metrics in Pure Python
- The Policy Sidecar: OPA
- Writing Rego Policies
- Gated Lifecycle: deploy and promote
- The Status Dashboard
- The Audit Report
The Chaos Experiments
Lessons Learned
Replication Guide

The Problem

Most DevOps tooling asks you to write config files by hand. You write a docker-compose.yml, an nginx.conf, maybe a Kubernetes manifest — and then you maintain all of them separately. When something changes, you update three files instead of one, and they drift apart.

SwiftDeploy flips this. You write one file — manifest.yaml — and the tool derives everything else from it. The generated files are artifacts, not sources.

Part B goes further: the tool now refuses to act unless the environment meets policy. It has eyes (Prometheus metrics), a brain (Open Policy Agent), and a memory (an append-only audit trail).

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                        Host Machine                          │
│                                                              │
│   swiftdeploy CLI (bash)                                     │
│        │                                                     │
│        ├── reads ──► manifest.yaml                          │
│        ├── writes ─► nginx.conf, docker-compose.yml         │
│        ├── queries ► OPA :8181 (127.0.0.1 only)            │
│        └── scrapes ► /metrics via nginx :8080               │
│                                                              │
│  ┌─────────────────────────┐   ┌──────────────────────┐    │
│  │   swiftdeploy-net        │   │      opa-net          │    │
│  │                          │   │                       │    │
│  │  ┌────────┐  ┌────────┐ │   │  ┌─────────────────┐ │    │
│  │  │  app   │  │ nginx  │ │   │  │       OPA        │ │    │
│  │  │ :3000  │◄─│ :8080  │ │   │  │      :8181       │ │    │
│  │  └────────┘  └────────┘ │   │  └─────────────────┘ │    │
│  │   (no host port)  (pub) │   │   (127.0.0.1 only)   │    │
│  └─────────────────────────┘   └──────────────────────┘    │
└─────────────────────────────────────────────────────────────┘

Key isolation properties:

The app container has no host-exposed port — all traffic routes through nginx
OPA lives on a separate Docker network (opa-net) with no connection to swiftdeploy-net
OPA's port is bound to 127.0.0.1 only — the CLI on the host can reach it, but nginx cannot proxy to it
No policy logic lives in the CLI — the CLI only sends data and reads decisions

Part A — The Engine

The Manifest: Single Source of Truth

Everything starts with manifest.yaml:

services:
  image: swift-deploy-1-node:latest
  port: 3000
  mode: stable        # stable | canary
  version: "1.0.0"

nginx:
  image: nginx:latest
  port: 8080
  proxy_timeout: 30

network:
  name: swiftdeploy-net
  driver_type: bridge

This is the only file an operator edits. Every other config file is generated from it. The grader's test is simple: delete nginx.conf and docker-compose.yml, run ./swiftdeploy init, and verify they come back correctly. If your tool breaks, your stack breaks.

The Tool That Writes Its Own Infrastructure

The core of swiftdeploy init is a Python template renderer embedded directly in the bash script:

import yaml
from pathlib import Path

with open("manifest.yaml") as f:
    m = yaml.safe_load(f)

ctx = {
    "service_image":  m["services"]["image"],
    "service_port":   m["services"]["port"],
    "mode":           m["services"].get("mode", "stable"),
    "nginx_port":     m["nginx"]["port"],
    "proxy_timeout":  m["nginx"].get("proxy_timeout", 30),
    "network_name":   m["network"]["name"],
    "network_driver": m["network"]["driver_type"],
}

for tmpl, out in [("templates/nginx.conf.j2", "nginx.conf"),
                  ("templates/docker-compose.yml.j2", "docker-compose.yml")]:
    src = Path(tmpl).read_text()
    for k, v in ctx.items():
        src = src.replace("{{ " + k + " }}", str(v))
    Path(out).write_text(src)

The templates use {{ variable }} placeholders. No Jinja2 library needed — a simple string replace is sufficient and keeps the image dependency-free.

The nginx template enforces the required access log format:

log_format swiftdeploy '$time_iso8601 | $status | ${request_time}s | $upstream_addr | $request';

And returns structured JSON error bodies for 502/503/504:

location @err502 {
    default_type application/json;
    return 502 '{"error":"Bad Gateway","code":502,"service":"app","contact":"ops@swiftdeploy.local"}';
}

The docker-compose template enforces security hardening on every deploy:

user: "1001:1001"
cap_drop:
  - ALL
expose:
  - "{{ service_port }}"   # internal only — never ports:

The API Service

The service is a pure Python stdlib HTTP server — no Flask, no FastAPI, no external dependencies. This keeps the Docker image under 80MB.

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/healthz":
            _json(self, 200, {"status": "ok", "uptime": round(time.time() - START_TIME, 2)})
        elif self.path == "/":
            _json(self, 200, {
                "message": f"Welcome to SwiftDeploy [{MODE}]",
                "mode": MODE,
                "version": VERSION,
                "timestamp": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
            })

Canary mode is controlled entirely by the MODE environment variable injected by docker-compose. The same image runs in both modes — no separate builds.

Chaos state is an in-process global dict protected by a threading.Lock:

_chaos = {"mode": None, "duration": 0, "rate": 0.0}

def _apply_chaos():
    with _chaos_lock:
        m = _chaos["mode"]
        if m == "slow":
            time.sleep(_chaos["duration"])
        elif m == "error":
            if random.random() < _chaos["rate"]:
                return True   # caller should return 500
    return False

The CLI Subcommands

validate runs 5 pre-flight checks before any deployment:

==> Running pre-flight checks
  [PASS] manifest.yaml exists and is valid YAML
  [PASS] All required fields present and non-empty
  [PASS] Docker image 'swift-deploy-1-node:latest' exists locally
  [PASS] Nginx port 8080 is not already bound
  [PASS] nginx.conf is syntactically valid

  Results: 5 passed, 0 failed

The nginx syntax check is interesting — it can't just run nginx -t because the app hostname doesn't resolve outside the compose network. The fix: substitute the upstream hostname with 127.0.0.1 in a temp copy before testing:

sed 's|http://app:|http://127.0.0.1:|g' nginx.conf > .nginx-check-tmp.conf
docker run --rm -v "$(pwd)/.nginx-check-tmp.conf:/etc/nginx/conf.d/default.conf:ro" \
  nginx:latest nginx -t

promote does a rolling restart of only the app container:

docker compose up -d --no-deps --force-recreate app

The --no-deps flag is critical — it prevents docker-compose from restarting nginx, which would cause a brief outage. Only the app restarts; nginx keeps serving traffic (returning 502s for the few seconds the app is down, which is expected and handled by the JSON error pages).

Part B — The Eyes and the Brain

Instrumentation: /metrics in Pure Python

Adding Prometheus metrics to a stdlib HTTP server requires no libraries. All state lives in module-level variables protected by a lock:

_BUCKETS = [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
_req_total = {}       # {(method, path, status_code): count}
_hist_buckets = {le: 0 for le in _BUCKETS}
_hist_sum = 0.0
_hist_count = 0

Every request records its duration and increments the appropriate counters:

def _record(method, path, status, duration):
    key = (method, path, str(status))
    with _metrics_lock:
        _req_total[key] = _req_total.get(key, 0) + 1
        _hist_sum += duration
        _hist_count += 1
        for le in _BUCKETS:
            if duration <= le:
                _hist_buckets[le] += 1
        _hist_inf += 1

The /metrics endpoint serialises this state into Prometheus text format:

# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",path="/",status_code="200"} 42
http_requests_total{method="GET",path="/healthz",status_code="200"} 180
# HELP http_request_duration_seconds Request latency histogram
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.005"} 210
http_request_duration_seconds_bucket{le="0.01"} 220
...
http_request_duration_seconds_bucket{le="+Inf"} 222
http_request_duration_seconds_sum 0.441200
http_request_duration_seconds_count 222
# HELP app_uptime_seconds Seconds since process start
# TYPE app_uptime_seconds gauge
app_uptime_seconds 3600.42
# HELP app_mode Current deployment mode (0=stable 1=canary)
# TYPE app_mode gauge
app_mode 1
# HELP chaos_active Active chaos state (0=none 1=slow 2=error)
# TYPE chaos_active gauge
chaos_active 0

The Policy Sidecar: OPA

OPA runs as a sidecar container. The key design decision is network isolation: OPA lives on opa-net, which has no connection to swiftdeploy-net (where nginx and the app live). The port is bound to 127.0.0.1 only:

opa:
  image: openpolicyagent/opa:latest
  command: ["run", "--server", "--addr", "0.0.0.0:8181", "/policies"]
  volumes:
    - ./policies:/policies:ro
  networks:
    - opa-net
  ports:
    - "127.0.0.1:8181:8181"

This satisfies the "no leakage" requirement: nginx cannot proxy to OPA because they share no network. The CLI on the host can reach OPA directly on localhost:8181, but a request through http://localhost:8080/v1/data/infra would return a 404 from nginx.

The policies/ directory is mounted read-only. OPA loads all .rego files and their companion data.json files at startup. Restarting OPA picks up policy changes without rebuilding anything.

Writing Rego Policies

The core principle: OPA never returns a bare boolean. Every decision carries the reasoning behind it.

Infrastructure Policy (policies/infra/policy.rego):

package infra

import rego.v1

default allow := false

allow if {
    count(deny) == 0
}

deny contains msg if {
    input.disk_free_gb < data.infra.min_disk_gb
    msg := sprintf(
        "Disk free %vGB is below minimum %vGB",
        [input.disk_free_gb, data.infra.min_disk_gb]
    )
}

deny contains msg if {
    input.cpu_load > data.infra.max_cpu_load
    msg := sprintf(
        "CPU load %v exceeds maximum %v",
        [input.cpu_load, data.infra.max_cpu_load]
    )
}

The thresholds live in policies/infra/data.json, not in the Rego file:

{ "min_disk_gb": 10, "max_cpu_load": 2.0 }

This separation matters. Changing a threshold is a data change, not a policy change. You can update data.json and restart OPA without touching the logic. A security audit of the policy file remains valid even after threshold adjustments.

Canary Safety Policy (policies/canary/policy.rego):

package canary

import rego.v1

default allow := false

allow if {
    count(deny) == 0
}

deny contains msg if {
    input.error_rate > data.canary.max_error_rate
    msg := sprintf(
        "Error rate %v%% exceeds maximum %v%%",
        [round(input.error_rate * 100), round(data.canary.max_error_rate * 100)]
    )
}

deny contains msg if {
    input.p99_ms > data.canary.max_p99_ms
    msg := sprintf(
        "P99 latency %vms exceeds maximum %vms",
        [input.p99_ms, data.canary.max_p99_ms]
    )
}

Each domain owns exactly one question. The infra policy only knows about disk and CPU. The canary policy only knows about error rate and latency. A change to one never requires touching the other.

Gated Lifecycle: deploy and promote

Pre-deploy gate collects host stats and sends them to OPA:

disk_free=$(df -g / | awk 'NR==2{print $4}')
cpu_load=$(uptime | awk -F'load averages?:' '{print $2}' | awk '{print $1}' | tr -d ',')

input_json=$(python3 -c "
import json
print(json.dumps({
    'action': 'deploy',
    'disk_free_gb': float('$disk_free'),
    'cpu_load': float('$cpu_load')
}))
")

When the disk is full, the output is unambiguous:

==> Checking infrastructure policy
  disk_free=2GB  cpu_load=0.8
allow=false
  [DENY] Disk free 2GB is below minimum 10GB
  [BLOCKED] Deployment denied by infrastructure policy

Pre-promote gate scrapes /metrics and computes P99 latency from the histogram:

# P99 interpolation from histogram buckets
target = total_obs * 0.99
for le, count in sorted(buckets):
    if count >= target:
        p99_ms = int(le * 1000)
        break

The input to OPA is different from the deploy check — it reflects the question being asked:

{
  "action": "promote",
  "error_rate": 0.052,
  "p99_ms": 620
}

OPA treats pre-deploy and pre-promote as completely separate questions. The CLI queries each domain independently and treats each answer as a separate signal.

OPA unavailability is handled gracefully — the CLI never crashes or hangs:

resp=$(curl -s ... 2>/dev/null) || {
    echo "  [WARN] OPA unreachable — policy check skipped"
    return 2
}

Return code 2 means "unavailable" — the CLI logs a warning and continues. Return code 1 means "denied" — the CLI prints the reasons and exits. Return code 0 means "allowed" — the CLI proceeds.

The Status Dashboard

./swiftdeploy status runs a live-refreshing terminal dashboard that scrapes /metrics every 5 seconds:

==> SwiftDeploy Status  [2026-05-06T20:10:00Z]  mode=canary

  Throughput : 7.3 req/s
  P99 Latency: 620ms
  Error Rate : 6.00%
  Uptime     : 700s
  Chaos      : error

  Policy Compliance:
    [PASS] infra   — disk=107GB cpu=0.8
    [FAIL] canary  — error_rate=6.00% p99=620ms

  (Ctrl+C to exit)

Every scrape appends a JSON record to history.jsonl:

{
  "ts": "2026-05-06T20:10:00Z",
  "mode": "canary",
  "req_per_s": 7.3,
  "p99_ms": 620,
  "error_rate": 0.06,
  "uptime_s": 700,
  "chaos": "error",
  "policy": {"infra": "PASS", "canary": "FAIL"}
}

The Audit Report

./swiftdeploy audit parses history.jsonl and generates audit_report.md — valid GitHub Flavored Markdown with four sections: timeline, violations, mode changes, and chaos events.

==> audit_report.md generated
    4 snapshots | 1 violations | 1 mode changes | 1 chaos events

The Chaos Experiments

Experiment 1: Error Injection

Promote to canary, then inject 50% error rate:

./swiftdeploy promote canary
curl -X POST http://localhost:8080/chaos \
  -H "Content-Type: application/json" \
  -d '{"mode":"error","rate":0.5}'

The status dashboard immediately captures the failure:

  Throughput : 8.1 req/s
  P99 Latency: 45ms
  Error Rate : 48.20%
  Chaos      : error

  Policy Compliance:
    [PASS] infra   — disk=107GB cpu=0.8
    [FAIL] canary  — error_rate=48.20% p99=45ms

Now try to promote to stable — the canary gate blocks it:

==> Checking canary safety policy
  error_rate=48.20%  p99=45ms
allow=false
  [DENY] Error rate 48% exceeds maximum 1%
  [BLOCKED] Promotion denied by canary safety policy

This is the correct behaviour. The canary is unhealthy. You must recover first:

curl -X POST http://localhost:8080/chaos \
  -H "Content-Type: application/json" \
  -d '{"mode":"recover"}'

Wait for the error rate to drop below 1% in the status view, then promote.

Experiment 2: Slow Responses

curl -X POST http://localhost:8080/chaos \
  -H "Content-Type: application/json" \
  -d '{"mode":"slow","duration":3}'

The status dashboard shows P99 climbing:

  P99 Latency: 3000ms
  Error Rate : 0.00%

  Policy Compliance:
    [FAIL] canary  — error_rate=0.00% p99=3000ms

The canary policy blocks promotion on P99 alone — even with zero errors, a 3-second P99 is unacceptable.

Experiment 3: Disk Full Gate

Simulate a full disk by temporarily lowering the threshold in policies/infra/data.json:

{ "min_disk_gb": 200, "max_cpu_load": 2.0 }

Restart OPA and try to deploy:

docker restart swiftdeploy-opa
./swiftdeploy deploy

==> Checking infrastructure policy
  disk_free=107GB  cpu_load=0.8
allow=false
  [DENY] Disk free 107GB is below minimum 200GB
  [BLOCKED] Deployment denied by infrastructure policy

The hard gate works. No containers are started, no configs are written.

Lessons Learned

1. Heredocs and set -euo pipefail don't mix well with pipes.

The biggest source of bugs was set -e killing the script when a pipe's right-hand side returned non-zero (e.g., grep finding no match). The fix: capture output into a variable and use if statements instead of && chains or piped until conditions.

2. macOS and Linux have different system utilities.

df -BG is Linux. df -g is macOS. uptime formats load averages differently. head -n -1 doesn't work on BSD head. Any tool targeting both platforms needs to handle these divergences explicitly.

3. OPA internal: true networks block host port bindings.

Setting internal: true on a Docker network prevents all external connectivity — including host port mappings. OPA needs to be reachable by the CLI on the host, so the network must not be internal. Isolation from nginx is achieved by simply not adding OPA to the nginx-facing network.

4. Separate policy logic from policy data.

Hardcoding thresholds in Rego files is a mistake. When you need to adjust a limit, you want to change a JSON file, not a policy file. Keeping them separate means your Rego logic can be audited and reviewed independently of operational tuning.

5. Never return a bare boolean from a policy engine.

A decision without reasoning is useless to an operator. Every OPA response includes a deny array with human-readable messages. The CLI surfaces these directly. When a deploy is blocked, the operator knows exactly why.

6. The --no-deps flag is essential for rolling restarts.

docker compose up -d --force-recreate app without --no-deps will restart nginx too, causing a full outage. With --no-deps, only the app container restarts. Nginx keeps serving (returning 502s briefly), which is the correct rolling restart behaviour.

Replication Guide

Prerequisites

Docker + Docker Compose v2
Python 3.10+ with pyyaml (pip install pyyaml)
curl, lsof

Step 1: Clone and build

git clone <your-repo-url>
cd stage4
chmod +x swiftdeploy
docker build -t swift-deploy-1-node:latest .

Step 2: Init and validate

./swiftdeploy init
./swiftdeploy validate

Expected: 5/5 checks pass.

Step 3: Deploy

./swiftdeploy deploy

Expected output:

==> Checking infrastructure policy
  disk_free=107GB  cpu_load=0.8
allow=true
  [PASS] Infrastructure policy
==> Initialising
  generated nginx.conf
  generated docker-compose.yml
==> Starting stack
  [+] Running 4/4
==> Waiting for health (timeout 60s)
  [OK] Stack is healthy

Step 4: Verify endpoints

curl http://localhost:8080/
curl http://localhost:8080/healthz
curl http://localhost:8080/metrics

Step 5: Promote to canary

./swiftdeploy promote canary

Verify X-Mode: canary header:

curl -sv http://localhost:8080/ 2>&1 | grep X-Mode

Step 6: Inject chaos and watch the status dashboard

In one terminal:

./swiftdeploy status

In another:

curl -X POST http://localhost:8080/chaos \
  -H "Content-Type: application/json" \
  -d '{"mode":"error","rate":0.5}'

Watch the canary policy flip to [FAIL] in the dashboard.

Step 7: Recover and generate audit report

curl -X POST http://localhost:8080/chaos \
  -H "Content-Type: application/json" \
  -d '{"mode":"recover"}'

./swiftdeploy audit
cat audit_report.md

Step 8: Teardown

./swiftdeploy teardown --clean

Built for HNG DevOps Track — Stage 4.

How I Built a Real-Time DDoS Detection Engine from Scratch

instanceofGod — Sun, 26 Apr 2026 23:43:18 +0000

Introduction

Imagine you run a cloud storage platform. Thousands of users upload files, share documents, and collaborate every day. Then one morning, a single IP address sends 500 requests in 60 seconds. Your server slows to a crawl. Users can't log in. Files won't upload. You're under attack.

This is a DDoS attack — Distributed Denial of Service. The goal is simple: flood your server with so many requests that it can't serve real users anymore.

In this post, I'll walk you through how I built an anomaly detection engine that watches all incoming HTTP traffic in real time, learns what normal looks like, and automatically blocks attackers — all without any third-party security libraries.

Here's what the system does:

Reads Nginx access logs line by line as they are written
Tracks request rates using sliding time windows
Learns normal traffic patterns using a rolling statistical baseline
Detects anomalies using z-score math
Blocks attacking IPs using Linux firewall rules (iptables)
Sends Slack alerts within 10 seconds
Shows a live web dashboard

Let's break down each piece.

The Architecture

Before diving into code, here's how all the pieces connect:

Internet → Nginx (reverse proxy) → Nextcloud
               ↓ writes JSON logs
          /var/log/nginx/hng-access.log
               ↓ reads logs
          Detector Daemon (Python)
          ├── monitor.py   — tails the log file
          ├── baseline.py  — learns what normal looks like
          ├── detector.py  — spots anomalies
          ├── blocker.py   — blocks IPs with iptables
          ├── unbanner.py  — auto-releases bans
          ├── notifier.py  — sends Slack alerts
          └── dashboard.py — live web UI

Nginx sits in front of Nextcloud and writes every HTTP request as a JSON line to a log file. The detector daemon tails that file continuously, processes each line, and acts when something looks wrong.

Everything runs in Docker. Nginx, Nextcloud, and the detector are three separate containers sharing a named volume so the log file is accessible to all of them.

Step 1 — Reading the Log File in Real Time

The first challenge is reading a log file as it grows. This is called "tailing" — like running tail -f in your terminal.

Here's the core idea:

f = open("/var/log/nginx/hng-access.log", "r")
f.seek(0, 2)  # jump to the end of the file

while True:
    line = f.readline()
    if line:
        process(line)
    else:
        time.sleep(0.05)  # nothing new yet, wait 50ms and try again

f.seek(0, 2) moves the file cursor to the very end. This means we only see new lines written after the daemon starts — not the entire history.

readline() returns an empty string "" when there's nothing new. That's our signal to sleep briefly and retry.

Handling log rotation: Nginx periodically rotates its log file (renames the old one, creates a new one). If we don't handle this, we'd keep reading the old file forever. The fix is to compare the file's inode (a unique ID the OS assigns to every file) on each empty read:

current_inode = os.stat(log_path).st_ino
if current_inode != saved_inode:
    f.close()
    f = open(log_path, "r")  # reopen the new file from the beginning
    saved_inode = current_inode

Each parsed JSON line gives us: source_ip, timestamp, method, path, status, response_size.

Step 2 — The Sliding Window

Now that we're reading log lines, we need to answer one question per request:

How many requests has this IP sent in the last 60 seconds?

A naive approach would be to store a counter per IP and reset it every minute. But that's wrong — it doesn't give you a true 60-second window. If an attacker sends 499 requests at 11:59:59 and 1 request at 12:00:01, a per-minute counter resets and misses the spike.

The correct approach is a sliding window using a deque (double-ended queue).

Here's the idea:

from collections import deque
import time

ip_windows = {}  # one deque per IP

def record_request(ip):
    now = time.time()
    if ip not in ip_windows:
        ip_windows[ip] = deque()

    # add this request's timestamp to the right
    ip_windows[ip].append(now)

    # evict timestamps older than 60 seconds from the left
    cutoff = now - 60
    while ip_windows[ip] and ip_windows[ip][0] < cutoff:
        ip_windows[ip].popleft()

    # the length is now the exact request count in the last 60 seconds
    return len(ip_windows[ip])

Visually, the deque looks like this:

ip_windows["1.2.3.4"] = deque([
    1714000001.1,   ← oldest (left side)
    1714000001.9,
    1714000002.3,
    1714000059.8    ← newest (right side)
])

As time passes, old timestamps fall off the left. New ones are added to the right. len(deque) always gives you the exact count for the last 60 seconds.

Why deque and not a regular list? Because deque is O(1) on both ends — appending to the right and removing from the left are both instant, regardless of how many items are in it. A regular list's pop(0) is O(n) — it shifts every element left, which gets slow under heavy traffic.

We maintain three sets of windows:

One deque per IP (per-IP request rate)
One global deque (total traffic rate across all IPs)
One error deque per IP (only 4xx/5xx responses)

Step 3 — The Rolling Baseline

The sliding window tells us the current rate. But is that rate normal or abnormal? To answer that, we need to know what normal looks like — and that changes over time. Traffic at 3am is different from traffic at 3pm.

This is the rolling baseline: a statistical model of recent traffic that updates automatically.

How it works:

Every second, we count how many requests arrived in that second and store it as a bucket:

second_bucket = int(time.time())
counts[second_bucket] += 1

Every 60 seconds, we look at the last 30 minutes of these per-second counts (up to 1800 buckets) and compute:

mean   = sum(counts) / len(counts)
stddev = sqrt(sum((c - mean)**2 for c in counts) / len(counts))

mean is the average requests per second over the last 30 minutes. stddev (standard deviation) measures how much the traffic varies — a low stddev means traffic is steady, a high stddev means it's spiky.

Per-hour slots: We also store counts grouped by UTC hour. If the current hour has 60 or more data points, we prefer it over the mixed 30-minute window. This makes the baseline more accurate — 2am traffic shouldn't influence the 2pm baseline.

Floor values: On a quiet server, mean and stddev could be very close to zero. If stddev is 0, dividing by it in the z-score formula would crash the program. So we apply a floor:

mean   = max(computed_mean,   1.0)
stddev = max(computed_stddev, 0.5)

This ensures the math always works, even on a server with almost no traffic.

Step 4 — Detecting Anomalies

Now we have:

current_rate — requests from this IP in the last 60 seconds
mean — average requests per second over the last 30 minutes
stddev — how much traffic normally varies

We use z-score to decide if the current rate is abnormal:

z = (current_rate - mean) / stddev

The z-score tells you how many standard deviations above normal the current rate is. A z-score of 1.0 means slightly above average. A z-score of 3.0 means extremely unusual — statistically, only 0.3% of normal traffic would ever reach this level.

We flag an IP as anomalous if either condition fires:

z = (ip_rate - mean) / stddev
if z > 3.0 or ip_rate > 5 * mean:
    # anomaly detected

The 5x mean rule is a safety net — if traffic is so extreme that even a high stddev wouldn't catch it, the multiplier rule fires first.

Error surge tightening: If an IP is also generating a lot of 4xx/5xx errors (failed login attempts, scanning for vulnerabilities), we tighten the detection threshold from 3.0 down to 2.0. This means we act faster on IPs that are both high-volume and generating errors.

The same logic applies globally — if total traffic across all IPs spikes, we send a Slack alert (but don't block a single IP since the attack may be distributed).

Step 5 — Blocking with iptables

When an IP is flagged as anomalous, we block it at the Linux kernel level using iptables.

iptables is Linux's built-in firewall. It processes every network packet before it reaches your application. A DROP rule tells the kernel to silently discard all packets from a specific IP — they never reach Nginx, never reach Nextcloud, never consume any application resources.

The command to block an IP:

iptables -I INPUT -s 1.2.3.4 -j DROP

Breaking this down:

-I INPUT — insert a rule into the INPUT chain (incoming traffic)
-s 1.2.3.4 — match packets from this source IP
-j DROP — silently drop them (no response sent to the attacker)

In Python, we run this as a subprocess:

import subprocess

subprocess.run(
    ["iptables", "-I", "INPUT", "-s", ip, "-j", "DROP"],
    check=True,
    capture_output=True,
)

We use a list (not a shell string) to avoid shell injection vulnerabilities. check=True raises an exception if the command fails so we can log it.

To unban:

iptables -D INPUT -s 1.2.3.4 -j DROP

-D deletes the rule instead of inserting it.

Step 6 — Auto-Unban with Backoff

We don't ban IPs forever (unless they keep attacking). The unban schedule uses exponential backoff:

Offense	Ban Duration
1st	10 minutes
2nd	30 minutes
3rd	2 hours
4th+	Permanent

A background thread checks every 30 seconds whether any ban has expired:

for ip, record in banned_ips.items():
    elapsed_minutes = (now - record["banned_at"]) / 60
    if elapsed_minutes >= record["duration_minutes"]:
        unban(ip)
        send_slack_unban_alert(ip)

The offense count persists across ban/unban cycles. So if an IP gets banned, serves its 10-minute ban, gets released, and attacks again — the next ban is 30 minutes.

Step 7 — Slack Alerts

Every ban, unban, and global anomaly sends a Slack message via webhook. The webhook is just an HTTPS POST to a URL Slack gives you:

import requests

requests.post(
    webhook_url,
    json={"text": ":rotating_light: *IP BANNED* ..."},
    timeout=8,
)

timeout=8 ensures we complete within the 10-second requirement even if Slack is slow. The POST runs in a background thread so it never blocks the detector loop.

A ban alert looks like this in Slack:

🚨 IP BANNED
• IP: 203.0.113.42
• Condition: z=4.21
• Current rate: 312 req/60s
• Baseline mean: 8.3421 | stddev: 2.1500
• Ban duration: 10 min
• Timestamp: 2025-04-20 14:32:01 UTC

Step 8 — The Live Dashboard

The dashboard is a FastAPI web app with two endpoints:

GET /metrics — returns live JSON data
GET /— returns an HTML page that polls /metrics every 3 seconds

The HTML page uses plain JavaScript to fetch /metrics and update the DOM without reloading the page:

async function refresh() {
    const r = await fetch('/metrics');
    const d = await r.json();
    document.getElementById('global-rate').textContent = d.global_req_per_60s;
    // ... update other elements
}
setInterval(refresh, 3000);

The dashboard shows:

Banned IPs with ban time, duration, and offense count
Global requests per 60 seconds
Top 10 source IPs
CPU and memory usage
Effective baseline mean and stddev
Daemon uptime

Putting It All Together

All the modules run as threads inside a single Python process:

main.py
├── Thread: monitor     → tails log, pushes events to queue
├── Thread: detector    → consumes queue, runs detection
├── Thread: baseline    → recalculates every 60 seconds
├── Thread: unbanner    → checks ban expiry every 30 seconds
└── Thread: dashboard   → uvicorn serving FastAPI on port 8080

The main thread just keeps the process alive with while True: sleep(1) and handles KeyboardInterrupt for clean shutdown.

All shared state (banned IPs, sliding windows, baseline values) is protected by threading.Lock() to prevent race conditions between threads.

Key Takeaways

Sliding windows with deques give you exact per-second rate tracking with O(1) performance
Rolling baselines let the system adapt to real traffic patterns instead of relying on hardcoded thresholds
Z-score detection is statistically sound — it flags things that are genuinely unusual relative to recent history
iptables blocks at the kernel level — the most efficient place to stop an attack
Backoff unbanning balances security with fairness — short bans for first offenses, longer for repeat attackers

The full source code is available at: https://github.com/nielvid/anomaly-detector

Setup & Walkthrough for Beginners

Now that you understand how the system works, let's get it running on your own machine. This step-by-step guide assumes you're on Ubuntu 22.04 or 24.04 (the instructions work on most Linux distributions with minor adjustments).

Prerequisites

Before starting, make sure you have:

A Linux machine (physical, VM, or cloud instance like DigitalOcean, AWS EC2, or Linode)
Root or sudo access (iptables commands require privileges)
Docker and Docker Compose installed

If Docker isn't installed yet, run:

sudo apt update
sudo apt install -y docker.io docker-compose
sudo systemctl enable --now docker
sudo usermod -aG docker $USER
# Log out and back in for group changes to take effect

Step 1: Clone the Repository

git clone https://github.com/nielvid/anomaly-detector.git
cd anomaly-detector

Step 2: Configure Nginx to Write JSON Logs

Nginx needs to output logs in JSON format so the Python detector can parse them easily. Create or edit /etc/nginx/nginx.conf and add a custom log format:

sudo nano /etc/nginx/nginx.conf

Inside the http block, add:

http {
    log_format json escape=json '{'
        '"timestamp":"$time_iso8601",'
        '"source_ip":"$remote_addr",'
        '"method":"$request_method",'
        '"path":"$request_uri",'
        '"status":$status,'
        '"response_size":$body_bytes_sent'
    '}';

    access_log /var/log/nginx/hng-access.log json;
}

Then restart Nginx:

sudo systemctl restart nginx

Verify it's working: Generate some traffic by visiting your server's IP in a browser, then run:

sudo tail -n 2 /var/log/nginx/hng-access.log

You should see JSON lines like:

{"timestamp":"2025-04-20T14:32:01+00:00","source_ip":"192.168.1.100","method":"GET","path":"/","status":200,"response_size":1234}

Step 3: Set Up the Python Environment

The detector runs outside Docker for simplicity in this beginner walkthrough. Install Python dependencies:

cd ~/anomaly-detector
python3 -m venv venv
source venv/bin/activate
pip install fastapi uvicorn requests psutil

Step 4: Configure Slack Alerts (Optional But Recommended)

Go to https://api.slack.com/apps
Click "Create New App" → "From Scratch"
Name it "Anomaly Detector" and choose your workspace
In the left sidebar, click "Incoming Webhooks"
Toggle "Activate Incoming Webhooks" to On
Click "Add New Webhook to Workspace"
Choose a channel (like #alerts or #security)
Copy the webhook URL (looks like https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX)

Create a config file:

nano ~/anomaly-detector/config.py

Add this content (replace the webhook URL with yours):

SLACK_WEBHOOK_URL = "https://hooks.slack.com/services/YOUR/URL/HERE"
DETECTION_WINDOW_SECONDS = 60
BASELINE_MINUTES = 30
ALERT_COOLDOWN_SECONDS = 300  # Don't spam the same alert

Step 5: Run the Detector

First, make sure you have write access to the log file (or run with sudo):

sudo chmod 644 /var/log/nginx/hng-access.log

Now start the detector:

cd ~/anomaly-detector
source venv/bin/activate
sudo python3 main.py   # sudo needed for iptables

You should see output like:

[2025-04-20 14:30:01] Monitor thread started, tailing /var/log/nginx/hng-access.log
[2025-04-20 14:30:01] Baseline thread started
[2025-04-20 14:30:01] Dashboard running at http://0.0.0.0:8080
[2025-04-20 14:30:01] Unbanner thread started

Step 6: Generate Test Traffic to See Detection in Action

Open a second terminal and use curl to simulate an attack:

# Normal traffic (won't trigger a ban)
for i in {1..10}; do curl -s http://localhost/ > /dev/null; sleep 1; done

# Simulated DDoS attack (will trigger a ban)
for i in {1..500}; do curl -s http://localhost/ > /dev/null & done
wait

Within 60 seconds, the detector will:

Notice the spike from your IP
Calculate a high z-score (likely > 10)
Run iptables -I INPUT -s YOUR_IP -j DROP
Send a Slack alert (if configured)
Show the ban in the dashboard

Step 7: View the Live Dashboard

Open a web browser and navigate to:

http://YOUR_SERVER_IP:8080

You'll see real-time metrics including:

Global request rate
Top talking IPs
Currently banned IPs with remaining time
Current baseline (mean and stddev)

Step 8: Verify iptables Block Is Working

In the terminal, check if your IP is blocked:

sudo iptables -L INPUT -n -v

You should see a line like:

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target     prot opt in     out     source               destination
  15  1260 DROP       all  --  *      *       YOUR_IP              0.0.0.0/0

To test that traffic is actually dropped, try curling from the blocked IP — the command will hang until it times out.

Step 9: Test Auto-Unban

Wait 10 minutes (or modify the ban duration in the code to 1 minute for testing). The unbanner thread will automatically remove the ban:

# After ban expires, check iptables again
sudo iptables -L INPUT -n -v
# The DROP rule should be gone

Step 10: Run as a System Service (Production)

To keep the detector running after you log out, create a systemd service:

sudo nano /etc/systemd/system/anomaly-detector.service

Add this content:

[Unit]
Description=Anomaly Detection Engine
After=network.target nginx.service

[Service]
Type=simple
User=root
WorkingDirectory=/home/YOUR_USERNAME/anomaly-detector
ExecStart=/home/YOUR_USERNAME/anomaly-detector/venv/bin/python3 /home/YOUR_USERNAME/anomaly-detector/main.py
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Enable and start it:

sudo systemctl daemon-reload
sudo systemctl enable anomaly-detector
sudo systemctl start anomaly-detector
sudo systemctl status anomaly-detector

Common Troubleshooting

Problem: Permission denied when opening log file

sudo chmod 644 /var/log/nginx/hng-access.log
# Or run the detector with sudo
sudo python3 main.py

Problem: iptables: Command not found

sudo apt install iptables

Problem: Detector can't see new log lines

# Check if log rotation is moving the file
ls -la /var/log/nginx/
# The detector should auto-detect inode changes

Problem: Dashboard shows 0 requests

Generate some traffic first
Verify the log file path in main.py matches your Nginx log location
Check that Nginx is actually writing JSON format

Problem: Too many false positives

Increase the z-score threshold from 3.0 to 4.0
Increase baseline minutes from 30 to 60
Whitelist known good IPs in the detector code

Testing Locally Without a Real Server

If you don't have a Linux server handy, you can test everything in a Docker container:

# Create a test Docker container with Ubuntu
docker run -it --rm --cap-add=NET_ADMIN ubuntu:22.04 bash

# Inside the container:
apt update && apt install -y iptables nginx python3 python3-pip
# Then follow steps 2-6 above

The --cap-add=NET_ADMIN is required for iptables to work inside Docker.

Next Steps & Customization

Once you have the basic system running, here are ways to extend it:

Add IP whitelisting — Prevent your office or trusted partners from being banned
Persist bans across restarts — Save banned IPs to a SQLite database
Add rate limiting per path — Block IPs that hammer /login but allow /static
Integrate with fail2ban — Use the detector as a faster, smarter rules engine
Add geolocation — Block or alert on traffic from unexpected countries

The full source code with all modules is available at: https://github.com/nielvid/anomaly-detector

Built as part of the HNG DevOps Track — Stage 3

PIN Encryption & Decryption with RSA in Node.js

instanceofGod — Tue, 17 Mar 2026 16:41:57 +0000

Why PIN encryption matters
Core terms defined
How RSA works
Anatomy of a key pair
The OpenSSLUtil class, line by line
The full encrypt → transmit → decrypt flow
Working with keys in the terminal
Common pitfalls & security notes
Conclusion

1. Why PIN encryption matters

Every time a user taps their 4-digit PIN into a mobile banking app or payment terminal, those four digits travel from device to server. If they travel as plain text — say, {"pin": "1234"} — anyone who can intercept the HTTP request, peek at server logs, or read a database dump will see that PIN immediately. That is a catastrophic failure mode.

The solution is asymmetric encryption. The server holds a private key that it never shares. The client has a copy of the corresponding public key. The client encrypts the PIN before it leaves the device, and only the server — with its private key — can decrypt it. No intermediary, no log, no network sniffer can read the PIN in transit.

The code snippet in this article is a production-ready implementation of this pattern, built on Node.js's built-in crypto module using RSA-OAEP encryption with SHA-256 — the modern, recommended approach. This article walks through every part of it, explaining the concepts before the code.

2. Core terms defined

Before touching a single line of code, you need a working vocabulary. These terms will come up again and again.

Term	Definition
Plaintext	The original readable data before any encryption. In our case, the PIN — e.g. the string `"1234"`.
Ciphertext	The scrambled output after encryption. Looks like random bytes. Example: a 256-byte binary blob that encodes to a long base64 string.
Encryption	The process of transforming plaintext into ciphertext using a key and an algorithm. Without the correct decryption key, ciphertext is meaningless.
Decryption	The reverse: transforming ciphertext back into plaintext using the correct key.
RSA	Rivest–Shamir–Adleman. A widely used asymmetric encryption algorithm based on the mathematical difficulty of factoring very large numbers.
Asymmetric encryption	A scheme that uses two different keys: a public key to encrypt, and a private key to decrypt. You can freely share the public key — knowing it does not help you decrypt.
Symmetric encryption	A scheme that uses the same key for both encryption and decryption (e.g. AES). Fast, but the key must be shared securely — which is the hard part.
Public key	One half of the RSA key pair. Safe to share with anyone. Used to encrypt data or verify signatures.
Private key	The other half. Must be kept secret. Used to decrypt data or create signatures. If an attacker gets this, all security is lost.
OAEP padding	Optimal Asymmetric Encryption Padding. The modern, secure padding scheme for RSA. Uses a hash function (SHA-256 in this codebase) to produce randomised, tamper-resistant ciphertext. Resistant to padding oracle attacks.
PKCS#8	A standard format for encoding private keys. Node.js v17+ requires private keys to be in PKCS#8 format (begins with `-----BEGIN PRIVATE KEY-----`).
PEM format	Privacy Enhanced Mail. A base64-encoded wrapper for certificates and keys with a human-readable header and footer. The `-----BEGIN ...-----` / `-----END ...-----` lines you see in key files.
Base64	An encoding (not encryption!) that converts binary data into printable ASCII characters. Used to safely include binary ciphertext in JSON or URL parameters.
Digital signature	A cryptographic proof that a message was created by the holder of a private key and has not been tampered with. Created with the private key; verified with the public key.
Buffer	Node.js's representation of raw binary data. Crypto operations work on Buffers, not JavaScript strings.
SHA-256	A cryptographic hash function that produces a fixed 256-bit digest of any input. Used both in OAEP padding and in the signing algorithm.

3. How RSA works

You do not need to understand the mathematics in depth, but a conceptual model will prevent a lot of confusion later.

RSA relies on a simple fact: multiplying two large prime numbers together is easy, but factoring the result back into those two primes is computationally infeasible. The public and private keys are derived from this relationship — one cannot be practically computed from the other.

Encryption with the public key:

Anyone can encrypt a message
Freely distribute your public key
OAEP padding makes output non-deterministic
Only the private key holder can read it

Decryption with the private key:

Only the key holder can decrypt
Private key never leaves the server
Decryption always yields the same original plaintext
Losing the private key = losing access

A common analogy: the public key is like an open padlock you hand to everyone. Anyone can snap it shut (encrypt). Only you have the physical key that opens it (decrypt).

Note — RSA is not for bulk data

RSA can only encrypt data smaller than its key size minus padding overhead. A 2048-bit key with OAEP-SHA256 can encrypt at most ~190 bytes of data. This is perfect for short secrets like PINs, passwords, or symmetric keys — but you would never RSA-encrypt a file. For large data, RSA is used to encrypt a random AES key, then AES encrypts the actual data. This hybrid approach is what TLS does.

4. Anatomy of a key pair

The example gives you a 2048-bit RSA key pair. Let's understand what you are looking at.

The public key

Safe to share, embed in your mobile app, or publish in your documentation. It begins with:

-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnFBPODvVUdzXxGkXG0OP
HQ0zDS3xf1uD7tY8mJKWrjSLMf2CeZtWYNlT2xq40R7x5VTn1rnKbgnBZg2ynyLh...
-----END PUBLIC KEY-----

This is PEM format — the binary key data encoded in base64, wrapped with header and footer lines. The actual key material is the base64 between the markers.

The private key (PKCS#8)

Never share this. It begins with -----BEGIN PRIVATE KEY----- (not BEGIN RSA PRIVATE KEY, which is the older PKCS#1 format). Node.js v17+ requires PKCS#8. This is why the setup guide includes a conversion command:

# Convert PKCS#1 to PKCS#8 (Node.js v17+ requires this)
openssl pkcs8 -topk8 -nocrypt -in old_private.pem -out new_private_pkcs8.pem

Key format summary

Header line	Format	Used for
`BEGIN RSA PRIVATE KEY`	PKCS#1	Older OpenSSL tools. Not accepted by Node.js v17+.
`BEGIN PRIVATE KEY`	PKCS#8	Modern standard. Required by Node.js `createPrivateKey`.
`BEGIN ENCRYPTED PRIVATE KEY`	PKCS#8 encrypted	Private key protected with a passphrase.
`BEGIN PUBLIC KEY`	X.509 SubjectPublicKeyInfo	Standard public key format.

5. The OpenSSLUtil class, line by line

Now let's walk through the implementation. The class has three main responsibilities: loading the key pair, encrypting with the public key, and decrypting with the private key. There is also a bonus signing method.

Constructor and key loading

class OpenSSLUtil {
  constructor() {
    this.keyPair = this._readKeyPair();
    // Key parsing happens once at startup — not on every request.
  }

  _readKeyPair() {
    const { privateKey: privatePem, passphrase } = config.rsaConfig;

    const privateKey = crypto.createPrivateKey({
      key: privatePem,
      format: 'pem',
      ...(passphrase && { passphrase })
    });

    // Derive the public key from the private key — no need to store both
    const publicKey = crypto.createPublicKey(privateKey);

    return { privateKey, publicKey };
  }
}

const redisClient = //redisClient;

const TIMESTAMP_DRIFT_MS = +/30 // 30secons;
const REPLAY_TTL_SECS = 60; // 2× the drift window
const REPLAY_CACHE_PREFIX = '';

Why derive the public key from the private key?

A private key mathematically contains all the information needed to reconstruct the public key. Calling crypto.createPublicKey(privateKey) is safe, convenient, and avoids having to store or sync two separate key files. You only need to protect the private key.

Encrypt method — OAEP-SHA256

encrypt(text) {
  try {
    const encrypted = crypto.publicEncrypt(
      {
        key: this.keyPair.publicKey,
        padding: crypto.constants.RSA_PKCS1_OAEP_PADDING,
        oaepHash: 'sha256'
        // OAEP with SHA-256: secure, modern, tamper-resistant
      },
      Buffer.from(text)  // convert string → binary Buffer
    );
    return encrypted.toString('base64');  // binary → base64 string
  } catch (e) {
    logger.error('Could not encrypt data: ', e);
    return null;
  }
}

 /**
   * Encrypt a PIN for secure transmission.
   *
   * Produces a base64-encoded RSA-OAEP ciphertext of "PIN|UTC_TIMESTAMP_MS".
   * The embedded timestamp lets decryptPin enforce a drift window and prevent
   * replay attacks.
   *
   * Returns the base64 ciphertext on success, or null on any failure.
   */
  encryptPin(pin) {
    try {
      const payload = `${pin}|${Date.now()}`;
      const encrypted = crypto.publicEncrypt(
        { key: this.keyPair.publicKey, padding: crypto.constants.RSA_PKCS1_OAEP_PADDING, oaepHash: 'sha256' },
        Buffer.from(payload)
      );
      return encrypted.toString('base64');
    } catch (e) {
      logger.error('Could not encrypt PIN: ', e);
      return null;
    }
  }

Breaking this down step by step:

Buffer.from(text) — Converts the string "1234" into raw bytes. RSA encryption operates on binary data, not JavaScript strings.
RSA_PKCS1_OAEP_PADDING — Applies OAEP padding using SHA-256 as the hash function. OAEP uses a random seed internally, so encrypting the same PIN twice produces completely different ciphertexts — making brute-force and rainbow-table attacks impractical.
oaepHash: 'sha256' — Specifies SHA-256 as the hash used inside the OAEP padding scheme. This must match on both encrypt and decrypt.
.toString('base64') — Encodes the binary output as a base64 string, making it safe to include in JSON request bodies or HTTP headers.

Decrypt method — OAEP-SHA256

decrypt(encrypted) {
  try {
    // Accept either a base64 string or a raw Buffer
    const buffer = typeof encrypted === 'string'
      ? Buffer.from(encrypted, 'base64')  // base64 string → binary Buffer
      : encrypted;                         // already a Buffer, use as-is

    const decrypted = crypto.privateDecrypt(
      {
        key: this.keyPair.privateKey,
        padding: crypto.constants.RSA_PKCS1_OAEP_PADDING,
        oaepHash: 'sha256'  // must match what was used in encrypt()
      },
      buffer
    );
    return decrypted.toString('utf8');  // binary → original string
  } catch (e) {
    logger.error('Could not decrypt data: ', e);
    return null;
  }
}


 /**
   * Decrypt a PIN payload encrypted as "PIN|UTC_TIMESTAMP_MS".
   * The client encrypts PIN|UTC_TIMESTAMP_MS (e.g. 1234|1744549200000) rather than just the PIN.
   * Guards:
   *  1. Replay — the ciphertext fingerprint is stored in Redis for 60 s after
   *     first use; a second submission of the same blob is rejected.
   *  2. Drift window — the embedded timestamp must be within ±30 s of the
   *     server clock to block pre-captured tokens.
   *
   * Returns the PIN string on success, or null on any failure.
   */
  async decryptPin(encrypted) {
    try {
      const buffer = typeof encrypted === 'string'
        ? Buffer.from(encrypted, 'base64')
        : Buffer.from(encrypted);

      const fingerprint = crypto.createHash('sha256').update(buffer).digest('hex');
      const cacheKey = REPLAY_CACHE_PREFIX + fingerprint;

      if (!redisClient.isReady) {
        await redisClient.connect();
      }

      const seen = await redisClient.get(cacheKey);
      if (seen) {
        logger.error('Replay attack detected: encrypted PIN payload reused');
        return null;
      }

      const decrypted = crypto.privateDecrypt(
        { key: this.keyPair.privateKey, padding: crypto.constants.RSA_PKCS1_OAEP_PADDING, oaepHash: 'sha256' },
        buffer
      ).toString('utf8');

      const sepIndex = decrypted.lastIndexOf('|');
      if (sepIndex === -1) {
        logger.error('Encrypted PIN payload is missing timestamp separator');
        return null;
      }

      const pin = decrypted.slice(0, sepIndex);
      const ts = Number(decrypted.slice(sepIndex + 1));

      if (!Number.isFinite(ts) || ts <= 0) {
        logger.error('Encrypted PIN payload contains an invalid timestamp');
        return null;
      }

      const drift = Math.abs(Date.now() - ts);
      if (drift > TIMESTAMP_DRIFT_MS) {
        logger.error(`Encrypted PIN timestamp outside drift window: drift=${drift}ms, limit=${TIMESTAMP_DRIFT_MS}ms`);
        return null;
      }

      // Mark ciphertext as consumed for the replay window
      await redisClient.setEx(cacheKey, REPLAY_TTL_SECS, '1');

      return pin;
    } catch (e) {
      logger.error('Could not decrypt PIN: ', e);
      return null;
    }
  }

Warning — Padding scheme and hash must match on both ends

Both encrypt() and decrypt() use RSA_PKCS1_OAEP_PADDING with oaepHash: 'sha256'. If you encrypt with one hash and attempt to decrypt with another, decryption will silently fail and return null. Always keep these parameters consistent across your entire stack — including any client-side or terminal-based encryption.

Sign method

Signing is separate from encryption. A signature does not hide the content — it proves authorship and integrity. You sign with your private key; anyone with your public key can verify the signature.

sign(text) {
  try {
    const sign = crypto.createSign('SHA256');
    // 1. Feed the data into the sign object
    sign.update(text, 'utf8');
    sign.end();
    // 2. Finalize: hash the data with SHA-256, then RSA-sign the hash
    return sign.sign(this.keyPair.privateKey, 'base64');
  } catch (e) {
    logger.error('Could not sign data: ', e);
    return null;
  }
}

The signing algorithm does two things internally: it first runs SHA256(text) to produce a 32-byte digest, then RSA-encrypts that digest with the private key. The result is a 256-byte (2048-bit) signature, returned as base64.

Module export as singleton

// Export a single shared instance (not the class itself)
module.exports = new OpenSSLUtil();

// Usage elsewhere in the codebase:
const openssl = require('./utils/openssl-util');
const encryptedPin = openssl.encrypt('1234');
const recoveredPin = openssl.decrypt(encryptedPin);

Exporting new OpenSSLUtil() rather than the class itself means the key parsing happens once at startup. Every module that requires this file gets the same instance with the already-loaded keys — no repeated I/O or parsing overhead per request.

6. The full encrypt → transmit → decrypt flow

Here is the complete lifecycle of a PIN from a client device to a server response, as the codebase implements it.

Client has public key
        ↓
User enters PIN
        ↓
Encrypt with public key (OAEP-SHA256)
        ↓
Base64 encode
        ↓
HTTPS POST
        ↓
Server decrypts with private key
        ↓
Plaintext PIN

Step 1 — Client encrypts the PIN (terminal simulation)

# Encrypt "1234" using OAEP with SHA-256 and output as base64
echo -n "1234" | openssl pkeyutl -encrypt -pubin -inkey public.pem \
    -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 | base64

# Encrypt a PIN with embedded timestamp to enforce drift window and prevent replay attacks
# Produces a base64-encoded RSA-OAEP ciphertext of "PIN|UTC_TIMESTAMP_MS".
echo -n "0008|$(python3 -c "import time; print(int(time.time() * 1000))")" | openssl pkeyutl -encrypt -pubin -inkey public_pkcs8.pem -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 | base64

echo -n "0008|$(node -e "process.stdout.write(String(Date.now()))")" | openssl pkeyutl -encrypt -pubin -inkey public_pkcs8.pem -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 | base64


# Output:
mK9xZq2Tp7fVnBsLuR4dAe8cWyOhXiGj...
# Run again — you'll get a completely different string (OAEP is non-deterministic)

Step 2 — Client sends the encrypted PIN

{
  "pin": "mK9xZq2Tp7fVnBsLuR4dAe8cWyOhXiGj..."
}

Why is this safe even over plain HTTP?

Even if an attacker intercepts this JSON, they see only the base64 ciphertext — not the PIN. Without the private key (which never leaves the server), they cannot decrypt it. That said, you should still use HTTPS. Defence in depth means multiple layers of protection.

Step 3 — Server decrypts and verifies

const openssl = require('../../utils/openssl-util');

router.post('/', async (req, res) => {
  const { username, pin } = req.body;

  // Decrypt the incoming base64 string back to plaintext
  const decryptedPin = openssl.decrypt(pin);

  if (!decryptedPin) {
    return res.status(400).json({ error: 'Invalid encrypted PIN' });
  }

  // Never store the raw PIN — hash it before comparison
  const pinHash = crypto
    .createHash('sha256')
    .update(decryptedPin)
    .digest('hex');

  // Compare hash against what is stored in the database
  const user = await findUserByPhone(phonenumber);
  if (!user || user.pinHash !== pinHash) {
    return res.status(401).json({ error: 'Invalid credentials' });
  }

  const token = generateSessionToken(user.id);
  res.json({ token });
});

7. Working with keys in the terminal

The provided setup guide includes several OpenSSL terminal commands. Here is what each one does and when to use it.

Generate a fresh key pair

# Generate a 2048-bit RSA private key and immediately convert to PKCS#8
openssl genrsa 2048 | openssl pkcs8 -topk8 -nocrypt -out private_pkcs8.pem

# Derive the public key from the private key
openssl rsa -in private_pkcs8.pem -pubout -out public.pem

Encrypt a PIN for testing (OAEP)

# Encrypt "1234" using OAEP with SHA-256 and output as base64
echo -n "1234" | openssl pkeyutl -encrypt -pubin -inkey public.pem \
    -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 | base64

# Verify round-trip: decrypt the output back to the original PIN
echo "mK9xZq2..." | base64 -d | openssl pkeyutl -decrypt \
    -inkey private_pkcs8.pem -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256
# Output: 1234

Convert the key to a JSON-safe escaped string

Config files like config.json cannot store multi-line strings. The private key PEM has newlines, so you need to escape them as \n.

# Node.js method
node -e "console.log(require('fs').readFileSync('private_pkcs8.pem','utf8').replace(/\n/g,'\\n'))"

# Python method (often more reliable across platforms)
python3 -c "print(open('private_pkcs8.pem').read().replace('\n', '\\n'))"

# awk method
awk 'NF {printf "%s\\n", $0} END {print ""}' private_pkcs8.pem

The resulting single-line string can be pasted directly into your config:

{
  "config": {
    "privateKey": "-----BEGIN PRIVATE KEY-----\nMIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEA...\n-----END PRIVATE KEY-----\n"
  }
}

Warning — Never commit private keys to version control

The private key in this article is a test key. Never commit a real private key to a git repository, even a private one. Store keys in environment variables, a secrets manager (AWS Secrets Manager, HashiCorp Vault), or an encrypted config service.

8. Common pitfalls & security notes

Don't log decrypted PINs

The catch blocks in the utility log errors. Never add a console.log(decryptedPin) for debugging. Application logs often end up in log aggregation platforms, and PINs in plaintext in logs are a compliance violation.

Singleton means shared state

The module exports new OpenSSLUtil() — a singleton. The key pair is shared across all requests. This is fine if keys are read-only after initialization. But if you ever implement key rotation, you need to handle the transition carefully to avoid decryption failures on in-flight requests.

OAEP encryption is non-deterministic — that's a feature

Encrypting "1234" twice will produce two completely different ciphertexts. OAEP padding incorporates a random seed on every operation. This means an attacker cannot use a pre-computed rainbow table to map ciphertexts back to PINs, and cannot tell whether two encrypted values are the same PIN. Each encryption is cryptographically unique.

Encryption vs signing — knowing when to use each

Scenario	Use	Why
Sending a PIN from client to server	Encrypt (public key, OAEP)	Keeps PIN confidential in transit
Server proves a response came from it	Sign (private key)	Client can verify with public key
Client proves a request came from it	Sign with client's private key	Requires client-side key management
Storing a PIN for comparison	Hash (bcrypt / SHA-256)	One-way — even server can't recover plaintext

9. Conclusion

PIN encryption with RSA-OAEP in Node.js is a well-trodden pattern that, when implemented correctly, provides strong guarantees: a PIN that leaves a client device is unreadable to anyone who intercepts it, because only the server's private key can unlock it.

The key ideas to carry forward are:

Asymmetric encryption solves the key-distribution problem. Share the public key freely; guard the private key absolutely.
OAEP with SHA-256 is the modern, secure padding choice. It is non-deterministic, tamper-resistant, and immune to padding oracle attacks — use it for all new RSA encryption.
Buffer ↔ string conversions are explicit in Node.js. Always know whether you are working with binary data or a string, and which encoding (base64, utf8, hex) bridges them.
PKCS#8 is the modern format. If you see BEGIN RSA PRIVATE KEY, convert it before passing it to Node.js v17+.
Encryption is not storage. After decrypting a PIN on the server, hash it before comparing to the stored value. RSA encryption is for transit security, not at-rest security.

From here, your natural next steps are implementing key rotation, considering hybrid encryption (RSA wrapping an AES key) for any payloads larger than a short PIN, and exploring the browser's WebCrypto API for client-side encryption using the same OAEP scheme.

Building a Production-Ready RAG Application with FastAPI, LangChain, and Google Gemini: A Deep Dive

instanceofGod — Fri, 20 Feb 2026 20:02:38 +0000

Picture this: You’ve just deployed a shiny new AI assistant for your company. Your stakeholders are thrilled. Then, a highly-valued client asks it a crucial question about your brand-new, unreleased product documentation. The AI, possessing all the confidence in the world, proceeds to completely hallucinate an answer based on what it read on the internet two years ago.

This is the fatal flaw of standard Large Language Models (LLMs). Out of the box, they are incredibly articulate but fundamentally ignorant about your proprietary, real-time, or deeply domain-specific data. They confidently guess when they don't know the answer.

Enter Retrieval-Augmented Generation (RAG)—the silver bullet for LLM hallucination.

Think of RAG as giving your AI an open-book test. Instead of relying on its static memory, a RAG system securely searches your private databases for the exact, relevant context and injects it directly into the prompt before the LLM generates a response. You instantly transform a generic AI into a precise domain expert on your custom data.

While there are plenty of simple "hello world" RAG tutorials out there, building something that scales requires careful software design. In this comprehensive deep-dive tutorial, we aren't just writing toy Jupyter cells. I will walk you through building a modular, multi-storage, production-ready RAG web service from scratch using FastAPI, LangChain, and Google Gemini—an architecture that you can actually deploy to the real world.

1. Our Technology Stack

Instead of bloated orchestrators, we'll build on a lean, highly performant stack:

FastAPI: A modern, wildly fast web framework for building our backend API. It gives us async capabilities, automatic data validation, and built-in Swagger UI.
LangChain & LCEL: The leading framework to orchestrate LLM workflows. We'll heavily utilize LangChain Expression Language (LCEL) for declarative, readable pipelines.
Google Generative AI (Gemini): We'll power our system using two models:
- gemini-embedding-001 for mapping text to vectors.
- gemini-2.5-flash for blindingly fast and intelligent text generation.
Durable Vector Stores: To store and query embeddings, our application dynamically supports:
- Pinecone: A fully managed, cloud-native vector database.
- FAISS: Facebook AI Similarity Search for blistering local performance.
- Cloud Backups: Built-in support for synchronizing our FAISS indexes with AWS S3 or Google Cloud Storage (GCS).

2. Project Architecture

Before jumping into the implementation, let's map out our project structure. A good RAG application separates the HTTP layer, the orchestration layer, and the data layer.

rag-app/
├── main.py                          # FastAPI application entry point
├── endpoints.py                     # API route definitions
├── rag.py                           # Core RAG logic & LLM orchestration
├── vector_stores/                   # Vector Database integrations
│   ├── __init__.py
│   ├── pinecone_store.py            # Managed Cloud Vector DB
│   ├── faiss_store.py               # Local Disk FAISS
│   ├── s3_store.py                  # AWS S3 backed FAISS
│   └── gcs_store.py                 # Google Cloud Storage backed FAISS
├── data/                            # Your raw PDFs, TXTs, MDs, etc.
├── .env                             # Environment variables
└── requirements.txt                 # Dependencies

To follow along, make sure your core dependencies are installed:

pip install fastapi uvicorn langchain langchain-core langchain-google-genai langchain-pinecone pinecone pydantic python-dotenv faiss-cpu google-cloud-storage boto3

3. The Data Layer: Multi-Backend Vector Stores

Vector stores hold our document "embeddings" (mathematical representations of text in n-dimensional space). This allows us to perform rapid "similarity searches" using cosine similarity or euclidean distance to find text chunks semantically related to a user's question.

Because production environments vary, we designed our data layer to be dynamic based on environment variables.

Option A: Cloud-Native with Pinecone

For a fully managed experience, we use Pinecone (vector_stores/pinecone_store.py):

import os
from langchain_pinecone import PineconeVectorStore
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import DirectoryLoader
from pinecone import Pinecone

def setup_pinecone(embeddings, data_dir):
    """Setup Pinecone vector store"""
    index_name = os.getenv("PINECONE_INDEX_NAME", "rag-index")
    pc = Pinecone(api_key=os.getenv("PINECONE_API_KEY"))
    index = pc.Index(index_name)

    # Idempotency: Avoid re-processing if the index is already populated
    if index.describe_index_stats()['total_vector_count'] > 0:
        print("Loading existing Pinecone index...")
        return PineconeVectorStore(index=index, embedding=embeddings)

    print("Creating new Pinecone index from documents...")
    loader = DirectoryLoader(data_dir, glob="**/*.*")
    documents = loader.load()

    # Chunk the documents to prevent hitting LLM context windows
    splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
    document_chunks = splitter.split_documents(documents)

    return PineconeVectorStore.from_documents(document_chunks, embeddings, index_name=index_name)

Key Takeaways here:

Idempotency: We check index.describe_index_stats() to ensure we don't ingest the same data multiple times on server restarts.
Chunking: RecursiveCharacterTextSplitter breaks huge documents into smaller 500-character chunks with a 50-character overlap (to prevent cutting ideas in half).

Option B: Local & Cloud-Backed FAISS

If you don't want to use a managed service, FAISS is incredibly fast. However, FAISS runs in-memory. To make it durable across deployments, we've implemented loaders that sync the index directory to S3 or GCS.

Here is a look at our GCS implementation (vector_stores/gcs_store.py):

from google.cloud import storage
from langchain_community.vectorstores import FAISS
# ... imports omitted for brevity

def save_faiss_to_gcs(vector_store, bucket_name, gcs_prefix="faiss_index"):
    local_path = "/faiss_index"
    vector_store.save_local(local_path)
    client = storage.Client()
    bucket = client.bucket(bucket_name)

    # Upload local FAISS index files to GCS
    for file in os.listdir(local_path):
        blob = bucket.blob(f"{gcs_prefix}/{file}")
        blob.upload_from_filename(os.path.join(local_path, file))

def load_faiss_from_gcs(embeddings, bucket_name, gcs_prefix="faiss_index"):
    # Downloads files back from GCS to local disk before instantiating FAISS
    # ...
    return FAISS.load_local(local_path, embeddings, allow_dangerous_deserialization=True)

Our setup_faiss_gcs function tries to load from GCS first; if the index isn't there, it creates it locally from the data/ folder and then pushes it to the bucket. We have an identical setup for AWS S3 using boto3.

4. The Brains: Orchestrating with LangChain (LCEL)

Now we need to glue together our embedding model, our vector store, our prompt, and our Gemini LLM. LangChain makes this elegant using the LangChain Expression Language (LCEL).

Let's look at rag.py:

import os
from dotenv import load_dotenv
from langchain_google_genai import GoogleGenerativeAIEmbeddings, ChatGoogleGenerativeAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
from vector_stores import setup_faiss, setup_pinecone, setup_faiss_gcs, setup_faiss_s3

load_dotenv()
google_api_key = os.getenv("GOOGLE_API_KEY")
data_dir = os.getenv("DATA_DIR", "data/")
vector_store_type = os.getenv("VECTOR_STORE", "faiss").lower()
use_gcs = os.getenv("USE_GCS_STORAGE", "false").lower() == "true"
use_s3 = os.getenv("USE_S3_STORAGE", "false").lower() == "true"

# 1. Initialize the Generator LLM (Gemini 2.5 Flash)
llm = ChatGoogleGenerativeAI(model="gemini-2.5-flash", google_api_key=google_api_key)

def setup_rag_system():
    # 2. Setup Embeddings
    embeddings = GoogleGenerativeAIEmbeddings(model="gemini-embedding-001", google_api_key=google_api_key)

    # 3. Dynamically route to the right vector database
    if vector_store_type == "pinecone":
        vector_store = setup_pinecone(embeddings, data_dir)
    elif use_gcs:
        vector_store = setup_faiss_gcs(embeddings, data_dir)
    elif use_s3:
        vector_store = setup_faiss_s3(embeddings, data_dir)
    else:
        vector_store = setup_faiss(embeddings, data_dir)

    # 4. Create the Retriever (Fetch Top 5 most relevant chunks)
    return vector_store.as_retriever(search_type="similarity", search_kwargs={"k": 5})

def get_qa_chain():
    retriever = setup_rag_system()

    # 5. Construct the System Prompt
    prompt = ChatPromptTemplate.from_template(
        "Answer the question based strictly on the following context:\n\n{context}\n\nQuestion: {question}"
    )

    def format_docs(docs):
        return "\n\n".join(doc.page_content for doc in docs)

    # 6. Build the LCEL Pipeline
    chain = (
        {"context": retriever | format_docs, "question": RunnablePassthrough()}
        | prompt
        | llm
        | StrOutputParser()
    )
    return chain

# Create a singleton instance to use across API calls
qa_chain_instance = get_qa_chain()

async def get_rag_response(query: str):
    return qa_chain_instance.invoke(query)

Deep Dive into LCEL: How data relates

The most beautiful part of this script is Langchain's piping syntax:

{"context": retriever | format_docs, ...} | prompt | llm | StrOutputParser()

Here is exactly what happens when qa_chain_instance.invoke("What is our refund policy?") is called:

Routing: The query "What is our refund policy?" is passed to the dictionary.
Retrieval: For the context key, the query goes into the retriever. The retriever converts the text to a vector using gemini-embedding-001, searches Pinecone/FAISS, and returns the top 5 Document objects.
Formatting: Those 5 documents are piped (|) into format_docs, which extracts the raw text and concatenates it with double newlines.
Prompting: The filled dictionary (context + question) is piped into the ChatPromptTemplate, injecting the values into the string.
Generation: The massive prompt is sent to gemini-2.5-flash.
Parsing: Because ChatModels return metadata-heavy AI Message objects, the StrOutputParser extracts just the raw markdown string.

All of this happens synchronously or asynchronously, abstracted cleanly into one chain.

5. Exposing the Pipeline via FastAPI

To facilitate the deployment of this pipeline for operational use, we expose it over HTTP using FastAPI. By leveraging FastAPI's Pydantic integrations, we effortlessly add strict input validation and automatic OpenAPI (Swagger) documentation.

In endpoints.py, we define the /chat route:

from fastapi import APIRouter, HTTPException
from pydantic import BaseModel
from rag import get_rag_response

router = APIRouter()

# Define our expected JSON Request Body
class Payload(BaseModel):
    chat_input: str

@router.post("/chat")
async def query_rag_system(request: Payload):
    try:
        # Pass the input securely to our RAG service
        response = await get_rag_response(request.chat_input)
        return {"input": request.chat_input, "response": response}
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

And in main.py, we strap it all together:

from fastapi import FastAPI
from endpoints import router

app = FastAPI(title="RAG Application with Langchain and Gemini")

# Register our route controllers
app.include_router(router)

6. Running, Testing, and Deployment Options

To fire up your robust new RAG server, spin up Uvicorn in your terminal:

uvicorn main:app --reload

By default, the server spins up on http://127.0.0.1:8000. Thanks to FastAPI, you instantly have an interactive UI available.

Visit http://127.0.0.1:8000/docs.
Open the POST /chat endpoint.
Pass a payload into the chat_input.

{
  "chat_input": "What data formatting does our application support?"
}

Behind the scenes, the system will embed your query, search your data sources, generate a context-aware prompt, consult Google Gemini, and seamlessly stream back a laser-accurate answer over HTTP.

Deploying to Production

Because the entire app is stateless (state is managed by Pinecone/S3/GCS), you can quickly containerize this application with Docker.

Example Docker command flow:

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

Deploy the image to Google Cloud Run, AWS App Runner, or Railway, provide your .env variables, and you have an instantaneously scalable AI backend.

Conclusion & Next Steps

Building an intelligent, context-aware chatbot doesn't require a messy monolith of code. By cleanly separating our concerns—FastAPI for networking, Langchain for orchestration, Pinecone/FAISS for storage, and Google Gemini for raw intelligence—we've crafted a highly scalable codebase.

Where to go from here?

Conversation Memory: Try wrapping our chain in RunnableWithMessageHistory to sustain complex, multi-turn chat interactions.
Metadata Filtering: If you serve multiple users, update the Pinecone/FAISS retriever to apply metadata filtering based on user_id, ensuring strict data isolation per user.
Agentic Workflows: Replace our linear chain with langgraph to allow the LLM to decide whether it needs to search the vector database, search the web, or use an API tool.

Happy Coding! Let me know in the comments how you plan to use RAG in your own projects, and if you have any questions setting up your vector indexes!

Secure PIN Processing: A Deep Dive into ISO 9564-1 PIN Blocks, RSA, and AES Encryption

instanceofGod — Fri, 13 Feb 2026 21:55:36 +0000

In the modern payments landscape, security isn’t just a feature — it’s the absolute foundation. Every time you type your PIN into an ATM, a point-of-sale (POS) terminal, or a payment app, you are trusting a complex, global system with your financial identity. That simple 4-to-12 digit number initiates a sophisticated cryptographic handshake, orchestrated in milliseconds, to ensure your PIN remains a secret (never travels in “clear text”) from the moment it leaves your fingertips to the moment it’s verified by your bank.

In this article, we’ll break down the mechanics of the ISO 9564–1 Format 0 (ISO-0) PIN block — the industry-standard recipe for protecting a PIN.

More importantly, we’ll build a robust, full-stack architecture that uses a powerful combination of RSA-OAEP and AES (Advanced Encryption Standard) to keep this sensitive data secure all the way from the browser to the backend.

Note on Security Standards: While many legacy systems still use Triple DES (3DES), it’s important to note that NIST deprecated 3DES in 2023 due to vulnerabilities (e.g., CVE-2016–2183) and its small 64-bit block size. Modern implementations should use AES-256 for symmetric encryption, which is what we’ll implement here.

Part 1: The Anatomy of an ISO-0 PIN Block

The ISO-0 format (also known as ANSI X9.8) is the classic, widely-adopted industry standard for protecting PINs. It works by combining the user’s PIN with their Card Number (PAN) using a bitwise XOR operation. Its genius lies in its simplicity: it combines the secret (PIN) with the known (PAN) to ensure the final encrypted block is unique to that specific card and transaction.

1. Constructing the PIN Field.

We start by creating a fixed-length, 8-byte (16 hex character) block dedicated to the PIN.

First Nibble (4 bits): The control digit. For Format 0, this is always 0. Second Nibble: The length of the PIN. A PIN 1234 has a length of 4, so we use 4.
The PIN: The actual PIN digits, stored as hexadecimal characters. 1234 becomes 12 34.
Padding: The remaining bytes are filled with the hexadecimal value F (1111 binary) to reach the full 8-byte length.
Example: PIN 1234 becomes the 16-character hex string: 0 4 12 34 FF FF FF FF -> 041234FFFFFFFFFF.

2. Constructing the PAN Field.

If we encrypted the PIN field directly, the same PIN for two different cards would produce the same encrypted block — a massive security vulnerability. To prevent this, we “salt” the PIN with the cardholder’s PAN. The standard method uses the last 12 digits of the card number, excluding the final check digit (the Luhn digit). This 12-digit number is then left-padded with zeros to create another 8-byte (16 hex character) block.

Example: PAN 4012345678901234 (last digit 4 is the check digit). We take 012345678901 and pad it: 0000123456789012.

3. The XOR Operation: The Digital Blender.

The final Clear PIN Block is produced by XORing the PIN Field and the PAN Field together. XOR is a bitwise operation that compares two bits. If the bits are the same, the result is 0; if they are different, the result is 1.

PIN Field:     04 12 34 FF FF FF FF FF
PAN Field:     00 00 12 34 56 78 90 12
XOR Operation: ---------------------------------
Clear PIN Block: 04 12 26 CB A9 87 6F ED

This resulting 8-byte block is now ready for its final layer of protection: encryption. The beauty of XOR is that it’s mathematically reversible. When the bank receives the encrypted block, they decrypt it and XOR the result with the same PAN digits to retrieve the original PIN.

Part 2: The Architecture of Trust — Never Expose the Master Key

The formatting is only half the battle. The true test of a secure system is how it manages its cryptographic keys.

A common mistake in payment app designs is placing the symmetric encryption key — the Zone PIN Key (ZPK) — directly on the frontend. This is a catastrophic security risk. If a malicious actor can inspect your JavaScript code, decompile your mobile app, or intercept network traffic, they can steal the key and decrypt any user’s PIN.

The solution to this is a Hybrid Encryption Architecture. This approach leverages the strengths of two different types of cryptography to create a robust, defense-in-depth security model.

The Cryptographic Handshake Workflow
Here is how a secure system handles a transaction:

1. Key Exchange (The Setup): When the client application (e.g., a web browser) starts, its first action is to request an RSA Public Key from the server. This key is designed for encryption only.
2. Frontend Encryption (The Seal): The user enters their PIN and PAN. The browser, using the Web Crypto API, performs a critical operation. It encrypts the raw JSON data containing the {pin, pan} using the server's RSA Public Key and the secure RSA-OAEP padding scheme. This is the seal. Because RSA is asymmetric, the frontend can encrypt data with the public key, but it is mathematically infeasible for it—or any attacker—to decrypt it. The private key is the only thing that can break the seal.
3. Secure Transmission (The Courier): The resulting encrypted binary blob is sent over HTTPS to the Node.js backend. Even if an attacker were to intercept this message, they would only see ciphertext that is useless without the private key.
4. Backend Processing (The Vault): Inside the secure environment of the server, the private key is used to decrypt the payload. Only now, with the raw pin and pan in memory, does the server perform the sensitive operations:
It constructs the ISO-0 clear PIN block using the XOR logic.
It then encrypts this block using the Zone PIN Key (ZPK) with the AES-256 algorithm.
This layered approach ensures that the most critical secret — the ZPK — remains safely inside the server’s “trusted zone” and is never exposed to the user’s device — a potentially hostile environment.

Part 3: Under the Hood — Implementation Highlights (AES Edition)

Let’s look at the code that brings this architecture to life, highlighting the critical security considerations at each layer — now updated to use AES.

Frontend: Sealing the Secret with Web Crypto (JavaScript)

Modern browsers provide the Web Crypto API, a powerful, native layer for performing secure cryptographic operations without relying on potentially vulnerable external libraries. This part remains unchanged, as it handles the RSA encryption layer.

// The data to protect
const dataToEncrypt = new TextEncoder().encode(
    JSON.stringify({ pin: userPIN, pan: userPAN })
);
const encryptedData = await window.crypto.subtle.encrypt(
    {
        name: "RSA-OAEP"
    },
    importedPublicKey, 
    dataToEncrypt
);

Backend: The Vault Logic with AES (Node.js)

On the server, we reconstruct the PIN block. A crucial detail here is the use of setAutoPadding(false). PIN block encryption is a very specific operation. The input must be exactly 8 bytes, and the output must be exactly 8 bytes. Standard block cipher padding schemes like PKCS#7 would alter the length, breaking the strict format required by the ISO standard.

With AES, we must also specify the correct mode. For PIN block encryption, AES in ECB mode is typically used to maintain compatibility with legacy systems, though other modes can be used with careful IV management.

const crypto = require('crypto');

/**
 * Encrypts an 8-byte clear PIN block using AES-256
 * @param {Buffer} clearBlockBuffer - The 8-byte clear PIN block
 * @param {string} zpkHex - The Zone PIN Key in hex (32 bytes for AES-256)
 * @returns {Buffer} - The 8-byte encrypted PIN block
 */
function encryptPinBlockWithAES(clearBlockBuffer, zpkHex) {
    // Ensure we're working with a 32-byte key for AES-256
    const key = Buffer.from(zpkHex, 'hex');
    if (key.length !== 32) {
        throw new Error('AES-256 requires a 32-byte (64 hex character) key');
    }

    // Create AES-256 cipher in ECB mode
    // Note: ECB mode is used here for PIN block compatibility
    const cipher = crypto.createCipheriv('aes-256-ecb', key, null);

    // CRITICAL: Disable automatic padding. PIN blocks are exactly 8 bytes.
    cipher.setAutoPadding(false); 

    // Encrypt the 8-byte clear block to produce an 8-byte encrypted block
    return Buffer.concat([cipher.update(clearBlockBuffer), cipher.final()]);
}

// Example usage:
// const encryptedPinBlock = encryptPinBlockWithAES(clearBlockBuffer, zpkHex);

Part 4: Beyond the PIN Block — The Wider World of Payment Security

While the ISO-0 PIN block is fundamental for “Card Present” transactions (like at an ATM or a chip-enabled terminal), it’s just one piece of a much larger security puzzle.

- CVV/CVC (Card Verification Value): This is the static 3 or 4-digit code printed on your card. It’s not a random number. It’s cryptographically calculated by the issuing bank using a secret Card Verification Key (CVK) and specific data from your card (like PAN and expiry date). It serves as proof that you are in physical possession of the card during a “Card Not Present” (online) transaction.
- 3D Secure (3DS): This is the protocol behind challenges like “Verified by Visa” or “Mastercard SecureCode.” It’s an active, real-time authentication layer that shifts liability for fraud from the merchant to the card issuer. By redirecting the user to their bank’s authentication page (or sending an OTP), it provides an extra layer of proof that the legitimate cardholder is initiating the transaction.
- Tokenization: Modern payment systems often replace sensitive PAN data with tokens. This further reduces risk by ensuring that even if data is intercepted, the actual card number isn’t exposed.

Conclusion: The Future of PIN Security
Building secure payment systems is not about using a single, unbreakable lock. It’s about understanding how to combine different cryptographic primitives — like XOR for formatting, asymmetric RSA for secure key exchange, and symmetric AES for robust data encryption — into a cohesive, layered defense.

By strictly separating the “encryption” layer on the client from the “translation and final sealing” layer on the server, we create a system where a user’s most sensitive data — their PIN — is protected from the moment it is typed until it reaches the secure confines of the banking switch. It’s a fascinating and critical field where mathematical principles are the bedrock of global financial trust.

For a complete, runnable implementation of this architecture with AES encryption, check out thi repository. The code provides a practical, end-to-end example of the concepts discussed in this article.

Cloud Resume API Challenge with GCP

instanceofGod — Sun, 21 Jul 2024 17:26:52 +0000

The job search can be a competitive landscape, and having a standout resume is crucial. But what if your resume could be even more dynamic and accessible? Enter the Cloud Resume API Challenge, a fantastic opportunity to showcase your technical skills and create a modern, API-driven resume.

The primary goal of the challenge is to construct a serverless API that can serve resume data in JSON format. In order to achieve the goal of the project i utilised google cloud resources, particularly Google Cloud Functions and Cloud Firestore, along with GitHub Actions for continuous delivery and continuous deployment (CI/CD).

Not only did I gain valuable insights into the world of resume APIs, but I also honed my skills in several key areas which I decided to share with you here:

1. Cloud Functions

I delved into the world of cloud functions, a serverless execution environment that allows you to run code without managing servers. This was a great way to create a lightweight and scalable backend for my API.

2. Cloud Firestore

Cloud Firestore, a NoSQL database, served as the perfect storage solution for my resume data. Its flexible schema and ease of use made it a breeze to manage my information.

3. Google APIs
Working with Google APIs provided a powerful way to interact with my Cloud Firestore data and expose it through the API.
Building the Cloud Function with Node.js

For the challenge, I opted to use Node.js with JavaScript as my development environment. Here's a sample code snippet showcasing a basic Cloud Function that retrieves resume data from Firestore:


const { Firestore } = require('@google-cloud/firestore')
const functions = require('@google-cloud/functions-framework')
const dotenv = require('dotenv')
dotenv.config()

functions.http('resume', (req, res) => {
  const firestore = new Firestore({
    projectId: process.env.PROJECT_ID
  })
  const document = firestore.collection('resume').doc('kZXU70VsRQ2OuDBAjE')
  document
    .get()
    .then((data) => res.send(data.data()))
    .catch((err) => {
      console.log(err)
    })
})

This function retrieves data from a collection named "resume" in Firestore.

4. CI/CD with GitHub Actions
I implemented a CI/CD pipeline using GitHub Actions to ensured automated deployment to cloud function whenever I pushed changes to my code.

Beyond the Basics

The Cloud Resume API Challenge offers a springboard for further exploration. Here are some ways to take your project to the next level:

Authentication and Authorization: Implement mechanisms to control access to your resume data using authentication and authorization techniques.
Versioning: Integrate versioning to track changes to your resume data over time.
Additional Data Points: Include additional data points like portfolio links or project details to make your resume more interactive.
A Rewarding Journey

Participating in the Cloud Resume API Challenge was a rewarding journey. It allowed me to showcase my technical skills, build a modern resume format, and gain valuable experience with cloud technologies. If you're looking for a way to stand out from the crowd and level up your technical expertise, I highly recommend giving this challenge a try!

4 Ways to backup mySql database to a csv file

instanceofGod — Thu, 13 Jun 2024 10:44:01 +0000

There are several methods to create a CSV backup of your MySQL database. Some third-party database management tools offer additional features for backing up to CSV.
If you prefer command-line control, the mysqldump utility is powerful and flexible. If you are familiar with the python programing language, there are packages to help you write python scripts to backup your database. The method you choose depends on your comfort level and technical expertise

In this article, i share 4 different ways to backup your MySQL database to a CSV file.

Using `mysqldump` and `mysql` Command-Line Tools

Export the Database to SQL File

Use mysqldump to create a dump of your database. This step is optional but useful for backing up the entire database structure and data.

   mysqldump -u username -p database_name > database_backup.sql

Replace username with your MySQL username, database_name with the name of the database, and database_backup.sql with the name you want for your backup file.

Export Table to CSV File

You can export a specific table to a CSV file using the SELECT statement with INTO OUTFILE in MySQL. Here’s how you can do it:

   SELECT * FROM table_name
   INTO OUTFILE '/path/to/your/file.csv'
   FIELDS TERMINATED BY ','
   ENCLOSED BY '"'
   LINES TERMINATED BY '\n';

Replace table_name with the name of the table you want to export, and /path/to/your/file.csv with the full path where you want to save the CSV file.

Ensure the MySQL server has the appropriate permissions to write to the specified path.
The FIELDS TERMINATED BY ',' specifies that fields are separated by commas.
The ENCLOSED BY '"' ensures that fields are enclosed in double quotes.
The LINES TERMINATED BY '\n' specifies the end of a row.

Run the SQL Command

You can execute the SQL command directly through the MySQL shell or using a script. Here’s how to do it from the MySQL shell:

   mysql -u username -p database_name -e "SELECT * FROM table_name INTO OUTFILE '/path/to/your/file.csv' FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n';"

Replace the placeholders with your actual database details.

Example in MySQL Shell

   mysql -u username -p

Select the database:

   USE database_name;

Export the table:

   SELECT * FROM table_name
   INTO OUTFILE '/path/to/your/file.csv'
   FIELDS TERMINATED BY ','
   ENCLOSED BY '"'
   LINES TERMINATED BY '\n';

Permissions Note

Make sure the MySQL user has the FILE privilege to write files to the server.
The directory specified in the INTO OUTFILE path must be writable by the MySQL server process.

Using a Bash Script

For automation, you can create a script (e.g., db_backup.sh) to export the table:

#!/bin/bash

DB_USER="username"
DB_PASS="password"
DB_NAME="database_name"
TABLE_NAME="table_name"
OUTPUT_FILE="/path/to/your/file.csv"

mysql -u $DB_USER -p$DB_PASS -e "SELECT * FROM $TABLE_NAME INTO OUTFILE '$OUTPUT_FILE' FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n';"

Make the script executable:

chmod +x db_backup.sh

The run the script:

./db_backup.sh

Using Python and the pandas library

Install the necessary packages: You need to install pandas and mysql-connector-python (or PyMySQL) to connect to the MySQL database and manipulate the data.

   pip install pandas mysql-connector-python

Create a Python script: sample script to export a MySQL table to a CSV file using pandas.

   import pandas as pd
   import mysql.connector

   # Database configuration
   db_config = {
    'user': 'username',
    'password': 'password',
    'host': 'localhost',
    'database': 'database_name'
   }

   # SQL query to select data from the table
   query = "SELECT * FROM table_name"

   # Connect to the MySQL database
   connection = mysql.connector.connect(**db_config)

   # Read data from the database into a pandas DataFrame
   df = pd.read_sql(query, connection)

   # Close the connection
   connection.close()

   # Export the DataFrame to a CSV file
   output_file = '/path/to/your/file.csv'
   df.to_csv(output_file, index=False)

   print(f"Data has been exported to {output_file}")

Code Explanation

Database Configuration:
- Replace 'username', 'password', 'localhost', and 'database_name' with your actual database credentials.
SQL Query:
- The query variable holds the SQL query to select all data from the specified table. Replace 'table_name' with the name of the table you want to export.
Connecting to the Database:
- Establish a connection to the MySQL database using mysql.connector.connect() with the provided configuration.
Reading Data into a DataFrame:
- Use pd.read_sql(query, connection) to execute the query and load the data into a pandas DataFrame.
Closing the Connection:
- Close the database connection using connection.close().
Exporting to CSV:
- Use df.to_csv(output_file, index=False) to export the DataFrame to a CSV file. Replace '/path/to/your/file.csv' with the desired file path for the CSV file.
Confirmation:
- Print a confirmation message indicating the location of the exported file.

Running the Script

Save the script to a file, for example export_to_csv.py, and run it:

python export_to_csv.py

This script will connect to the specified MySQL database, execute the query to retrieve data from the specified table, and export the data to a CSV file.

Using csvkit to Export a MySQL Table to a CSV File

csvkit is a suite of command-line tools for converting to and working with CSV files.
To export a MySQL table to a CSV file using csvkit, you can use the csvsql command.
Certainly! csvkit is a suite of command-line tools for converting to and working with CSV files. To export a MySQL table to a CSV file using csvkit, you can use the csvsql command.

Install csvkit and mysql-connector-python: Make sure you have csvkit and mysql-connector-python installed. You can install it using pip:

   pip install csvkit mysql-connector-python

import subprocess
import mysql.connector

# Database configuration
db_config = {
    'user': 'username',
    'password': 'password',
    'host': 'localhost',
    'database': 'database_name'
}

# Table name and output file
table_name = 'table_name'
output_file = '/path/to/your/file.csv'

# Connect to the MySQL database to get the connection details
connection = mysql.connector.connect(**db_config)
cursor = connection.cursor()

# Construct the csvsql command
csvsql_command = [
    'csvsql',
    '--db',
    f'mysql+mysqlconnector://{db_config["user"]}:{db_config["password"]}@{db_config["host"]}/{db_config["database"]}',
    '--query',
    f'SELECT * FROM {table_name}',
    '--output',
    output_file
]

# Execute the csvsql command
subprocess.run(csvsql_command, check=True)

# Close the connection
cursor.close()
connection.close()

print(f"Data has been exported to {output_file}")

Database Configuration:
- Replace 'username', 'password', 'localhost', and 'database_name' with your actual database credentials.
Table Name and Output File:
- Replace 'table_name' with the name of the table you want to export.
- Replace '/path/to/your/file.csv' with the desired file path for the CSV file.
Connecting to the Database:
- Establish a connection to the MySQL database using mysql.connector.connect() with the provided configuration.
Constructing the csvsql Command:
- Use csvsql with the --db option to specify the database connection string.
- The --query option specifies the SQL query to run.
- The --output option specifies the output file for the CSV data.
Executing the Command:
- Use subprocess.run() to execute the constructed csvsql command.
Closing the Connection:
- Close the database connection and cursor.
Confirmation:
- Print a confirmation message indicating the location of the exported file.

Save the script to a file, for example export_to_csv_with_csvkit.py, and run it:

python export_to_csv_with_csvkit.py

This script will connect to the specified MySQL database, execute the query to retrieve data from the specified table, and export the data to a CSV file using csvkit.

Note: The subprocess module in Python is used to spawn new processes, connect to their input/output/error pipes, and obtain their return codes

It allows you to run external commands and interact with them programmatically. This is especially useful for automating command-line tasks and integrating external tools into your Python scripts.

Key Functions in subprocess
subprocess.run(): This function runs a command, waits for it to complete, and then returns a CompletedProcess instance.
subprocess.Popen(): This is a more powerful and flexible function for spawning new processes. It allows more complex interactions with the process.

Example Usage
Here’s a simple example demonstrating the use of subprocess.run():

import subprocess

# Define the command to run
command = ['echo', 'Hello, World!']

# Run the command
result = subprocess.run(command, capture_output=True, text=True)

# Print the command's output
print(result.stdout)

Explanation of Parameters:

command: A list where the first element is the command to run, and the subsequent elements are the arguments to the command.

capture_output: If set to True, it captures the standard output and standard error.

text: If set to True, the output is returned as a string instead of bytes.

NOTE: the csvkit command an also be run as a one line command using if you have csvkitn installed. You can install csvkit with pip install csvkit:

csvsql --db mysql://[username]:[password]@localhost/[database_name] --query "SELECT * FROM [table_name]" > [output_file.csv]

JavaScript Array methods

instanceofGod — Sat, 01 Jun 2024 14:10:54 +0000

JavaScript provides a rich set of array methods to manipulate and work with arrays. Here is a comprehensive list of these methods along with sample usage:

Mutator Methods
These methods modify the array they are called on.

push()

Adds one or more elements to the end of an array and returns the new length of the array.

sample:

let arr = [1, 2, 3];
arr.push(4);  // arr is now [1, 2, 3, 4]

pop()

Removes the last element from an array and returns that element.

sample:

let arr = [1, 2, 3];
let lastElement = arr.pop();  // arr is now [1, 2]; lastElement is 3

shift()

Removes the first element from an array and returns that element.

sample:

let arr = [1, 2, 3];
let firstElement = arr.shift();  // arr is now [2, 3]; firstElement is 1

unshift()

Adds one or more elements to the beginning of an array and returns the new length of the array.

sample:

let arr = [1, 2, 3];
arr.unshift(0);  // arr is now [0, 1, 2, 3]

splice()

Adds and/or removes elements from an array.

sample:


let arr = [1, 2, 3, 4];
arr.splice(1, 2, 'a', 'b');  // arr is now [1, 'a', 'b', 4]

sort()

Sorts the elements of an array in place and returns the sorted array.

sample:


let arr = [3, 1, 4, 1, 5, 9];
arr.sort();  // arr is now [1, 1, 3, 4, 5, 9]

reverse()

Reverses the order of the elements in an array in place.

sample:

let arr = [1, 2, 3];
arr.reverse();  // arr is now [3, 2, 1]

Accessor Methods

These methods do not modify the array and typically return a new array or a single value.

concat()

Merges two or more arrays.

sample:

let arr1 = [1, 2];
let arr2 = [3, 4];
let newArr = arr1.concat(arr2);  // newArr is [1, 2, 3, 4]

join()

Joins all elements of an array into a string.

sample:


let arr = [1, 2, 3];
let str = arr.join('-');  // str is '1-2-3'

slice()

Returns a shallow copy of a portion of an array into a new array.

sample:

let arr = [1, 2, 3, 4, 5];
let slicedArr = arr.slice(1, 3);  // slicedArr is [2, 3]

indexOf()

Returns the first index at which a given element can be found in the array.

sample:

let arr = [1, 2, 3];
let index = arr.indexOf(2);  // index is 1

lastIndexOf()

Returns the last index at which a given element can be found in the array.

sample:

let arr = [1, 2, 3, 2];
let index = arr.lastIndexOf(2);  // index is 3

includes()

Determines whether an array includes a certain value.

sample:

let arr = [1, 2, 3];
let hasTwo = arr.includes(2);  // hasTwo is true

toString()

Returns a string representing the array.

sample:

let arr = [1, 2, 3];
let str = arr.toString();  // str is '1,2,3'

toLocaleString()

Returns a localized string representing the array.

sample:


let arr = [1, 2, 3];
let str = arr.toLocaleString();  // str might vary based on locale

Iteration Methods

These methods iterate over the array, typically applying a function to each element.

forEach()

Executes a provided function once for each array element.

sample:

let arr = [1, 2, 3];
arr.forEach(element => console.log(element));  // logs 1, 2, 3

map()

Creates a new array with the results of calling a provided function on every element.

sample:


let arr = [1, 2, 3];
let doubled = arr.map(x => x * 2);  // doubled is [2, 4, 6]

filter()

Creates a new array with all elements that pass the test implemented by the provided function.

sample:

let arr = [1, 2, 3, 4];
let evens = arr.filter(x => x % 2 === 0);  // evens is [2, 4]

reduce()

Applies a function against an accumulator and each element to reduce it to a single value.

sample:

let arr = [1, 2, 3, 4];
let sum = arr.reduce((acc, curr) => acc + curr, 0);  // sum is 10

reduceRight()

Applies a function against an accumulator and each element (from right to left) to reduce it to a single value.

sample:

let arr = [1, 2, 3, 4];
let sum = arr.reduceRight((acc, curr) => acc + curr, 0);  // sum is 10

some()

Tests whether at least one element passes the test implemented by the provided function.

sample:

let arr = [1, 2, 3];
let hasEven = arr.some(x => x % 2 === 0);  // hasEven is true

every()

Tests whether all elements pass the test implemented by the provided function.

sample:

let arr = [1, 2, 3];
let allPositive = arr.every(x => x > 0);  // allPositive is true

find()

Returns the first element that satisfies the provided testing function.

sample:


let arr = [1, 2, 3];
let found = arr.find(x => x > 1);  // found is 2

findIndex()

Returns the index of the first element that satisfies the provided testing function.

sample:


let arr = [1, 2, 3];
let index = arr.findIndex(x => x > 1);  // index is 1

flat()

Creates a new array with all sub-array elements concatenated into it recursively up to the specified depth.

sample:

let arr = [1, [2, [3, 4]]];
let flatArr = arr.flat(2);  // flatArr is [1, 2, 3, 4]

flatMap()

First maps each element using a mapping function, then flattens the result into a new array.

sample:


let arr = [1, 2, 3];
let flatMapped = arr.flatMap(x => [x, x * 2]);  // flatMapped is [1, 2, 2, 4, 3, 6]

entries()

Returns a new Array Iterator object that contains the key/value pairs for each index in the array.

sample:

let arr = ['a', 'b', 'c'];
let iterator = arr.entries();
for (let [index, value] of iterator) {
  console.log(index, value);  // logs 0 'a', 1 'b', 2 'c'
}

keys()

Returns a new Array Iterator that contains the keys for each index in the array.

sample:

let arr = ['a', 'b', 'c'];
let iterator = arr.keys();
for (let key of iterator) {
  console.log(key);  // logs 0, 1, 2
}

values()

Returns a new Array Iterator object that contains the values for each index in

Securely Connecting to PostgreSQL on a Virtual Machine from Anywhere

instanceofGod — Mon, 22 Apr 2024 11:21:47 +0000

Prerequisites:

A virtual machine set up on a cloud provider (AWS, Google Cloud, Azure) or a local virtualization platform (VirtualBox).
Basic understanding of command line and Linux concepts.

Deploying PostgreSQL on a Virtual Machine (VM) and connecting to it from anywhere involves several steps. This guide walks you through deploying PostgreSQL on a Virtual Machine (VM) and connecting to it remotely with enhanced security practices.

Once your VM is running, you'll need to install PostgreSQL. This process varies depending on the operating system of your VM, but it generally involves downloading and installing the PostgreSQL package.

1. Install and Configure PostgreSQL

1.1. Update and Install:

sudo apt update
sudo apt install postgresql postgresql-contrib -y

1.2. Verify PostgreSQL user:

PostgreSQL creates a user named postgres. Verify its existence:

sudo cat /etc/passwd | grep -i postgres
postgres:x:113:120:PostgreSQL administrator,,,:/var/lib/postgresql:/bin/bash

To connect to Postgres, switch to the postgres user and run psql:

sudo -i -u postgres

psql
postgres=#

1.3. Create a Secure User (Optional):

For better security, create a dedicated user for remote access instead of using the default postgres user. We'll create a user named ubuntu and grant them necessary privileges based on your specific needs. Avoid granting superuser access (equivalent to root in Linux) unless absolutely essential.

To create a Postgres user run the following command which will give an interactive prompt for configuring the new user. For the sake of simplicity, the ubuntu user will be a superuser, which is the equivalent of being a root user on linux. The super user will have the ability to create/delete/modify databases and users.

createuser --interactive
Enter name of role to add: ubuntu
Shall the new role be a superuser? (y/n) y

# Set password for myuser (replace 'password' with a strong password)
ALTER USER ubuntu PASSWORD 'password';

# Grant appropriate privileges (adjust as needed)
GRANT CONNECT ON DATABASE postgres TO ubuntu;
GRANT CREATE DATABASE TO ubuntu;

\q
exit

psql
Type "help" for help.

postgres=# \du
                                   List of roles
 Role name |                         Attributes                         | Member of
-----------+------------------------------------------------------------+-----------
 postgres  | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
 ubuntu    | Superuser, Create role, Create DB                          | {}

postgres=#

Exit out of the psql by running \q and also exit out of the postgres user by running exit on the command line

Let's try to run psql as the ubuntu user now. An error similar to the one below should be observed

$ psql
psql: error: could not connect to server: FATAL:  database "ubuntu" does not exist

The reason for this is that Postgres by default tries to connect to a database that is the same name as the user. Since the user is ubuntu it tries to connect to a database called ubuntu as well which does not exist. We can go in and create a database called ubuntu so that it will automatically connect, however I find this unnecessary. Instead we can pass in the -d flag and connect to a database that we know exists like the postgres

psql -d postgres

With the above steps we have successfully installed and configured Postgres on the server.

Configure PostgreSQL to accept connections from any location.

By default PostgreSQL is configured to be bound to “localhost”, on port 5432 . It means any attempt to connect to the postgresql server from outside the machine will be refused.

netstat -nlt
Proto Recv-Q Send-Q Local Address           Foreign Address         State

tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN

To allow your postgres server to accept connection from any location or a specific location (IP address), you will need to edit two files:

i.pg_hba.conf

ii.postgresql.conf

You can use the following command to easily locate the files on your server

sudo -u postgres psql -c "SHOW hba_file" -c "SHOW config_file"

`pg_hba.conf`

From 20.1. The pg_hba.conf File: "_Client authentication is controlled by a configuration file, which traditionally is named pg_hba.conf and is stored in the database cluster's data directory. (HBA stands for host-based authentication.) _"

host    all   all   0.0.0.0/0   trust
host    all   all   ::/0        trust

`postgresql.conf`

The configuration file comes with helpful hints to get this working

To allow your Postgres server to accept connection from any location or a specific location (IP address), change the listen_addresses from ‘localhost’ to the IP address of the server you want to connect from. To allow for connection from anywhere( not advised), change ‘localhost” to “*”.

Remember, exposing your database to the internet comes with security risks. Always ensure you have strong, unique passwords and consider setting up additional security measures.

listen_addresses = 'localhost'          # what IP address(es) to listen on;
                                        # comma-separated list of addresses;
                                        # defaults to 'localhost'; use '*' for all
                                        # (change requires restart)

For a quick and dirty solution just change it to

listen_addresses = '*'

Restart postgres

After the configuration, you will need to restart the postgres server. Once PostgreSQL is restarted, it will start listening on all IP addresses.

To restart PostgreSQL:

$ sudo systemctl restart postgresql@11-main # change 11 to the version of your postgres

# or

$ pg_ctl restart

Open Postgres port on server firewall rule

Remember to open the port that PostgreSQL uses (default is 5432) on your VM's firewall. This step will depend on your VM's operating system and firewall settings.

Finally, you can connect to your PostgreSQL database from anywhere using a PostgreSQL client. You'll need your VM's public IP address, the database name, and the login credentials.

Secure Remote Connection with SSH Tunneling

Another option to securely connect to postgres server is by using SSH Tunneling. To enable secure remote access, we'll leverage SSH Tunneling. This creates an encrypted tunnel between your local machine and the VM, allowing you to connect to PostgreSQL through this secure channel.

4.1. Establish an SSH Tunnel:

Use your preferred SSH client to create a tunnel. Here's an example command:

Bash

ssh -L 5433:localhost:5432 <username>@<VM_IP_Address>

Replace the placeholders:

<username> with your VM username.
<VM_IP_Address> with the public IP address of your VM.

This command creates a local port (5433) on your machine that forwards traffic to the PostgreSQL port (5432) on the VM.

4.2 Connect to PostgreSQL Remotely:

Use a PostgreSQL client on your local machine and connect to:

localhost:5433  # Port you forwarded in the tunnel (5433)

Use the credentials of the user you created (e.g., ubuntu).

The steps above establish a secure connection using SSH Tunneling. To further enhance security, consider these additional measures:

Never modify listen_addresses in postgresql.conf to '*'. This exposes your database server to the entire internet, making it vulnerable to attacks.
Configure firewall rules on your VM to restrict access to the PostgreSQL port (default 5432) from only your specific IP address. This adds an extra layer of protection.
Implement strong password policies and enforce secure user authentication methods. Utilize the pg_hba.conf file to control which users can connect from where and how they can authenticate. Refer to PostgreSQL documentation for detailed configuration options.

5. Conclusion:

By following these steps, you can securely connect to your PostgreSQL database on a VM from anywhere or by using SSH tunneling. Remember, security should be a top priority. Implement the additional security measures mentioned above to create a more robust defense for your database server.

For further details on PostgreSQL configuration and security best practices, refer to the official PostgreSQL documentation https://www.postgresql.org/docs/.

Helpful link and references:

https://stackoverflow.com/questions/24504680/connect-to-postgres-server-on-google-compute-engine

Enable Password Authentication in AWS EC2 Instance

instanceofGod — Mon, 16 May 2022 13:56:26 +0000

Password authentication is disabled by default in aws ec2 instance. The only way to access your server is by using ssh with -i flag and followed by private key attached to the server.

However, there are scenarios when you may need to access the server with just username and password. Take note that this method of accessing your server is not encouraged, as it could open your server to attacks.

How to Enable Password Authentication in EC2

First, you have to ssh into your server with your private key to set password for the user and also make some changes to the sshd_config file.

The steps to enable password authentication are highlighted below:

ssh -i privatekey username@host_ip

Setup a password for the user using the passwd command

sudo passwd username

Open and modify the sshd_config file.

sudo vim /etc/ssh/sshd_config

sudo nano /etc/ssh/sshd_config

If you prefer to use nano as editor

Changes the PasswordAuthentication line from ‘no’ to ‘yes’

PasswordAuthentication yes

Enable root login, Change value from “prohibit-password” to “yes” (Optional)

PermitRootLogin yes

Restart the “sshd” service using the following command.

sudo service sshd restart

or 
sudo systemctl restart ssh

That’s all you have to do. Now you can login to the ec2 server using the password you set for the user, without the private key.

When you type the command below, you will be prompted to enter your password!

A beginners' Journey into Web Development.

instanceofGod — Mon, 16 Aug 2021 22:03:27 +0000

Hey, thinking of starting a career into web development but don't know how and where to begin?

One of the most difficult decision to make by beginners when venturing into web development is how to begin, and what resources are available to aid their learning.

If this is where you find yourself now, like me, when I first started learning web development ( I wish I had seen and read an article like this then), this article will help to provide a roadmap to help you to learn faster and better.

Start from the Basic.

As foundation is important in construction, so is it in software development. Starting from the basic means starting from ground up, learning the foundational concepts of HTML such as tags, semantics, structure, layout etc. Want to Start Learning HTML now, **https://www.tutorialrepublic.com/html-tutorial/** is one of the good site to start your journey into web development.

You need to understand how to arrange elements on the web page, that is, putting elements where they ought to be on the page. One good tool that than help you achieve this is FIGMA. Learn more about figma **HERE https://trydesignlab.com/figma-101-course/introduction-to-figma/**

Advance to Aesthetics

After you are comfortable with html and have a good knowledge of laying out your page, giving your code good visual appearance with CSS is the next step. CSS is used for styling your HTML pages. While HTML is like an unpainted building, CSS is the 'paint' that makes the we page appealing to the eyes.

Other Things to learn includes Version Control. Want to know more about Version Control using GIT, go to : https://www.freecodecamp.org/news/learn-the-basics-of-git-in-under-10-minutes-da548267cc91/

Choose a Programming Language

There are several programming languages that does different or same thing in different ways.

You will need to decide your goal and objective for the future before deciding which one to learn first. Of course, you can learn many languages over time, but it is better to pick one and be a master of it, than not being able to do anything productive with the knowledge of several programming languages.

Examples of Programming language are: PHP, Python, Java, JavaScript, Go, Rust, C#, C++, MySql, Kotlin, Ruby, Perl and many more. Make proper research about the language, what you can use it to achieve, whether it fulfils your future goal as a software developer.

Fortunately, the internet abounds with resources- Video, Articles, Guides or Courses to make your learning journey smooth and sound. You can checkout the following resource:

PHP - https://www.w3schools.com/php/default.asp

JavaScript - https://www.w3schools.com/js/default.asp

Python - https://www.w3schools.com/python/default.asp

Go - https://www.w3schools.com/go/index.php

ESLint Prettier & Husky Configuration for Project

instanceofGod — Thu, 18 Mar 2021 11:39:57 +0000

Setting up Prettier with ESLint, and also checking commit with husky.

You will need to install Eslint, Prettier and Pre-Commit.
The 3 packages does different things, their usefulness are highlighted below:

ESLint - For Linting, and Enforcing Style Guide.
Prettier - For Code Formatting.
Pre-Commit - For Making sure code that's being committed is followed guidelines and is formatted properly.

Installation

The packages can be installed locally in your project or globally:

Setting up ESLint

npm install --save-dev eslint

After installing eslint, you then need to set it up, that is, configure it.

npx eslint --init

Sample of the text that will be inside the eslintrc.json file

{
"extends": ["airbnb", "prettier" "eslint:recommended"],
"plugins": ["prettier"],
"rules": {
"semi": ["error", "always"],
"quotes": ["error", "double"]

"prettier/prettier": ["error"] // if you choose to use prettier rules.
},

}

Setting up Prettier

npm install --save-dev prettier eslint-config-prettier eslint-plugin-prettier

Then create .prettierrc file in your project’s root directory.

You can type echo {} .prettierrc in your terminal to create the file

touch {} .prettierrc

This will be where you configure your prettier formatting settings.

Sample of the settings you can configure inside the prettierrc.json file


{ 
"useTabs": true,
"tabSize": 2,
"semi": true,
"singleQuote": true,
"trailingComma": "es5",
"bracketSpacing": true,
"parser":"typescript"

}

Setting up Pre-commit and Pre-push

This is a pretty important step. This will ensure that before any developer in your team commits any code, the changes made by him are validated with
EsLint, and the code is properly formatted.

The fastest way to start using lint-staged is to run following command in your terminal:

npx mrm lint-staged

It will install and configure husky and lint-staged depending on code quality tools from package.json dependencies, so please make sure you install (npm install --save-dev) and configure all code quality tools like Prettier, ESlint prior that.

Another alternative to implement this; we will need to configure some packages

The First package we need is husky which will make adding these hooks very easy.
We also need a package called lint-staged that will let us check only the pages which are changed. So, only the staged files are checked and the rest of the code remains untouched.
pretty quick will check for any unformatted files and format them using Prettier.

Installation of the packages

npm install --save-dev husky lint-staged pretty-quick

After installing these packages we need to add configuration for these in our package.json file

"lint-staged": {
"*.js": "eslint --cache --fix"
},
"husky": {
"hooks": {
"pre-commit": "lint-staged && pretty-quick --staged"
}
}

As the name suggests, when you're trying to commit the changes, this will run both commands "lint-staged and pretty-quick".

The lint-staged will run eslint command on javascript files that are staged,
The pretty-quick will format the JavaScript files if they aren't using Prettier.

If any of the staged files are not properly linted or formatted this will give you a warning and will stop you from committing the
changes in the code.

DEV Community: instanceofGod

SwiftDeploy: Building a Self-Writing Infrastructure Tool with OPA Policy Gates and Prometheus Observability

Table of Contents

The Problem

Architecture Overview

Part A — The Engine

The Manifest: Single Source of Truth

The Tool That Writes Its Own Infrastructure

The API Service

The CLI Subcommands

Part B — The Eyes and the Brain

Instrumentation: /metrics in Pure Python

The Policy Sidecar: OPA

Writing Rego Policies

Gated Lifecycle: deploy and promote

The Status Dashboard

The Audit Report

The Chaos Experiments

Experiment 1: Error Injection

Experiment 2: Slow Responses

Experiment 3: Disk Full Gate

Lessons Learned

Replication Guide

Prerequisites

Step 1: Clone and build

Step 2: Init and validate

Step 3: Deploy

Step 4: Verify endpoints

Step 5: Promote to canary

Step 6: Inject chaos and watch the status dashboard

Step 7: Recover and generate audit report

Step 8: Teardown

How I Built a Real-Time DDoS Detection Engine from Scratch

Introduction

The Architecture

Step 1 — Reading the Log File in Real Time

Step 2 — The Sliding Window

Step 3 — The Rolling Baseline

Step 4 — Detecting Anomalies

Step 5 — Blocking with iptables

Step 6 — Auto-Unban with Backoff

Step 7 — Slack Alerts

Step 8 — The Live Dashboard

Putting It All Together

Key Takeaways

Setup & Walkthrough for Beginners

Prerequisites

Step 1: Clone the Repository

Step 2: Configure Nginx to Write JSON Logs

Step 3: Set Up the Python Environment

Step 4: Configure Slack Alerts (Optional But Recommended)

Step 5: Run the Detector

Step 6: Generate Test Traffic to See Detection in Action

Step 7: View the Live Dashboard

Step 8: Verify iptables Block Is Working

Step 9: Test Auto-Unban

Step 10: Run as a System Service (Production)

Common Troubleshooting

Testing Locally Without a Real Server

Next Steps & Customization

PIN Encryption & Decryption with RSA in Node.js

Table of Contents

1. Why PIN encryption matters

2. Core terms defined

3. How RSA works

4. Anatomy of a key pair

The public key

The private key (PKCS#8)

Key format summary

5. The OpenSSLUtil class, line by line

Constructor and key loading

Encrypt method — OAEP-SHA256

Decrypt method — OAEP-SHA256

Sign method

Module export as singleton

6. The full encrypt → transmit → decrypt flow

Step 1 — Client encrypts the PIN (terminal simulation)

Step 2 — Client sends the encrypted PIN

Step 3 — Server decrypts and verifies

7. Working with keys in the terminal

Using `mysqldump` and `mysql` Command-Line Tools

`pg_hba.conf`

`postgresql.conf`