DEV Community: arif

i built a cad editor in the browser, then taught an llm to use it

arif — Fri, 03 Jul 2026 14:27:04 +0000

the demo moment i did not plan: i asked my app "how many doors and windows are there?" and the ai answered the counts, then added, unprompted:

note D3 is only 300 mm wide, likely a mis-detected door. want me to check it?

it was right. my extraction pipeline had turned a piece of geometry into a 30 cm door. no human had noticed. the model read the quantity take-off, saw a door narrower than a shoebox, and flagged it.

that little moment is the payoff of a longer story: parsing one of the most hostile file formats in existence (autocad dwg), reconstructing real building models from thousands of anonymous line segments, building a 2d cad editor from scratch in canvas, and then handing the whole thing to claude as a set of tools it can call.

this is how i built it, including the parts that went wrong.

what the thing does

          upload                durable etl job                     viewer + editor
 browser ────────▶ fastapi ───────────────────▶ dwg → dxf → model ────▶ react + three.js
 (react)           + dbos        (libredwg + ezdxf, by layer)           2d cad editor
                                       │                                    ▲
                                       ▼                                    │
                                 entity rows in db  ◀──── ai copilot ───────┘
                                 (walls, doors, rooms)     (claude + tools)

you drop a dwg floor plan in the browser. a durable background job converts it, extracts the geometry, and reconstructs a structured building model: walls as centerlines, doors and windows placed on their host walls, rooms as polygons with names and areas. you get a 3d viewer, a full 2d cad editor (snapping, dimensions, undo, print to scale), quantity take-offs, and export back to dwg.

and then there is a chat panel where an ai agent can read the model and edit it. for real. the edits land in the database, show up live in 2d and 3d, and sit in your undo history like any manual edit.

the important thing to understand before any of that: a dwg file does not contain walls. it contains lines. everything interesting in this project is the distance between those two sentences.

part 1: dwg is a format that hates you

dwg is a closed binary format with about 30 years of versions. there is no official spec. the open source world has exactly one serious answer: libredwg, which can read basically everything and write reliably to one version (r2000).

two decisions saved me weeks here:

run libredwg as a subprocess, not a library. it is gpl, my code is not, and a subprocess boundary keeps the licenses clean. it also means a segfault in a 30-year-old format parser kills a child process, not my server. the converter turns dwg into dxf, which is the same data model as text, and from there ezdxf (a superb python library) takes over.

never trust the file. my favorite example: dwg files carry a $INSUNITS header that declares the drawing units. it lies constantly. i had a file that declared "metres" while every coordinate was clearly millimetres. trust it and your apartment renders a thousand times too big.

the fix is embarrassingly simple: ignore the header, look at the numbers.

def infer_metres_per_unit(spans: list[float], kind: str) -> float:
    # a building floor plan is 5 to 100 metres across. if the raw
    # coordinate span is ~8000, the drawing is in millimetres,
    # whatever the header claims.
    span = max(spans)
    for scale in (1.0, 0.001, 0.01, 0.0254, 0.3048):
        metres = span * scale
        if plausible_extent(metres, kind):
            return scale
    return 1.0

same paranoia everywhere else. real files come structurally broken: truncated sections, invalid handles, entities the strict parser refuses. when ezdxf.recover gives up, i do not: a dxf is at heart a stream of (group code, value) pairs, so i drop to a raw tag scanner that walks the text and pulls out every LINE, LWPOLYLINE, CIRCLE and TEXT it can find, geometry only, no object model. you lose elegance, you keep the floor plan. and a converter that emits latin-1 diagnostics into a utf-8 pipe cannot blow up the job either (capture bytes, decode with errors="replace", a real crash i hit on 7 of my first 35 test files).

to prove the pipeline was solid i collected 231 real dwg files from the internet: architectural, structural, mechanical, electrical, every version from r1.4 to 2025. a smoke test harness runs each file through the full pipeline in an isolated subprocess with a timeout, so one hang cannot take down the run.

result after fixing everything the corpus surfaced: 231 files, 0 crashes. graceful "i could not reconstruct much from this" is allowed. a traceback is not.

part 2: from thousands of line segments to actual walls

extraction gives you a soup. blocks (reusable symbols like door leafs and window frames) get exploded recursively into their primitive entities, arcs get sampled into chords, polylines get split into edges. what comes out is tens of thousands of bare segments, each carrying exactly one piece of metadata: its layer name.

layer names are the only semantics a dwg has, and they are a free-text field filled by humans. german architects write WAND or MAUER, english ones write A-WALL or WALLS, and everyone abbreviates differently. so classification is humble substring matching over a multilingual hint list:

_WALL_PATS = ("wall", "wand", "mauer", "gebäude", "gebaeude")
_DOOR_PATS = ("door", "tür", "tuer", "porte")
_WIN_PATS  = ("window", "fenster", "wind")

the same trick, with a different hint list (flurstück, parcel, gelände, straße...), decides whether the whole drawing is a building floor plan or a site plan, because the two need completely different reconstruction. and when a file has no recognizable wall layer at all (everything dumped on layer 0, a classic), the pipeline falls back to "every segment that is not obviously a door, window, text or dimension" and attaches a warning saying results are approximate. honesty over confidence.

now the real problem. on the wall layer, a wall is never one line. it is drawn as parallel pairs, and each line is broken into fragments wherever a door or window interrupts it, often drawn in opposite directions by different commands years apart:

 what the dwg contains on the wall layer:

   ──────▶      ◀──────────        ──▶        four fragments, mixed directions

 what the model needs:

   ══════════════════════════════════         one wall centerline
         └─ 900mm gap ─┘   └─ 1200mm gap ─┘   with its gaps remembered

merging those fragments is a grouping problem: which segments lie on the same infinite line? i hash each segment by its quantized angle (4 degree buckets, modulo 180 so direction does not matter) plus its perpendicular offset from the origin (50 mm buckets). segments that share a hash are collinear neighbours.

the bug that taught me the most lived right here. the offset is computed along the line's normal vector, and a segment drawn right-to-left has its normal pointing the opposite way from its left-to-right twin, which flips the sign of the offset, which puts the two halves of the same wall into different hash buckets. the fix is one canonicalization:

# flip the normal into one half-plane so segments drawn in opposite
# directions on the SAME line (two halves of a wall split by a door,
# extremely common in real dwgs) hash together
if ny < 0 or (ny == 0 and nx < 0):
    nx, ny = -nx, -ny
offset = round((a[0] * nx + a[1] * ny) / 50.0)

before that fix, half my test corpus reconstructed twice as many walls as the buildings had.

within each group the merge becomes one-dimensional: project every fragment onto the group's direction, sort the intervals, and sweep. touching or overlapping intervals fuse. gaps up to 2600 mm (a generous door width) get bridged and recorded, because a door-sized hole in a wall line is not noise, it is a door. gaps bigger than that split the wall in two. the recorded gaps come out the other side as opening candidates for free.

with centerlines in hand, classification is geometric. a wall whose midpoint hugs the bounding box perimeter is external (230 mm brick by default), everything else is a partition (100 mm block). testing the midpoint rather than the endpoints matters: an interior wall whose ends touch the facade would otherwise get promoted to external.

doors and windows then snap to their hosts: take each segment from a door or window layer, find the nearest wall within a 500 mm perpendicular tolerance, and project the midpoint onto the wall's axis. that projection is the door's position, stored as one number, center_mm along the wall. openings with the same type and size (rounded to 50 mm) share a schedule mark, which is how D1, D2, W1 get assigned exactly the way a human drafter would.

every dimension nobody drew (storey height 3000, door height 2100, window sill 900) is a stated construction default, appended to the model's warnings so the ui can show "assumed" instead of pretending the file said so.

part 3: rooms are faces of a planar graph

rooms are the part that feels like magic and is actually graph theory.

once you have wall centerlines, a floor plan is a planar subdivision: the lines carve the plane into faces, and the bounded faces are the rooms. extracting them is a classic computational geometry exercise:

 1. split every segment at every         2. walk the half-edges, always
    intersection, snap nodes                taking the sharpest clockwise
    to a 60mm grid                          turn at each node

    ┌──────┬────────┐                       ┌──────┬────────┐
    │      │        │                       │  R1  │   R2   │
    ├──────┴───┬────┤          ──▶          ├──────┴───┬────┤
    │          │    │                       │    R3    │ R4 │
    └──────────┴────┘                       └──────────┴────┘

 3. every minimal face gets traced exactly once.
    one face is the unbounded outside: drop it.

step 2 is the elegant bit. from every directed edge, keep turning as sharply clockwise as possible and you trace exactly one minimal face, and every interior face of the plan gets traced exactly once. no recursion, no flood fill, just angles.

step 3 hid my favorite geometry bug. you get all the rooms plus one extra polygon: the outer boundary of the whole building, traced from the outside. my first heuristic dropped any face bigger than some fraction of the bounding box. worked on every rectangular test building, then an l-shaped floor plan arrived. the outer face of an l-shape is nowhere near its bounding box area, so it slipped under the threshold and appeared in the ui as a giant phantom room covering the entire footprint. the correct rule is topological, not metric: in a connected arrangement the unbounded face encloses everything else, so it is always the single largest face. drop the max, keep the rest. no threshold to tune, no shape that breaks it.

naming the survivors is almost an afterthought: floor plans carry text labels ("kitchen", "büro 2.13"), so run point-in-polygon of each label against each room face and the names fall into place. rooms without a label become "room 7", which is at least honest.

it is worth pausing on what happened across these two parts. the input was an unordered pile of anonymous line segments. the output is: 7 walls with types and materials, 6 doors and 3 windows that know which wall they sit on and where, 4 rooms with names, areas and perimeters, and a structural grid derived from the dominant wall axes. nothing in the file said any of that. it was all latent in the geometry.

part 4: the model is rows, not a blob

the easy design would be: etl emits a json file, viewer reads the json file. done.

i did not do that, and it turned out to be the single most important decision in the project. the extraction writes the model into the database as entity rows:

walls        (id, x1, y1, x2, y2, thickness_mm, height_mm, type, material)
openings     (id, mark, type, wall_id, center_mm, width_mm, height_mm, sill_mm)
rooms        (id, name, polygon, floor_finish, wall_finish)
dimensions   (id, x1, y1, x2, y2, offset_mm)
labels       (id, target_kind, target_id, text, tx, ty)

one endpoint assembles the rows into the model json the viewers consume. another validates an edited model against pydantic schemas and writes it back to the rows.

notice what that buys you. editing is just row updates. dwg export is just reading rows and emitting dxf entities (real DIMENSION and TEXT entities, mitered wall outlines). and, foreshadowing, an ai tool that edits the model is just another caller of the same validated save path.

a door is not a hole in a picture. it is a row that knows which wall it lives on and how far along it sits. everything downstream falls out of that.

part 5: a cad editor is mostly three problems

i researched the buy option first. commercial browser cad sdks either cannot actually edit dwg (they are viewers with markup) or start around $7,500 a year for a general-dwg scope i did not need. i only needed to edit my reconstructed model. so i built it on a raw canvas 2d context.

a cad editor that feels professional is mostly three problems: rendering speed, snapping, and undo.

rendering. my first version redrew everything on every mousemove. on a 963-wall plan that was 1154 ms per frame. one frame per second. the fix is the oldest trick in graphics: split the scene by how often it changes.

 layer 3: overlay canvas    cursor, snap markers, drag previews   every mousemove
 ─────────────────────────────────────────────────────────────────────────────────
 layer 2: model cache       walls, rooms, doors, dimensions       on commit
 ─────────────────────────────────────────────────────────────────────────────────
 layer 1: base cache        grid + original dwg linework          on pan/zoom

layers 1 and 2 are offscreen canvases, redrawn only when their content changes. mousemove touches only the overlay. walls get batched into two Path2D objects (one per wall type) so the browser does two fill calls instead of a thousand.

1154 ms became 82 ms under a software rasterizer, and comfortable 60 fps on a real gpu. no webgl, no framework, just not repainting things that did not change.

snapping. autocad muscle memory is real: endpoints beat intersections beat midpoints beat grid. i keep two r-tree spatial indexes (rbush), one for segments and one for points, query a small box around the cursor, and pick the winner by priority:

endpoint(10) > intersection(9) > midpoint(8) > perpendicular(7) > on-line(5) > grid(2)

one detail that separates "toy" from "correct": a strong object snap overrides ortho lock. if you are drawing orthogonally but hover an endpoint that is 2 degrees off axis, real cad snaps to the endpoint. get that wrong and anyone who has used autocad feels it instantly.

undo. i used snapshots, not command objects. before any transaction the store clones the model (structuredClone, about 200 kb for 1000 walls, 80 snapshots deep). drags mutate the live model freely so every side panel updates in real time, and if you press escape the snapshot rolls back.

execute(mutate: (m: BuildingModel) => void) {
  this.begin();          // clone the model
  mutate(this.model);    // do anything
  this.commit();         // snapshot -> undo stack, autosave kicks in
}

boring, memory-hungry, and completely immune to the classic command-pattern bug where one action forgets to implement its inverse. for models of this size it is the right trade.

saving is a debounced single-flight put of the whole model, with an abort controller so a newer save supersedes an in-flight older one. writes cannot land out of order.

part 6: give the model tools, not a text box

then my company announced an ai hackathon, and i had claude opus 4.8 available through an azure ai foundry endpoint (foundry serves the native anthropic api, so the official sdk works as-is, just point base_url at it).

the tempting demo is rag: "chat with your floor plan." i wanted the other thing. an agent that does work: place the door, label the rooms, run the take-off, fix the storey height.

the architecture is a tool-use loop:

 user: "add a 1500mm window to the longest external wall"
   │
   ▼
 claude ── tool: list_elements(kind=walls) ──▶ backend runs it ──▶ json result
   │                                                                  │
   ◀──────────────────────────────────────────────────────────────────┘
   │   "W1 and W3 are both 10m. W1 already has a door and a
   │    window, so W3 it is."
   ▼
 claude ── tool: add_openings([{wallId: "W3", type: "window",
   │                            width_mm: 1500}])
   ▼
 backend: mutate model -> validate -> save rows -> stream result
   │
   ▼
 claude: "done. added window W3 (1500x1200, sill 900) centred on
          wall W3, with a leader label."

thirteen tools total. three read (summary, element listing, quantity take-off), ten write (walls, openings, rooms, annotations, storey, rename, delete with cascade).

three rules made this safe instead of terrifying:

1. every mutation goes through the same validated path the ui uses.

def _save(project_id: str, model: dict) -> None:
    try:
        save_edited_model(project_id, model)   # pydantic-validated, same as PUT /model
    except Exception as exc:
        raise ToolError(f"edit rejected by model validation: {exc}") from exc

the agent physically cannot persist a model the editor and the dwg exporter would not accept. i did not write a second, special, "for the ai" write path. that is the whole trick, and it only works because part 4 made the model rows with a single schema-checked door in front of them.

2. tool errors are recoverable, not fatal. a ToolError ("no wall with id W99", "a 5000 mm door does not fit on a 3200 mm wall") is fed back to the model as an error result. the model reads it, adjusts, and tries something else. the stream never dies because the agent guessed wrong once. watching it recover mid-conversation is half the magic of the demo.

3. domain guardrails live in the tools, not the prompt. default door sizes, auto-assigned marks, fit checks, cascade deletes (kill a wall, its doors and labels go with it). prompts drift. code does not.

the loop itself is about 60 lines: stream a response, forward text deltas to the browser as server-sent events, and when the model stops with tool_use, execute the tools, append the results, and continue, up to 12 rounds.

the part nobody talks about: ai edits and your undo stack

here is the ux problem that makes or breaks a copilot in an editor. the ai edits the database server-side, but the user is holding a live client-side editing session with its own undo history. if those drift apart you get the worst bug class in collaborative software.

my solution has two halves.

flush before the agent reads. the editor autosaves on a 900 ms debounce, so when you send a chat message there may be an unsaved wall on your screen. before every agent turn the frontend cancels the pending timer and force-saves. the agent always sees your latest state, and, just as important, a stale queued autosave can no longer fire mid-run and silently overwrite what the agent wrote.

fold the result into undo as one step. when the agent reports that it changed something, the frontend refetches the model and applies it through the normal store transaction:

const applyServerModel = async () => {
  const res = await api.getResult(project.id);
  const fresh = structuredClone(res.model);
  store.execute((m) => {
    Object.assign(m, fresh);   // one commit -> one undo entry
  });
};

whatever the agent did, five tool calls, twenty labels, becomes exactly one entry in the undo stack. ctrl+z reverts the entire ai edit, and the autosave writes the revert back to the database. i have an end-to-end browser test that asserts the full round trip: agent adds a door, db count goes up by one, ctrl+z, db count comes back down.

an ai copilot without undo is a liability. with undo it is just a very fast colleague.

two bugs worth telling

the invisible stream. my first browser test passed the "request completed" check but showed an empty chat bubble. the backend streamed sse frames, curl showed them fine, the browser parsed nothing. cause: my parser split frames on \n\n, but the sse library (sse-starlette) terminates lines with \r\n. curl's terminal output hides the difference completely. the fix is one regex:

// frames end with a blank line; the server uses \r\n line endings
while ((m = buf.match(/\r?\n\r?\n/))) {
  const frame = buf.slice(0, m.index);
  buf = buf.slice(m.index + m[0].length);
  // parse "event:" and "data:" lines with split(/\r?\n/)
}

if you ever hand-parse sse from a fetch body: it is always the line endings.

the minus one millimetre door. i wrote a fuzz harness that reconstructs models from the whole dwg corpus and fires every tool at them with valid, edge-case, and garbage inputs, asserting after every single mutation that the model still validates and still exports to dxf. it found that add_openings with width_mm: -1 sailed straight through my fit check:

if width + 100 > length:   # -1 + 100 = 99, "fits" on any wall
    raise ToolError(...)

a door with negative width, persisted to the database. would claude ever send that? probably not, the tool schema declares minimums. but "probably not" is not an invariant, and the boundary has to hold on its own. range checks went in, and the fuzz harness lives in the repo and runs against the corpus like the etl smoke test does.

fuzz your tool layer. the llm is a fuzzer with excellent grammar, so meet it with one that has none.

what i learned

the hard part was never the ai. it was turning anonymous line segments into walls, doors and rooms. i spent 90% of the time on extraction, geometry and the editor, and the ai feature took days, not weeks, precisely because that foundation existed.
never trust file headers. measure the geometry. the units, the layers, the structure: verify everything against what is actually drawn.
geometry bugs hide in the shapes you did not test. rectangular buildings passed for weeks; one l-shaped plan broke the room detection. topological rules (drop the largest face) beat metric thresholds (drop faces over x%) every time.
one validated write path. the ui, the import pipeline, and the ai all write through the same schema-checked door. every safety property you enforce there is enforced everywhere, forever.
let the agent fail cheaply. recoverable tool errors turn "the ai crashed" into "the ai corrected itself", which users read as intelligence.
the ai needs an undo story. refetch and fold into one transaction is simple and it works. ship nothing agentic without it.
smoke test against reality. 231 hostile files taught me more than any spec. the corpus, not my imagination, decided what "robust" means.

the stack, for the curious: fastapi, dbos durable workflows, libredwg + ezdxf, sqlmodel on sqlite (postgres-ready), react + vite + typescript, three.js for 3d, raw canvas 2d for the editor, rbush for spatial indexing, and the anthropic python sdk pointed at claude opus 4.8 on azure ai foundry.

the whole thing runs on one small box. the most satisfying part is still that first conversation: asking a floor plan a question, getting a correct answer, and watching a door it drew for you appear in the drawing, with its swing arc, its mark tag, and a ctrl+z that takes it away again.

Your SSH key is already an account: building multiplayer apps over SSH

arif — Fri, 26 Jun 2026 15:59:36 +0000

Almost every app starts with the same chore. Before a user can do anything, you build a way to know who they are: an email field, a password hash, an OAuth redirect, a confirmation link, a session cookie. It is real work, and none of it is the thing you set out to build.

SSH has handled this since the 1990s, and almost nobody builds on it.

When a client connects over SSH, the handshake cryptographically proves the client holds the private key for the public key it presents. By the time your code runs, the connection has already answered the question every login form exists to answer. The key fingerprint is a stable, unique, verified id for that user. No email, no password, no signup, no third party. You have an account before the first byte of your app runs.

So I pulled the common parts into a small Go framework called wharf, built on Charm's wish and bubbletea.

TL;DR

The SSH key fingerprint is a free, verified, zero-signup user id.
wharf gives you identity, presence, multiplayer rooms, and per-user persistence. You write a Bubble Tea model, it does the rest.
Each room is a single goroutine, so broadcasting is one line and there are no locks.
Storage is a tiny interface with adapters for memory, SQLite, Postgres, Redis, and bbolt, each its own module so the core has no database dependency.
Go, MIT. Repo: github.com/doganarif/wharf

The smallest app

An app is a function from a session to a Bubble Tea model. That is the whole contract.

func main() {
    wharf.New(":2222").
        App("shout", shout).
        Run()
}

func shout(s *wharf.Session) tea.Model {
    return model{
        me:   s.User,          // verified identity, for free
        room: s.Join("shout"), // a shared room
    }
}

ssh shout@your.host and you are in. The room traffic shows up as normal messages in your Update:

func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
    switch msg := msg.(type) {
    case tea.KeyMsg:
        if msg.Type == tea.KeyEnter {
            m.room.Broadcast(line{Who: m.me.ShortID, Text: m.input})
            m.input = ""
        }
    case wharf.Event:    // someone broadcast to the room
        m.lines = append(m.lines, msg.Payload.(line))
    case wharf.Presence: // the roster changed (join or leave)
        m.roster = msg.Roster
    }
    return m, nil
}

That is multiplayer. No socket handling, no membership bookkeeping, no auth.

Identity is free, and persistent

s.User is derived from the public key. The fingerprint is the real id, and wharf turns it into a stable handle and color so the same key is always the same brave-otter in the same shade. Two strangers who never signed up now have distinct, durable identities.

Use the fingerprint as a key into a store and you get per-user state with no login:

b := s.Bucket("visits") // namespaced to this app
raw, _, _ := b.Get(s.User.Fingerprint)
// read/unread, bookmarks, "continue where you left off", a history
// that is simply there when they reconnect tomorrow.

It is anonymous and persistent at once. You never learn anyone's email, so you are not holding data you have to protect, but you recognize them perfectly across visits.

A room is a single goroutine

The naive multiplayer design is a shared map of members behind a mutex, and it races the moment people join and leave while messages fly.

wharf models each room as an actor. One goroutine owns the member set. Join, leave, and broadcast are messages sent to it over channels. Because exactly one goroutine ever touches that map, there are no locks and no data races by construction. Concurrency is message passing, not shared memory.

A nice second-order effect: since the room already serializes every mutation, it is the natural home for shared state. wharf has stateful rooms that hold a value and a reducer, hand each newcomer a snapshot on join, and broadcast the new state after every action. A live poll or a small game board becomes a reducer and a render, with the hard part solved one layer down.

Persistence without forcing a database on you

It would have been easy to import a Postgres driver into the core and make everyone carry it. Instead, persistence is a five-method interface. The in-memory implementation lives in the core and nothing else. Every real backend is its own Go module:

import "github.com/doganarif/wharf/store/sqlite"

st, _ := sqlite.Open("wharf.db")
wharf.New(":2222").Store(st).App("shout", shout).Run()

Importing the core pulls in no database. Importing an adapter pulls in exactly one. SQLite, Postgres, Redis, and an embedded single-file option all pass one shared conformance suite, so swappable is something I can prove rather than claim. The Postgres and Redis adapters are verified against live servers, and per-key history that works in memory survives a restart unchanged when you point it at a file or a server.

A bug that made the design click

Early on, presence was sometimes wrong: one of two connected users would not see the other arrive. Broadcasts worked, identity worked, only the join ordering was off, intermittently.

The cause was timing. Before rendering, the terminal layer asks the client about its background color and waits for a reply. Real terminals answer in milliseconds. A client that never answers makes the code wait for the timeout, and in my first version that wait happened before the session joined its room. A slow client would join late, after someone who connected after it, and events arrived out of order.

The fix was to join the room before negotiating anything about rendering. The lesson generalized: the moment you stop assuming "connected" and "in the room" happen at the same instant, a class of ordering bugs disappears.

What you can build, and the one catch

wharf is not a chat library, even though chat is the obvious demo. The primitives (identity, presence, rooms, shared state, storage) fit a lot of things. The examples I shipped are a collaborative drawing canvas where you watch other cursors move, a keyless chat room, and a live poll. The same shapes cover guestbooks, small multiplayer games, status boards, internal tools your team reaches with keys they already have, and an assistant you talk to in your terminal.

The honest catch: SSH is pull-only. Nobody re-runs a command on a Tuesday for no reason, so an SSH app cannot tap someone on the shoulder. The experience is great and the retention is near zero unless you add a way back in, usually a small web mirror for discovery or a notification when there is an actual reason to return. The terminal is the experience, not the reminder.

Try it

go run github.com/doganarif/wharf/cmd/wharf@latest
# then, in two terminals:
ssh -p 2222 canvas@localhost
ssh -p 2222 chat@localhost

Repo (Go, MIT): https://github.com/doganarif/wharf

If you build something odd with it, I would like to see it.

Shipping 100,000 construction PDFs a month: what actually breaks

arif — Thu, 18 Jun 2026 22:07:44 +0000

After a year running a document processing pipeline through hundreds of thousands of construction documents (tender packs, permit applications, site surveys, BIM exports, drawing sets at A0 and larger), I can tell you what actually breaks.

It is not the PDFs.

That is the thing most people get wrong about systems like this. Every PDF tutorial focuses on the parser: which library, which model, which extraction service. After a year, the PDFs themselves rank third on the list of things that break. The first is the error taxonomy. The second is coordination between documents. The actual content of the files is, mostly, a tractable engineering problem with off-the-shelf tools.

TL;DR

One pipeline run per document, not per upload. Per-document isolation pays for itself the first time a corrupt PDF lands in a 2,000-file archive.
PDF malformation has a bounded long tail. Five categories cover most of what production traffic produces. Handle the categories, not the cases.
Vision LLMs are not the bottleneck. Coordination is. The completion fence is two SQL queries and it caused more incidents than the model layer ever did.
Large-format pages require tiling before submission. A0 drawings at 300 DPI exceed every vision API's pixel budget. Detect at the geometry level, subdivide into a grid, and batch-submit tiles through a separate quota pool.
The most effective intervention against hallucinated numbers is not a better model. It is grounding. Feed the model the deterministically-extracted text alongside the image and it becomes a layout mediator rather than a content generator.

A hundred thousand documents a month is one new document every twenty-six seconds on average, in bursts of several thousand at a time. Most are uneventful. The ones that are not built this pipeline.

The shape of the pipeline

ZIP upload ──► extract recursively ──► fan-out (one run per file)
                                            │
                                            ▼
                            ┌─────────────────────────────┐
                            │ format normalization        │
                            │   (Office → PDF, flatten)   │
                            ├─────────────────────────────┤
                            │ extraction                  │
                            │   (parser / OCR / vision)   │
                            ├─────────────────────────────┤
                            │ enrichment                  │
                            │   (classify, embed)         │
                            ├─────────────────────────────┤
                            │ persistence                 │
                            │   (vector + search + DB)    │
                            └─────────────────────────────┘
                                            │
                                            ▼
                            project-wide completion fence

The architecture is conventional. The interesting parts live in the seams between stages and in the orchestration around them.

One run per document

The first instinct is to batch. Receive a ZIP of two thousand files, process them together, commit them together. Batching minimises overhead. It also couples receipt, processing, and commit into a single failure domain, which is the opposite of what you want when input variance is this high. A single corrupt PDF inside the batch fails the batch. Retries become coarse. Observability collapses into "this upload did not finish." The user sees an error on a project that contains one bad file and 1,999 good ones.

We run one pipeline run per document.

The coordinator receives a ZIP, expands it recursively to a configured nesting limit, uploads each file to object storage, and dispatches one child pipeline run per file. The launch is fire-and-forget through the orchestrator's task API. The per-file descriptor passed between stages is intentionally small: a storage key, a filename, a relative archive path, an extension, a size, a nesting flag. No bytes. Each child worker fetches its own file from storage.

This decouples extraction throughput from processing memory. The coordinator's working set stays flat regardless of whether the archive contains fifty documents or fifty thousand. Child pipeline runs execute on uniformly sized container tasks (4 vCPU, 30 GiB RAM, 100 GiB ephemeral disk), so capacity planning is one number multiplied by concurrency.

Within a child run, the stages are strictly sequential. Threading inside a run would muddle error attribution, complicate the status state machine, and burn file descriptors. Parallelism lives between runs, not inside them.

Every stage is wrapped by the same helper. It opens a trace span, awaits the stage coroutine, and on failure calls one of two status writers before re-raising:

async def _run_step(name: str, coro, doc_id: int):
    with trace_span(name):
        try:
            return await coro
        except UnprocessableError as e:
            db.mark_unprocessable(doc_id, name, str(e))
            raise
        except Exception as e:
            db.mark_failed(doc_id, name, str(e))
            raise

The choice between the two writers is the architectural decision that took the longest to get right. A document the extraction service classifies as permanently unprocessable (DRM-locked, zero-byte, malformed) gets a different label in the database than a document that timed out talking to an API. On-call sees the difference at a glance and does not waste a retry on a file that will never succeed.

Status writers function as a product-level state machine. Failure modes become product states.

Before any work runs, the child inserts a database row with status processing and updates the run's display label to match the row's ID. If the run crashes anywhere downstream, the row stays, with the step name and stack trace attached. Nothing disappears between systems.

PDFs we did not write

The PDF specification is permissive and two decades old. Most generators violate parts of it. A few violate most of it. Each one looks valid until you try to read it.

Construction PDFs are particularly varied. A typical project contains:

Tender documents (long, text-heavy, sometimes with embedded scans)
Permit applications (form-heavy, populated through Acrobat or web tools)
Site surveys (scanned or photographed, sometimes rotated)
Construction drawings (A1, A0, vector-heavy with thousands of dimension annotations)
BIM-exported sheet sets (huge, layered, often with missing fonts)
Material specifications (mixed text and tables)

After a year, the failure modes are stable. Five categories cover most of it: form-heavy, scanned, structurally corrupt, oversized, and mislabeled.

Form-heavy

Open a permit application in a browser and you see numbers. Extract the text programmatically and you see Field1: [empty]. The visible content lives in form widgets, not page content; the two are stored separately inside the file, and a parser that does not know to look for widget annotations returns the empty shell.

The flattener inspects every page for widget and free-text annotations. If any exist, PyMuPDF bakes them into static page content and re-serialises the file with its highest garbage-collection pass: unreferenced objects removed, object numbers compacted, cross-reference table rebuilt.

def needs_flattening(doc: fitz.Document) -> bool:
    for page in doc:
        if page.widgets() or page.first_annot is not None:
            return True
    return False

def flatten(doc: fitz.Document) -> bytes:
    for page in doc:
        page.apply_redactions()  # bakes widget values into content
    return doc.tobytes(garbage=4, deflate=True, clean=True)

A particular subtlety surfaced in production: non-UTF-8 annotation names cause PyMuPDF to throw SystemError, UnicodeDecodeError, or RuntimeError from inside the widget enumeration call. The detection catches all three and uses them as signal.

def needs_flattening_safe(doc: fitz.Document) -> bool:
    try:
        return needs_flattening(doc)
    except (SystemError, UnicodeDecodeError, RuntimeError):
        return True  # crash on enumeration = file definitely needs flattening

A parser crash on widget enumeration is, in practice, a near-certain indicator that the file needs flattening. The crash became the heuristic.

Scanned

Site surveys arrive looking like real PDFs. They are not. There is no text layer. A naive parser returns an empty string and reports success. The pipeline detects scans before any extraction attempt with a text-density heuristic.

SCAN_THRESHOLD = 0.5  # at most half the pages have text -> treat as scan

def is_scanned(doc: fitz.Document) -> bool:
    if len(doc) == 0:
        return False
    with_text = sum(1 for page in doc if page.get_text().strip())
    return with_text / len(doc) <= SCAN_THRESHOLD

The threshold is set above zero so mixed documents (a scanned cover sheet stapled to typed pages, common in tender packs) route correctly rather than failing the binary check on page one. OCR output is written back to the same storage key so reprocessing skips the OCR step on a second attempt.

Structurally corrupt

Print-to-PDF drivers occasionally emit documents with one or more pages containing empty content streams. The extraction service signals permanent failure and refuses the whole file. The pipeline does not give up. It scans every page for zero-length content streams, removes the offending pages, rebuilds the file, and retries.

def repair_empty_pages(doc: fitz.Document) -> bool:
    """Returns True if the document was modified."""
    bad = [i for i, page in enumerate(doc) if not page.read_contents()]
    for i in reversed(bad):  # delete in reverse so indices stay valid
        doc.delete_page(i)
    return bool(bad)

A forty-page drawing set with two corrupt pages becomes a thirty-eight-page document that succeeds.

The general principle here is worth naming: do not try to predict which inputs are corrupt. Let the service tell you, then repair surgically. The same principle returns later for rate limits.

Oversized

When the extraction service signals page-count exceeded or payload too large, the pipeline splits the document into hundred-page chunks and submits each independently with up to five attempts per chunk (exponential backoff, 4 to 90 seconds). The full document is always tried first. Chunking is reactive, not proactive. Most files are not large; the ones that are land in their own code path without slowing down the median.

The waterfall

These behaviours feed a priority waterfall over three extraction tiers:

Primary structured extraction service. Handles the common case. Returns text blocks with bounding boxes, tables, and figure renditions.
Secondary extraction service. Takes over on permanent failure from tier 1. Its own retry budget.
Vision-model fallback. Activates if tier 2 also exhausts.

Three tiers down and still failing, the document is marked permanently unprocessable and surfaced to the user. A retry at one tier never consumes the budget of another. This sounds obvious until you find a system where it does.

Office files (.docx, .doc, .xlsx, .xls, .pptx, .ppt) are converted to PDF through a cloud conversion API before they enter the extraction path. Three task-level retries with 10, 30, 60 second delays. Conversion errors are classified into permanent (genuinely corrupt source files) and transient (API issues) using the API's response metadata. Email files are converted in-process using a local parser and a PDF rendering library: header and body onto A4 pages, no external API call. The in-process path keeps email content off third-party infrastructure. Customers do not always realise their PDF pipelines contain their inbox.

Tiling large-format pages

A single A0 sheet rasterised at 300 DPI produces an image roughly 9900 x 14000 pixels: around 140 megapixels. Vision APIs enforce hard limits on both total pixel count and the length of any single side. Sending the raw image does not produce an error in every case. Some endpoints silently crop or downsample before inference, which means the model sees a different image than the one you submitted, with no signal that anything went wrong. The only safe path is to detect large-format pages before rasterisation and subdivide them.

Detection at the geometry level

PDF coordinates are measured in points (1 pt = 1/72 inch). PyMuPDF exposes page dimensions directly:

_LARGE_FORMAT_MIN_POINTS = 1190  # long edge of A3 landscape in PDF points

def is_large_format_page(page) -> bool:
    return max(page.rect.width, page.rect.height) >= _LARGE_FORMAT_MIN_POINTS

The threshold is 1190 points because A3 landscape measures 1190 x 842 pt. Any page whose longest side meets or exceeds that value is treated as a large-format page and routed to tiling. PDF metadata is not used for this decision. Pages exported from BIM tools routinely carry an A4 /MediaBox while the actual geometry describes an A1 drawing.

Tile grid construction

The grid divides the page into equal-sized cells, each no larger than the threshold constant:

import math

def compute_tile_grid(width_pt: float, height_pt: float):
    cols = max(1, math.ceil(width_pt / _LARGE_FORMAT_MIN_POINTS))
    rows = max(1, math.ceil(height_pt / _LARGE_FORMAT_MIN_POINTS))
    tile_w = width_pt / cols
    tile_h = height_pt / rows
    return rows, cols, tile_w, tile_h

An A1 page (roughly 1684 x 2384 pt) produces a 2 x 2 grid (4 tiles). An A0 page (roughly 2384 x 3370 pt) produces a 2 x 3 grid (6 tiles). Tiles are edge-to-edge with zero overlap. This means any entity whose geometry straddles a tile boundary (a dimension string, a room label, a wall line) is split across two tiles. There is no stitching pass after inference. The grounding mechanism described in the next section is what makes split entities recoverable: if the text layer was extracted, the dimension is already present verbatim in both tiles' grounding context regardless of where the tile cut falls.

Rasterisation

Tiles are rendered at a fixed 200 DPI, not the 300 DPI used for normal pages:

import fitz  # PyMuPDF

def prepare_tile(page, clip_rect: fitz.Rect) -> bytes:
    mat = fitz.Matrix(200 / 72, 200 / 72)
    pix = page.get_pixmap(matrix=mat, clip=clip_rect)
    raw = pix.tobytes("png")
    return _downscale_if_needed(raw)  # enforces 3072 px longest-edge cap

200 DPI is chosen because each tile is already a sub-region of the full page. At that resolution, a 1190 pt tile rasterises to roughly 3306 pixels on that edge, which exceeds the 3072 px per-dimension cap. The _downscale_if_needed guard re-encodes the tile to fit before submission. Using 300 DPI on the same tile would produce an even larger intermediate image, burning CPU for no benefit. Normal (non-tiled) pages start at 300 DPI and scale down adaptively to stay within a pixel-area budget just under 180 million total pixels and a longest-edge cap of 3072 pixels.

Tiles are encoded as RGB PNG. PNG is lossless and avoids format-rejection errors from CMYK or palette-mode inputs that some vision APIs refuse with a 400.

Batch submission and polling

Tiled pages are not submitted through the synchronous inference path. They go to a batch inference endpoint backed by a separate quota pool. Batch requests draw on a quota that is independent of the real-time API, so a burst of A0 drawings does not saturate the quota used by the rest of the pipeline.

Each batch job gets a unique scoped prefix in object storage. Tile images are uploaded in sequential insertion order. A manifest is written containing one request object per tile, each referencing its storage reference and inference parameters. The batch platform writes results to the same prefix as one or more result files.

The poller checks job status every 30 seconds. The timeout ceiling is 6 hours. On timeout, the job is cancelled and a TimeoutError is raised. The staging prefix is left intact for inspection.

Reassembly

Output lines from the batch are not guaranteed to arrive in submission order. Each input request embeds an identifier that maps back to its insertion index. The collector builds a reverse-lookup map from identifier to index and places each response into a pre-allocated list at the correct position:

id_to_index = {record.id: idx for idx, record in enumerate(records)}
descriptions = [""] * len(records)

for line in output_lines:
    request_id = line["request_id"]
    idx = id_to_index[request_id]
    descriptions[idx] = line["response"]["text"]

The final list is in original tile order, which maps directly to reading order across the page grid.

Partial success is a hard failure

If any tile slot is still empty after all output files are consumed, the batch runner raises RuntimeError and returns nothing:

produced = sum(1 for d in descriptions if d)
if produced < len(records):
    raise RuntimeError(
        f"Batch incomplete: {produced}/{len(records)} tiles returned"
    )

A document missing one tile out of six would look complete to a downstream consumer. In a construction context that means a floor plan with a silent gap in the extracted data. The pipeline raises rather than persist that state. The task-level retry will requeue the entire document.

Cleanup

The staging prefix is not deleted in the error path. A storage lifecycle policy reclaims any prefix older than a fixed TTL. The TTL covers the longest plausible batch job window plus inspection time. This keeps error-path code simple and ensures orphaned prefixes from crashed workers are cleaned up without any explicit handler.

Blueprint understanding

A single A1 sheet combines dimension chains at 0°, 90°, and 45° rotation; room labels; hatch patterns; section cut symbols; and revision clouds, all at different scales within the same sheet. The text density is high but non-uniform. Symbols carry meaning that differs by convention and trade. A "300" in one region is a room number; in another it is a dimension in millimetres.

Pure vision inference on these inputs produces two failure modes. First, the model misreads values that are visually ambiguous at the rendered resolution (rotated text, condensed fonts, overlapping annotations). Second, the model invents plausible-looking numbers when it is uncertain. In construction, a wrong dimension is a specification error. It enters procurement quantities and fabrication orders. The liability exposure from a single incorrect quantity in a bill of materials is orders of magnitude larger than a missing word in a document summary.

The hybrid path

When the structured extraction service processes a PDF successfully, it returns both the rendered page images and a list of text blocks, each with a bounding box in PDF point coordinates. The pipeline does not discard this structured output when it calls the vision model. It uses it as ground truth.

Spatial join

For each figure (a rendered tile or a figure rendition), the pipeline finds all text blocks whose bounding boxes overlap the figure's region:

def bounds_overlap(a: list[float], b: list[float]) -> bool:
    # a and b are [x0, y0, x1, y1] in PDF points
    return not (a[2] < b[0] or b[2] < a[0] or a[3] < b[1] or b[3] < a[1])

def collect_grounding(
    figure_bounds: list[float],
    text_blocks: list[dict],
    page: int,
) -> list[str]:
    parts = []
    for block in text_blocks:
        if block["page"] == page and bounds_overlap(figure_bounds, block["bounding_box"]):
            parts.append(block["text"])
    return parts

No spatial index is needed. With tens to low hundreds of text blocks per page, a linear scan adds under a millisecond per figure. The result is the set of text blocks whose bounding boxes overlap the figure's region.

Grounded prompt assembly

The collected strings are deduplicated before injection. Any word appearing more than 5 times across the concatenated text is collapsed to a single occurrence, and identical lines are removed. This handles the common case of warehouse floor plans and grid-heavy drawings where a single label repeats across every bay.

from collections import Counter

def deduplicate_grounding(parts: list[str]) -> str:
    combined = "\n".join(parts)
    words = combined.split()
    freq = Counter(words)
    seen_words: set[str] = set()
    filtered_words = []
    for w in words:
        if freq[w] > 5:
            if w in seen_words:
                continue
            seen_words.add(w)
        filtered_words.append(w)
    raw = " ".join(filtered_words)
    seen_lines: set[str] = set()
    lines = []
    for line in raw.splitlines():
        if line not in seen_lines:
            seen_lines.add(line)
            lines.append(line)
    return "\n".join(lines)

def assemble_grounded_prompt(base_prompt: str, grounding_text: str) -> str:
    if not grounding_text.strip():
        return base_prompt
    return (
        f"{base_prompt}\n\n"
        f"Extracted text in this region:\n{grounding_text}"
    )

With grounding in place, the vision model maps regions and describes visual structure. The numbers come from the text layer, not from model inference. The model cannot invent a number that was not in the grounding text.

The grounding text is also appended to the stored segment output, not just the prompt. This means the downstream vector index writer receives both the vision narrative and the verified dimension values in the same chunk. Retrieval does not depend on the model having faithfully reproduced the number; the number is present in the chunk regardless.

The 429 chain under blueprint-heavy load

Blueprint-heavy workloads arrive in bursts. A tender pack or a permit application can contain dozens of A0 and A1 sheets. When several such archives land at once, the tiling path generates hundreds of synchronous vision calls in a short window.

The retry logic operates at two levels. The inner level handles transient quota exhaustion on a single model:

from tenacity import retry, stop_after_attempt, wait_random_exponential
from tenacity import retry_if_exception

def is_rate_limit(exc: Exception) -> bool:
    return getattr(exc, "status_code", None) == 429

@retry(
    retry=retry_if_exception(is_rate_limit),
    wait=wait_random_exponential(multiplier=1, max=20),
    stop=stop_after_attempt(3),
    reraise=True,
)
async def call_vision_model(model: str, prompt: str, image_bytes: bytes) -> str:
    ...  # vendor call

Three attempts, randomised exponential backoff, maximum 20 seconds between retries. If all three fail, the exception propagates. Randomisation prevents the thundering herd pattern where multiple workers retry in lockstep.

The outer level rotates through a list of 4 model candidates on both HTTP 429 (after the inner retries are exhausted) and HTTP 503 (immediately, without consuming the retry budget):

async def describe_with_rotation(
    model_candidates: tuple[str, ...],
    prompt: str,
    image_bytes: bytes,
) -> str:
    last_exc: Exception | None = None
    for model in model_candidates:
        try:
            return await call_vision_model(model, prompt, image_bytes)
        except RateLimitError as exc:
            # inner retries exhausted; move to next model
            last_exc = exc
            continue
        except ServiceUnavailableError as exc:
            # 503: skip immediately, no retry budget consumed
            last_exc = exc
            continue
    raise VisionServiceError("all model candidates exhausted") from last_exc

Each candidate in the rotation list draws on an independent quota pool. A 429 on one model does not predict a 429 on another. A 503 indicates a model-level availability issue, not a quota issue, so retrying the same model would waste time.

The outer task-level retry (5 attempts at 10, 30, 60, 120, and 300 seconds) handles failures that survive both the inner backoff and the model rotation, such as a batch job timeout or a network partition.

Why wrong numbers are the failure mode that matters

A missing tile means a blank region in the extraction output. That is detectable: the document has fewer segments than expected, downstream validation flags it, the task retries. A hallucinated dimension is undetectable without the source document. It looks like a valid result. It passes schema validation. It enters retrieval. If a quantity surveyor or a structural engineer queries it and acts on the number, the error has left the system.

Construction drawings with one wrong measurement are litigation problems. Grounding makes the pipeline's numeric accuracy independent of model behavior on any given input.

Orchestration under load

At a hundred thousand documents a month, coordination failures between pipeline runs cause more incidents than malformed documents do. The PDFs are the easy part. The coordination between PDFs is the system.

The first sign something was wrong was PoolTimeout errors appearing before a single child pipeline run had started. The first time a 5,000-file archive landed, every launch request went into the same HTTP connection pool and the pool ran out before the first child flow began. The arithmetic was fast and ugly: five thousand task-launch requests, a connection pool sized for ten, an orchestrator API that sustains roughly ten concurrent launches.

The coordinator now caps in-flight launches and rate-limits them with a semaphore and a sleep inside the same block.

launch_semaphore = asyncio.Semaphore(10)

async def launch_child(file_meta):
    async with launch_semaphore:
        await asyncio.sleep(0.1)
        await orchestrator.dispatch(file_meta)

The semaphore controls connection pool depth. The sleep controls burst rate. Concurrency control and rate limiting look like the same problem until they fail.

After fan-out, the coordinator waits for the children. Same problem in reverse: five hundred simultaneous long-poll coroutines would hold five hundred connections open. The coordinator groups child run references into batches of fifty and awaits each batch with asyncio.gather(return_exceptions=True).

BATCH_SIZE = 50

async def wait_for_all(runs):
    results = []
    for i in range(0, len(runs), BATCH_SIZE):
        batch = runs[i:i + BATCH_SIZE]
        results.extend(await asyncio.gather(
            *(wait_for_run(r) for r in batch),
            return_exceptions=True,
        ))
    return results

Exceptions are collected per child rather than propagated, so a single failure does not abort the wait for the rest of the batch.

Backpressure between the coordinator and the cluster is handled by the orchestrator's own concurrency settings, not by the coordinator. Container task capacity forms the lower bound. Adding a coordinator-side cap would create another constant to tune as load changes.

The vector index writer is the one place backpressure is deliberately turned off. During indexing bursts, the writer bypasses the index service's built-in flow control and absorbs transient overload with task-level retries instead. Service-side throttling would smooth the load curve and protect the storage tier from peaks. Higher peak load on the storage tier is an acceptable tradeoff for predictable end-to-end latency. Users observe indexing latency directly.

Completion is harder than fan-out. Multiple ZIP archives for the same project can arrive concurrently, and none of them can signal "project ready" until every archive has finished. The fence is two database queries:

-- 1. mark this upload processed (idempotent on second call)
UPDATE uploads
SET    processed_at = NOW()
WHERE  project_id = $1
  AND  upload_id  = $2
  AND  processed_at IS NULL;

-- 2. count remaining unprocessed uploads for this project
SELECT COUNT(*)
FROM   uploads
WHERE  project_id = $1
  AND  processed_at IS NULL;

Before the fence closes, the pipeline checks whether this is the last outstanding upload; if so, a cross-document analysis step runs synchronously before the completion timestamp is stamped, so results are visible the moment the project is marked ready. Only when the count reaches zero does a final call fire to mark the project indexed and send a notification. The queries are idempotent. The downstream analysis step is gated by the same fence, so it sees a complete snapshot rather than a moving target.

Boring SQL. Surprisingly load-bearing. The fence is what saves you.

Observability is wired into the database, not into a separate log pipeline. Each document pipeline run writes its run identifier into the database before processing starts and updates its display label to match the database-assigned document ID. The link is bidirectional and queryable without polling. Treating the database as the source of truth and tracing as decoration is contrarian; most teams do the opposite. It is also the right call for any batch workload where the database already tracks state for product reasons. Two sources of truth is one too many.

Distributed tracing spans the full pipeline. Span data is buffered in process and exported at run completion; a crash before export loses that run's spans. Export failures are silently discarded so an observability outage cannot kill a document run.

The completion notification carries total file count, success count, failure count, and up to ten failed document records with step name and full stack trace. Failed records are fetched from the database at notification time rather than accumulated in memory, which keeps the coordinator's working set bounded regardless of failure rate.

What scales

Two things in this system cost the most to get right, and they cost the most for the same reason: the feedback loop runs at the speed of production traffic.

The error taxonomy, because every decision about whether to retry depends on it. Mark a failure permanent and a week later you find out the fix was a one-line threshold change. Mark it transient and you queue retry work that will never succeed. Neither wrong answer announces itself.

The completion fence, because partial signals are invisible. A downstream consumer reading a half-indexed project does not know it is reading a partial result.

Both belong to a category of bug that no amount of load testing surfaces until production traffic starts. At this scale, the cleaner your test set, the more dangerous the gap.

Everything else is the cost of accepting input from the world.

Control plane, data plane, durable storage: structuring a managed search engine in Rust

arif — Fri, 05 Jun 2026 09:56:44 +0000

When you ask a search service to "create an index," it reads like a single write to a single place. While building a managed search engine in Rust this year, I started treating that one word as three separate facts, each owned by a different system: the record that an index exists, the index itself, and a durable copy for when the machine holding it dies.

Pulling those apart changed how I reason about failure and scaling, and it sharpened my sense of what a relational database is actually good for. Here is the structure I settled on, and the path a single create request takes through it.

  create request
       |
       v
  Control plane   .  Postgres          does it exist? who owns it? quota
       |
       |  commit first (cheap, authoritative)
       v
  Data plane      .  tantivy           the inverted index, BM25, local disk
       |
       |  materialize second (expensive, can fail)
       v
  Durability      .  object storage    authoritative copy, survives the node

Three planes, one job each

The control plane is Postgres. It holds metadata and nothing else: which indexes exist, which tenant owns each one, the field configuration, and the quota. The table is tiny.

CREATE TABLE indexes (
    id         TEXT PRIMARY KEY,   -- a uuid, the real identifier
    org_id     TEXT NOT NULL,      -- the owning tenant
    name       TEXT NOT NULL,      -- a human label, not unique
    settings   JSONB NOT NULL,     -- field config, stored as an opaque blob
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX indexes_org_idx ON indexes (org_id);

Look at what is missing. There is no document text and no tsvector. The settings column is JSONB, but I never query into it. It gets read back whole and parsed in the application. Postgres knows that an index exists and who owns it. It has no idea how to search one.

That indexes_org_idx is worth an aside, because the word "index" trips people up here. It is a plain B-tree that speeds up "list the indexes for this tenant." It has nothing to do with full-text search. Two unrelated meanings of the same word, sitting in the same paragraph.

The data plane is tantivy, a Lucene-class search library for Rust. The real inverted index lives here, on local disk, memory-mapped. Document text, the BM25 postings, the term dictionary, the stored originals: all on disk in the data plane, none of it in Postgres.

Durability is object storage, an S3-compatible bucket. I treat the local tantivy files as a working copy. The authoritative durable copy lives in the bucket. If the box holding the working copy vanishes, the next boot rebuilds it from object storage.

One more piece sits in memory: a map from index id to its open tantivy handle, so a query does not reopen files on every request. That map is not a source of truth. It is reconstructed from the control plane at startup.

Walking a create through all three

The HTTP handler does almost nothing interesting. It checks that the API key has write permission and belongs to a tenant, then hands off to the index manager. The ordering inside the manager is where the design lives.

pub async fn create(&self, org_id: &str, settings: IndexSettings) -> Result<String> {
    validate(&settings)?;          // name length, language, vector dims
    let id = new_uuid();

    // 1. Control plane commits first. This insert is also the quota gate.
    if !self.catalog.insert(&id, org_id, &settings).await? {
        bail!("index quota reached");
    }

    // 2. Data plane materializes second.
    let index = match FtsIndex::open(&self.dir_for(&id), settings.clone()) {
        Ok(index) => index,
        Err(e) => {
            // Opening the on-disk index failed. Roll back the catalog row
            // so we never leave a record pointing at nothing.
            self.catalog.delete(&id).await?;
            return Err(e);
        }
    };

    // 3. Publish the open handle so queries can find it.
    self.live.write().insert(id.clone(), Entry { org_id: org_id.into(), index });
    Ok(id)
}

The shape here is simple to say: commit the cheap authoritative thing first, materialize the expensive thing second, and undo the first if the second fails. The catalog row is a few bytes in a transactional database. Opening a tantivy index touches the filesystem and can fail for boring reasons like no disk, bad permissions, or a corrupt directory. Writing the row first means the quota is enforced atomically. Rolling it back on failure avoids the worst state in this whole system: a row in Postgres that claims an index exists with no data behind it.

After the manager returns, the handler does one more thing. It syncs the new index directory to object storage, and if that sync fails it returns 503 instead of 200. I would rather tell the caller "this did not durably succeed" than report success for something that only exists on one disk.

The quota check is a single statement

The quota gate looks small, and getting it wrong is a classic race. The naive version reads the count, compares it to the limit, then inserts:

count = SELECT count(*) ...        // says 4 of 5 used
if count < limit: INSERT ...       // ok, insert

Run two creates at the same moment and both read "4 of 5," both pass the check, and both insert. Now the tenant has 6 indexes on a limit of 5. The check and the write were two steps with a gap between them, and concurrency lives in that gap.

The fix is to make the check and the insert the same statement, so the database serializes them:

INSERT INTO indexes (id, org_id, name, settings)
SELECT $1, $2, $3, $4
WHERE (SELECT count(*) FROM indexes WHERE org_id = $2)
    < (SELECT max_indexes FROM organizations WHERE id = $2);

If the tenant is at the limit, the WHERE is false, no row goes in, and rows_affected() comes back as 0. The application reads that zero as "quota reached." There is no window for two creates to both win, because there is no separate read to race against.

Why split metadata from the index at all

The split costs something. Every create now writes to two systems, and I have to keep them consistent. The payoff shows up in three places.

Search traffic never touches the control-plane database. A burst of queries hits the in-process tantivy reader and the local disk. It does not compete for connections with the database that authenticates requests and enforces quotas. The thing under the most load and the thing that must stay correct are physically different systems, and they scale on different axes.

Failure has a clear blast radius. Losing the data plane is annoying but recoverable, since the durable copy is in object storage and the catalog still knows what should exist. Losing the control plane is the real outage, which is exactly why I keep it small, transactional, and boring. A few kilobytes per index in Postgres is cheap insurance for the gigabytes of index data it describes.

Boot becomes a reconciliation. On startup the engine reads every row from the catalog and, for each one, opens the local copy or restores it from object storage if the local copy is missing. Then it sweeps the bucket for leftover data that has no matching catalog row, which is how a delete that got interrupted halfway gets cleaned up. The control plane is the list of what should exist, and startup makes reality match the list.

What a field actually turns into

One detail from the data plane is worth showing, because it explains why search and a database want different things from the same string. When I create a text field that is both searchable and filterable, it becomes two physical fields in tantivy:

title       ->  analyzed field   (tokenized, lowercased, stemmed)   for BM25 matching
title__kw   ->  keyword field    (the raw string, untouched)        for exact filters

Full-text matching wants "Running" to find "runs," so it tokenizes and stems. An exact filter on a brand name wants "ACME" to mean exactly "ACME," with no stemming and no folding. The same input needs opposite treatment depending on the question, so it gets stored twice, once for each job. Vectors are a third case. They never enter the tantivy schema at all; each vector field gets its own approximate-nearest-neighbor store on disk, beside the text index but separate from it.

What it costs to run this way

The architecture has real rough edges, and I would rather name them than pretend otherwise.

Writes commit synchronously. Each batch fsyncs and reloads the reader so documents are searchable the instant the call returns. That is good for correctness and bad for tiny frequent writes, so bulk loads have to batch thousands of documents per request or throughput falls off a cliff.

Cold starts pay for the durability. After a deploy, an index restores from object storage and pages in from disk, so the first handful of queries run slower until the cache warms. Warm and cold are genuinely different latency regimes.

A single index lives on a single node. There is no sharding of one index across machines yet. That is fine for tens of millions of documents on one large server and wrong for trillions. It is a deliberate ceiling, and worth stating plainly rather than hiding.

The takeaway

If there is one rule worth keeping from this, it is the ordering. Commit the cheap, authoritative fact first. Materialize the expensive thing second. Roll back the first if the second fails.

The record that an index exists belongs in a small transactional database. The index itself belongs in a purpose-built search library on fast local disk. The durable copy belongs in object storage. Wire a create through the three in that order and you get independent scaling and clean recovery for very little extra code, while the database goes back to doing the one thing it is best at: being the boring, correct source of truth for what exists.

I build backend systems with this kind of separation: control planes, search, and correctness under concurrency. If that is the problem in front of you, get in touch.

I Benchmarked 7 LLMs on Cross-Domain MCP Orchestration. All 7 Found the Same Gap.

arif — Mon, 09 Mar 2026 03:38:02 +0000

The Problem

The Model Context Protocol (MCP) has 106 official servers and 50+ academic papers. But every single study evaluates one server at a time. Nobody has asked: what happens when you compose multiple MCP servers for cross-domain tasks?

I decided to find out.

What I Did

Experiment A: Real MCP Tool Calls

I connected 6 MCP servers simultaneously:

arXiv (academic preprints)
PubMed (biomedical literature)
Firecrawl (web search and scraping)
Context7 (library documentation)
Memory (persistent knowledge graph)
Filesystem (file operations)

Then I ran 3 cross-domain case studies with 17 real tool calls and 0% failure rate.

The key discovery: insights that are impossible within any single server become visible when you combine them. For example, arXiv papers on AI mental health chatbots and PubMed clinical trials on the same topic turned out to be two disconnected research communities working on the same problem with zero cross-citations. This gap was only visible by querying both databases and linking the results through a knowledge graph.

Experiment B: 7-Model Benchmark

I took the data collected from those MCP servers and sent identical prompts to 7 different LLMs:

Model	Latency	Tokens	KG Entities	KG Relations
GPT-5.4	54.7s	2,352	14	15
DeepSeek R1	33.9s	4,296	6	4
Mistral Large 3	6.5s	1,857	9	8
Llama 4 Maverick	3.0s	1,374	3	3
Gemini 2.5 Flash	15.6s	4,592	12	11
Claude Sonnet 4.5	21.3s	2,411	13	11
Claude Haiku 4.5	9.3s	2,136	8	6

Every model successfully produced 5 cross-domain insights. But the depth varied wildly: GPT-5.4 built a knowledge graph with 14 entities and 15 relations, while Llama 4 Maverick produced only 3 entities in 3 seconds.

The Finding All 7 Models Agreed On

Here is the most interesting part. Without being told what to look for, all 7 models independently identified the same research gap:

LangChain provides MultiServerMCPClient (the mechanism). Benchmarks evaluate individual tool calls. But nobody documents composition patterns for how to effectively orchestrate multi-server workflows.

I call this the mechanism-pattern gap. It is like how HTTP existed for years before REST patterns told people how to use it effectively.

5 Composition Patterns

From my experiments, I identified 5 recurring patterns:

Sequential Pipeline - Server A output feeds Server B query
Parallel Fan-Out - Same query to multiple servers at once
Cross-Reference Verification - Validate findings across servers
Iterative Refinement - Cross-server context narrows queries
Domain Bridging - Synthesize insights from unrelated domains

Domain Bridging is the most valuable. It produces insights that exist in neither source alone.

Try It Yourself

All servers are open source and require no API keys:

uvx arxiv-mcp-server
npx @cyanheads/pubmed-mcp-server
npx @modelcontextprotocol/server-memory
npx firecrawl-mcp
npx @upstash/context7-mcp
npx @modelcontextprotocol/server-filesystem

The benchmark script and full results are on GitHub:

github.com/doganarif/mcp-bench

Paper: Zenodo DOI

If you are an arXiv endorser for cs.SE and this seems reasonable, I would appreciate the help: endorsement link

I am Arif Dogan, a software engineer and independent researcher in Berlin. I build things with Go, Python, and LLMs. arif.sh | GitHub

AI Agent Failures Are Distributed Systems Failures. Here's the Complete Mapping.

arif — Fri, 20 Feb 2026 01:57:01 +0000

A few months into building an AI agent pipeline for a fintech client, we had a silent failure that cost us three days.

The agent processed a document. Returned a confident-looking response. No error, no exception, no log entry that suggested anything was wrong. That output went into the next step, which used it to write a decision record. The decision record went downstream. Three steps later, a human reviewer flagged something that did not add up.

The root cause was a hallucinated intermediate field. One field. The model had made up a plausible-sounding value for something it should have extracted from the document. Everything downstream had treated that invented value as real.

I had seen this failure mode before. Not in AI. In distributed systems.

The microservice that returns 200 OK while writing corrupted data. The queue consumer that marks a message processed before finishing the work. The retry that fires twice because the ACK never arrived.

Same pattern. Different worker.

The mental model that changes everything

An AI agent is not a program that figures things out. It is a graph of tasks, each processed by a nondeterministic worker. The worker is expensive, slow, and occasionally wrong in ways that look correct. Everything else about the system, including how it should be built, is a distributed systems problem.

Here is the complete mapping:

Distributed systems concept	AI agent equivalent
Message queue	Task buffer between agent steps
Dead letter queue	Failed tasks awaiting human review
Idempotency key	Deterministic task ID preventing duplicate LLM calls
Circuit breaker	Model fallback when primary LLM degrades or rate-limits
Saga pattern + compensation	Multi-step workflow rollback when a step fails midway
Two-phase commit	Human approval gate before any irreversible action
Distributed tracing	Per-step observability across agent execution
Backpressure	Concurrency limits on parallel LLM calls
Bulkhead pattern	Cost and rate isolation between tenants or features

Every one of these concepts exists because engineers got paged at 3am when they did not have it. AI agent engineers are getting paged for exactly the same reasons. They are building the same workarounds and calling them new things.

The people writing most of the content about AI agents come from ML backgrounds. They have never been responsible for a distributed system that split-brained in production. They are reinventing concepts that have names.

Problem 1: Silent failures that compound across steps

This is the issue nobody who writes tutorials has actually experienced in a high-stakes system. LLM workers do not fail loudly. They return well-formed, confident-looking output that is semantically wrong. In a single-step system a human catches it. In a multi-step agent, the wrong output becomes the input to the next step.

By the time you see the problem, the causal chain is four steps long and the original error is buried.

The distributed systems answer: validate at every step boundary. Do not pass raw LLM output downstream. Treat every LLM response as untrusted input until it passes a validation check.

type StepResult struct {
    Raw      string
    Valid    bool
    Schema   string
}

type Validator interface {
    Validate(output, schema string) error
}

func (w *Worker) processStep(ctx context.Context, input string, schema string) (*StepResult, error) {
    raw, err := w.llm.Complete(ctx, input)
    if err != nil {
        return nil, err
    }

    if err := w.validator.Validate(raw, schema); err != nil {
        // self-correction: one retry with the validation error in context
        corrected, err2 := w.llm.Complete(ctx, fmt.Sprintf(
            "Your previous response failed validation: %s\n\nOriginal task: %s\n\nTry again.",
            err.Error(), input,
        ))
        if err2 != nil {
            return nil, err
        }
        if err3 := w.validator.Validate(corrected, schema); err3 != nil {
            return nil, fmt.Errorf("validation failed after correction: %w", err3)
        }
        raw = corrected
    }

    return &StepResult{Raw: raw, Valid: true, Schema: schema}, nil
}

Self-correction on first validation failure recovers about 90% of cases in practice. The remaining 10% go to a dead letter queue, not to the next step in the pipeline.

Problem 2: Idempotency, the hardest problem in agent systems

This is the one I underestimated the most.

In standard distributed systems, idempotency is straightforward: assign a key, deduplicate on that key, retry safely. In agent systems, there is a complication. The LLM worker is nondeterministic. Retry with the same input and you may get a different output.

This creates a genuine design question: what does it mean to retry an agent step idempotently?

The answer that works in practice: record the output the first time the step runs. On retry, return the recorded output rather than calling the LLM again. The task ID becomes the idempotency key.

type TaskStore interface {
    Get(ctx context.Context, taskID string) (string, bool, error)
    Set(ctx context.Context, taskID string, result string) error
}

func (w *Worker) run(ctx context.Context, task Task) (string, error) {
    // check if already completed
    if result, ok, err := w.store.Get(ctx, task.ID); err != nil {
        return "", fmt.Errorf("store get: %w", err)
    } else if ok {
        return result, nil
    }

    result, err := w.processStep(ctx, task.Input, task.Schema)
    if err != nil {
        return "", err
    }

    if err := w.store.Set(ctx, task.ID, result.Raw); err != nil {
        return "", fmt.Errorf("store set: %w", err)
    }

    return result.Raw, nil
}

This also makes your workflows resumable. A pipeline that crashes midway can restart from the last completed step rather than from scratch. In the fintech system this was essential: reprocessing a document from scratch on retry risked producing a different decision record.

The one place this pattern breaks down: when you want a fresh LLM response on retry because the recorded one failed validation. The solution is to record the output only after validation passes. Failed attempts do not get persisted.

Problem 3: Multi-step workflows need a rollback plan

Here is a scenario that forced me to think about this properly.

A four-step pipeline: extract data from a document, validate it against business rules, write a decision record, trigger a downstream system. The first three steps succeed. Step four fails. What do you do?

Without explicit design, you are stuck. The decision record exists. Retrying the whole pipeline from the start would write a duplicate. Doing nothing leaves the system in a broken intermediate state.

The Saga pattern from distributed systems has a direct answer: every step that has side effects needs a compensating action that undoes it.

type Step struct {
    Name       string
    Execute    func(ctx context.Context, state *State) error
    Compensate func(ctx context.Context, state *State) error
}

type Saga struct {
    steps    []Step
    executed []int
}

func (s *Saga) Run(ctx context.Context, state *State) error {
    for i, step := range s.steps {
        if err := step.Execute(ctx, state); err != nil {
            log.Printf("step %s failed, rolling back %d completed steps", step.Name, len(s.executed))
            for j := len(s.executed) - 1; j >= 0; j-- {
                if cerr := s.steps[s.executed[j]].Compensate(ctx, state); cerr != nil {
                    log.Printf("compensation for step %s failed: %v", s.steps[s.executed[j]].Name, cerr)
                }
            }
            return fmt.Errorf("step %d (%s) failed: %w", i, step.Name, err)
        }
        s.executed = append(s.executed, i)
    }
    return nil
}

Applied to the document pipeline: if the downstream notification fails, the compensation for the decision-record step deletes it. The system returns to a clean state and the whole workflow can safely retry.

Most agent frameworks do not give you this. They give you a chain of steps and leave failure handling as an exercise for the reader.

Problem 4: Human-in-the-loop as blast radius management

One of the cleanest mental models I have encountered for human review in agent systems: scope the human-in-the-loop requirement to the blast radius of the action.

A read-only action that retrieves data: no approval needed, just log it.
A write action that can be undone: soft approval, allow with logging and rollback capability.
An irreversible action (sending an email, triggering a payment, deleting a record): hard approval gate, requires explicit human confirmation before execution.

This is not a safety net. It is an architecture decision about which operations an automated system is permitted to take unilaterally.

type BlastRadius int

const (
    BlastRadiusRead        BlastRadius = iota // read-only, no approval
    BlastRadiusReversible                     // can be undone, soft approval
    BlastRadiusIrreversible                   // cannot be undone, hard approval
)

type ActionGate interface {
    Request(ctx context.Context, action Action, radius BlastRadius) (Approval, error)
}

type Action struct {
    ID          string
    Description string
    Params      map[string]any
    Radius      BlastRadius
}

In the German government system, any output below a confidence threshold and any action classified as irreversible went to a human review queue. Not because we lacked confidence in the model, but because the downstream consequences of an error had legal weight. The auditors asked specifically about this control. Having it designed explicitly made the audit straightforward.

The engineers I see struggling most with this are the ones trying to automate 100% of cases. A more honest design goal: automate the 90-95% the system can handle confidently, and escalate the rest with full context attached.

Problem 5: Circuit breakers for model failures

LLM APIs have bad days. Rate limits, degraded performance, elevated error rates. You need a circuit breaker.

type State int

const (
    StateClosed   State = iota // normal operation
    StateOpen                  // failing fast
    StateHalfOpen              // testing recovery
)

type CircuitBreaker struct {
    mu          sync.Mutex
    state       State
    failures    int
    threshold   int
    lastFailure time.Time
    timeout     time.Duration
}

func (cb *CircuitBreaker) Allow() bool {
    cb.mu.Lock()
    defer cb.mu.Unlock()

    switch cb.state {
    case StateOpen:
        if time.Since(cb.lastFailure) > cb.timeout {
            cb.state = StateHalfOpen
            return true
        }
        return false
    default:
        return true
    }
}

func (cb *CircuitBreaker) Record(err error) {
    cb.mu.Lock()
    defer cb.mu.Unlock()

    if err != nil {
        cb.failures++
        cb.lastFailure = time.Now()
        if cb.failures >= cb.threshold {
            cb.state = StateOpen
        }
    } else {
        cb.failures = 0
        cb.state = StateClosed
    }
}

In a multi-model setup, combine the circuit breaker with a fallback: primary model trips the circuit, requests route to a cheaper backup. This keeps the system available during provider incidents. It also gives you a cheap escape valve when costs spike unexpectedly.

Speaking of costs.

Problem 6: Observability and cost attribution

In the summer of last year, a team publicly shared that they had spent $47k running AI agents in production before understanding where the money was going. The number got attention because people recognized the scenario.

LLM costs are invisible until they are not. A feature that costs nothing at 100 users is a real number at 100,000. An edge case that you never hit in testing might consume 40x the tokens of a typical case.

You cannot find this without logging every call with the context you need to understand it.

type CallTrace struct {
    TraceID          string
    FeatureName      string
    StepName         string
    Model            string
    PromptTokens     int
    CompletionTokens int
    CacheHit         bool
    LatencyMs        int64
    ValidationPassed bool
    Attempt          int
    Timestamp        time.Time
}

func tokenCost(model string, prompt, completion int) float64 {
    rates := map[string][2]float64{
        "gpt-4o":              {0.0000025, 0.000010},
        "gpt-4o-mini":         {0.00000015, 0.0000006},
        "claude-3-5-sonnet":   {0.000003, 0.000015},
        "claude-3-5-haiku":    {0.0000008, 0.000004},
    }
    r, ok := rates[model]
    if !ok {
        return 0
    }
    return float64(prompt)*r[0] + float64(completion)*r[1]
}

Store the traces. Aggregate costs by feature, by step, by user. Build a dashboard before the bill surprises you, not after.

The secondary value: when something fails in production, you have the full execution trace. Input, intermediate steps, output, validation result. In a regulated environment this is not optional. Auditors ask for it. In any environment it makes debugging tractable instead of theoretical.

What this means for frameworks

The popular agent frameworks optimize for developer experience. They make it fast to connect LLM calls, experiment with different prompting strategies, and ship a demo.

That is genuinely useful. I have used some of them for prototyping.

The problem is they tend to abstract away exactly what matters in production: queue semantics, idempotency, compensation flows, circuit breakers, cost attribution. You end up responsible for a system you do not fully understand, where the hard parts live inside library code you cannot easily inspect.

The pattern that works: use a framework to figure out what your workflow should do. When you are ready to ship, write the orchestration layer with infrastructure primitives. Keep the LLM integration logic. Replace the framework's queue and retry machinery with something you control.

This sounds like more work. It is less work in practice, because when something goes wrong at 2am you can actually find the problem.

The Germany constraint that turned out to be good engineering

The German government clients had a requirement I initially saw as a handicap: every decision made by the system had to be explainable to a civil servant who had no background in AI.

That requirement killed several architectural ideas that would have worked fine in a consumer product. It forced the system toward a design where every step is logged with its inputs and outputs, the LLM is one component in a documented process rather than a black box, and there is always a human review path for anything below a defined confidence threshold.

The resulting system is more verbose. It is also significantly easier to debug, trivial to audit, and more reliable under real load.

The explainability requirement is coming for regulated industries whether teams are ready for it or not. Financial services, healthcare, legal, government. Design for it now and you get better engineering as a side effect.

Where to start

If you are building an agent system and currently thinking mainly about prompts, here is a concrete sequence:

Define your task schema first. What does a valid output look like at each step? What validation rules apply? This determines your circuit-break conditions and your dead letter queue trigger.
Add idempotency before you add features. Assign a deterministic task ID to every step. Store results. Make it safe to retry.
Design your Saga before you wire up the steps. For every step that writes something, know what the compensation action is.
Set your blast radius policy. Which actions require human approval before execution? Write it down as code, not documentation.
Add the cost trace from day one. Not later. The cost dashboard is significantly harder to retrofit than to build initially.

The prompt engineering is the interesting part. The infrastructure is what makes it shippable.

If you are working on production AI systems and want to compare notes, I am at arif.sh/work.

Building Production LLM Pipelines in Go: Lessons From the Field

arif — Tue, 17 Feb 2026 23:56:33 +0000

I've spent the last two years integrating LLMs into production systems - a fintech platform processing millions of transactions and a document processing pipeline for government-scale workloads. Go is not the obvious choice for AI work. Python gets all the attention. But for production backends where reliability, cost, and latency actually matter, Go has been the better call every time.

Here's what I've learned.

Why Go for LLM pipelines?

The Python AI ecosystem is rich but chaotic. LangChain adds abstractions on top of abstractions. Dependency conflicts are constant. Async code that works in a notebook falls apart under real concurrency.

Go gives you:

Predictable memory usage under load
Goroutines for cheap, real concurrency (not async/await workarounds)
Compile-time errors that catch the stupid mistakes before they hit production
A single binary that deploys anywhere

The tradeoff is you write more code. There is no LangChain equivalent in Go. But in my experience, that is a feature. You understand exactly what your pipeline is doing.

The architecture that works

After iterating on several production systems, I keep coming back to this structure:

Request → Validator → ContextBuilder → LLMClient → ResponseParser → Cache → Response

Each stage is a function that takes input and returns output. No magic, no framework. Just functions.

type Pipeline struct {
    validator      Validator
    contextBuilder ContextBuilder
    llmClient      LLMClient
    parser         ResponseParser
    cache          Cache
}

func (p *Pipeline) Run(ctx context.Context, req Request) (Response, error) {
    if err := p.validator.Validate(req); err != nil {
        return Response{}, fmt.Errorf("validation: %w", err)
    }

    context, err := p.contextBuilder.Build(ctx, req)
    if err != nil {
        return Response{}, fmt.Errorf("context: %w", err)
    }

    cacheKey := p.cache.Key(req, context)
    if cached, ok := p.cache.Get(cacheKey); ok {
        return cached, nil
    }

    raw, err := p.llmClient.Complete(ctx, context)
    if err != nil {
        return Response{}, fmt.Errorf("llm: %w", err)
    }

    result, err := p.parser.Parse(raw)
    if err != nil {
        return Response{}, fmt.Errorf("parse: %w", err)
    }

    p.cache.Set(cacheKey, result)
    return result, nil
}

Simple. Each component is testable in isolation. Swap the LLM client without touching anything else.

The mistakes that cost real money

1. Not caching aggressively enough

LLM calls are expensive. In a document processing pipeline, the same context gets rebuilt and sent to the model repeatedly for similar inputs. We added a two-layer cache: in-memory for hot keys, Redis for warm keys. Cost dropped 40% overnight.

The key insight: you do not need semantic similarity matching for most caching. A hash of the normalized input is enough for 80% of cache hits.

func (c *PipelineCache) Key(req Request, ctx PipelineContext) string {
    h := sha256.New()
    h.Write([]byte(req.NormalizedInput()))
    h.Write([]byte(ctx.SystemPrompt))
    return hex.EncodeToString(h.Sum(nil))
}

2. Treating LLM errors like HTTP errors

LLMs fail in ways HTTP servers do not. You get:

Rate limit errors (back off and retry)
Malformed JSON responses (retry with a stricter prompt)
Responses that are technically valid but semantically wrong (detect and escalate)
Context window overflows (truncate and retry)

Each failure mode needs a different response. A generic retry loop handles none of them well.

func (c *LLMClient) Complete(ctx context.Context, req CompletionRequest) (string, error) {
    var lastErr error
    for attempt := 0; attempt < c.maxRetries; attempt++ {
        resp, err := c.call(ctx, req)
        if err == nil {
            return resp, nil
        }

        var rateLimitErr *RateLimitError
        var contextErr *ContextWindowError

        switch {
        case errors.As(err, &rateLimitErr):
            wait := time.Duration(rateLimitErr.RetryAfterSeconds) * time.Second
            select {
            case <-time.After(wait):
            case <-ctx.Done():
                return "", ctx.Err()
            }
        case errors.As(err, &contextErr):
            req = req.WithTruncatedContext(contextErr.MaxTokens)
        default:
            lastErr = err
            time.Sleep(backoff(attempt))
        }
    }
    return "", fmt.Errorf("max retries exceeded: %w", lastErr)
}

3. Building prompts with string concatenation

It starts fine. Then someone adds a special case. Then another. Then you have a 200-line function full of if/else building a string and nobody knows what the model is actually seeing.

Treat prompts like templates. Keep them in separate files. Version them.

type PromptTemplate struct {
    System string
    User   string
}

func (t *PromptTemplate) Render(vars map[string]any) (CompletionRequest, error) {
    systemTmpl, err := template.New("system").Parse(t.System)
    if err != nil {
        return CompletionRequest{}, err
    }
    // ... render both templates
}

When a prompt changes, it is a code change with a diff. Not a hidden string mutation buried in a function.

4. No observability until something breaks in production

Add structured logging from the start. Log the input hash, model, token counts, latency, and cache hit/miss on every call. When costs spike or latency degrades, you need this to debug.

c.logger.Info("llm_call",
    "input_hash", req.Hash(),
    "model", c.model,
    "prompt_tokens", usage.PromptTokens,
    "completion_tokens", usage.CompletionTokens,
    "latency_ms", latency.Milliseconds(),
    "cache_hit", false,
)

I use this to build a cost dashboard per feature, per user, per day. Without it you are flying blind.

RAG in Go: keep it boring

Retrieval-augmented generation does not need a vector database for most use cases. I have seen teams spin up Pinecone or Weaviate for a use case that would work fine with Postgres and pgvector.

Start with pgvector. It is boring. It works. You already have Postgres. The query looks like:

SELECT content, 1 - (embedding <=> $1) AS similarity
FROM documents
WHERE 1 - (embedding <=> $1) > 0.7
ORDER BY similarity DESC
LIMIT 5;

Migrate to a dedicated vector store when you have a real performance problem. Not before.

For generating embeddings in Go:

type EmbeddingClient struct {
    httpClient *http.Client
    apiKey     string
    model      string
}

func (e *EmbeddingClient) Embed(ctx context.Context, text string) ([]float32, error) {
    // standard HTTP call to your embedding endpoint
    // returns float32 slice ready for pgvector
}

No SDK needed. The API is a POST request. Write the 30 lines and move on.

Structured output without the drama

Getting JSON out of an LLM reliably is harder than it looks. Models hallucinate keys, nest objects incorrectly, or return valid JSON wrapped in markdown code blocks.

What works in production:

Use a model that supports native JSON mode or function calling
Define your schema explicitly in the prompt with an example
Validate with a strict parser, not json.Unmarshal directly
Retry on parse failure with a self-correction prompt: "Your previous response was not valid JSON. Here is the error: {error}. Please try again."

The self-correction step recovers 90% of parse failures without a human in the loop.

Cost math before you build

Before writing a line of code, do the math:

daily_requests * avg_input_tokens * input_price
+ daily_requests * avg_output_tokens * output_price
= daily_cost

Then ask: what is the cache hit rate I can realistically achieve? What is the monthly cost at 10x scale?

I have seen products that were profitable at 100 users become unprofitable at 1000 because nobody did this calculation. LLM costs scale linearly. Your pricing needs to account for it.

What I would do differently

If I were starting a new LLM-heavy Go service today:

Build the cache layer before the LLM integration, not after
Log token usage from day one, not when the bill arrives
Keep prompts in files, not code
Write an LLM interface with a mock implementation for tests
Do the cost math before committing to a feature

The patterns that work in Go for LLM pipelines are not exotic. They are the same patterns that work for any external API integration: encapsulate, cache, observe, handle errors explicitly. The LLM just has a few extra failure modes.

If you are building something that needs a production AI backend and want to talk through the architecture, I am available for contracts. arif.sh/work

The Real Deal Behind EngageCraft.io’s Creation

arif — Wed, 28 Feb 2024 09:03:00 +0000

Hello World! I’m here to pull back the curtain on how EngageCraft.io went from a sketch on a napkin to a live SaaS platform in the span of a single weekend. No fluff, just the straight-up journey of building my first indie project.

The Spark and the Sprint

The idea? Simple yet ambitious: create a tool where users feed in their product URL, and out pops ready-to-use social media content for platforms like Twitter and Reddit. The catch? I wanted to see it live by Monday morning. Why wait, right?

Choosing Laravel: No Regrets

Laravel was my weapon of choice for this sprint. I needed something that would let me move fast without breaking things (too much). Laravel Forge and Breeze weren’t just tools; they were my lifelines to a smooth deployment and an MVP that didn’t look like it was held together by duct tape. I chose Laravel because it promised—and delivered—a way to develop quickly and deploy smoothly, a non-negotiable for my crazy deadline.

Marketing: The Unexpected Journey

With EngageCraft.io built, I stared down my next challenge: marketing. I was a developer stepping into a marketer’s shoes, and it felt like learning to walk all over again. But here’s the thing—I was building EngageCraft.io for people like me, and who better to sell it to them than one of their own? So, I dove headfirst into marketing with the same gusto I applied to coding.

The Takeaway

Building EngageCraft.io on a weekend was a wild ride. There were moments of doubt, sure, but also moments of pure adrenaline. It was more than just coding; it was about bringing a vision to life, learning new skills on the fly, and pushing beyond what I thought was possible.

Looking Forward

This isn’t just the end of a weekend project; it’s the beginning of something bigger. EngageCraft.io is live, but it’s also evolving. I’m on a mission to refine it, scale it, and see it become the go-to tool for social media content generation.

So, what’s the moral of the story? Don’t wait. If you’ve got an idea, chase it. Build it. Launch it. The world needs more creators who are willing to leap, especially over a weekend.

Embarking on 2024: Dreams, Challenges, and Adventures

arif — Sun, 31 Dec 2023 23:34:09 +0000

Happy New Year, everyone! As we step into 2024, I am filled with a mix of excitement and anticipation. This year, I am determined to create new habits and reinvent myself. There is something about the start of a new year that gives me hope and ignites the urge to chase my dreams.

I am incredibly grateful to have my wife by my side on this journey. Her constant encouragement and unwavering belief in the power of pursuing our dreams have been contagious. She reminded me that it is never too late to set new goals and turn our aspirations into reality.

Here is my ambitious game plan for 2024:

European Escapades : This year, I have promised myself to finally explore the vibrant streets of Europe. From the historical alleys of Rome to the scenic beauty of the Swiss Alps, I am eagerly looking forward to immersing myself in the diverse cultures and landscapes.
A Year of Books : My bookshelf is overflowing with titles that have been waiting to be read. This year, I am committed to finally giving those pages the attention they deserve. I aim to delve into a wide range of genres, from classic literature to modern tech manuals.
Fitness Journey : It is time for me to prioritize my health. I will start with a few mornings of exercise each week. I understand that this won't be an easy journey, but I am fully committed to making it a part of my lifestyle.
Coding Mastery : As a programmer, I recognize the importance of continuously improving my skills. To achieve this, I am setting a goal to tackle three LeetCode problems daily. This will undoubtedly be challenging, but I am ready to embrace the task and grow as a developer.

To make all of this happen, I will focus on taking small, consistent steps. I understand the value of building habits gradually and not overwhelming myself with too much at once.

Additionally, reflection will be a vital part of my journey. I will use this blog to not only share my triumphs but also to document the inevitable setbacks that come along the way. After all, personal growth is not a linear path.

I am incredibly excited to share these experiences with all of you. I hope that my journey will inspire you to chase your own dreams, no matter their size. I am also eager to hear about your plans and aspirations – let's support and motivate each other!

So, here's to an unforgettable year in 2024! We have dreams to chase, habits to form, and adventures to embark on. This year will be a time of discovery, learning, and undoubtedly, a lot of fun. Let us make the most of it together!

Cheers to a year of turning dreams into reality. I cannot wait to embark on this adventure with all of you. Let the journey begin!

Creating Real-Time WebSockets with Go and WebAssembly

arif — Thu, 09 Nov 2023 12:21:07 +0000

Writing WebAssembly with Go was always something I wanted to do, but finding examples or tutorials was hard, so I decided to shuffle through pages.

In this post, I'll share with you my learnings and some code examples.

This project is an exciting exploration into the world of real-time web communication. We're delving into connecting a WebSocket directly from WebAssembly (Wasm) code, enabling dynamic interactions within the web page. By leveraging WebSocket events, we can manipulate the Document Object Model (DOM) in response to live data, creating an interactive and responsive user experience.

Project Outline

In this project, we will create a WebSocket service (using Gorilla WebSocket), a WASM service, and an index.html (this is solely to see what we've done).

Folder Structure

In this basic project, we're keeping the folder structure straightforward to avoid complexity. All that's required is a wasm/ directory for the WebAssembly files, a pkg/socket/ directory for the socket service logic, and a public/ directory to house our frontend assets.

So project will look like this.

main.go
wasm/
pkg/socket/
public/
Makefile

Let's Code!

public/index.html

<!DOCTYPE html>
<html lang="en">

<head>
  <meta charset="UTF-8">
  <title>Go WASM Button Example</title>
  <script src="wasm_exec.js"></script>
  <style>
    #btn {
      position: absolute;
      top: 50%;
      left: 50%;
      transform: translate(-50%, -50%);
    }
  </style>
</head>

<body>
  <p id="text"></p>
  <button id="btn">Click me</button>
  <script>
    const go = new Go();

    WebAssembly.instantiateStreaming(fetch("main.wasm"), go.importObject).then((result) => {
      go.run(result.instance);
    });

    document.getElementById("btn").addEventListener("click", () => {
      onButtonClick();
    });
  </script>
</body>

</html>

Our HTML layout is intentionally minimalistic, featuring a single button centrally placed on the screen. When clicked, this button activates a 'click' event, which in turn calls the buttonClick() function.

Additionally, positioned at the top of the screen is a <p> tag with id text, which we will dynamically manipulate from our WebAssembly code.

pkg/socket/manage.go

package socket

import (
    "log"
    "math/rand"
    "net/http"
    "sync"

    "github.com/gorilla/websocket"
)

// Pre-configure the upgrader, which is responsible for upgrading
// an HTTP connection to a WebSocket connection.
var (
    websocketUpgrader = websocket.Upgrader{
        ReadBufferSize:  1024,
        WriteBufferSize: 1024,
    }
)

// NotifyEvent represents an event that contains a reference
// to the client who initiated the event and the message to be notified.
type NotifyEvent struct {
    client  *Client
    message string
}

// Client represents a single WebSocket connection.
// It holds the client's ID, the WebSocket connection itself, and
// the manager that controls all clients.
type Client struct {
    id         uint32
    connection *websocket.Conn
    manager    *Manager

    writeChan chan string
}

// Manager keeps track of all active clients and broadcasts messages.
type Manager struct {
    clients ClientList

    sync.RWMutex

    notifyChan chan NotifyEvent
}

// ClientList is a map of clients to keep track of their presence.
type ClientList map[*Client]bool

// NewClient creates a new Client instance with a unique ID, its connection,
// and a reference to the Manager.
func NewClient(conn *websocket.Conn, manager *Manager) *Client {
    return &Client{
        id:         rand.Uint32(),
        connection: conn,
        manager:    manager,
        writeChan:  make(chan string),
    }
}

// readMessages continuously reads messages from the WebSocket connection.
// It will send any received messages to the manager's notification channel.
func (c *Client) readMessages() {
    defer func() {
        c.manager.removeClient(c)
    }()

    for {
        messageType, payload, err := c.connection.ReadMessage()

        c.manager.notifyChan <- NotifyEvent{client: c, message: string(payload)}

        if err != nil {
            if websocket.IsUnexpectedCloseError(err, websocket.CloseGoingAway, websocket.CloseAbnormalClosure) {
                log.Printf("error reading message: %v", err)
            }
            break
        }
        log.Println("MessageType: ", messageType)
        log.Println("Payload: ", string(payload))
    }
}

// writeMessages listens on the client's write channel for messages
// and writes any received messages to the WebSocket connection.
func (c *Client) writeMessages() {
    defer func() {
        c.manager.removeClient(c)
    }()

    for {
        select {
        case data := <-c.writeChan:
            c.connection.WriteMessage(websocket.TextMessage, []byte(data))
        }
    }
}

// NewManager creates a new Manager instance, initializes the client list,
// and starts the goroutine responsible for notifying other clients.
func NewManager() *Manager {
    m := &Manager{
        clients:    make(ClientList),
        notifyChan: make(chan NotifyEvent),
    }

    go m.notifyOtherClients()

    return m
}

// otherClients returns a slice of clients excluding the provided client.
func (m *Manager) otherClients(client *Client) []*Client {
    clientList := make([]*Client, 0)

    for c := range m.clients {
        if c.id != client.id {
            clientList = append(clientList, c)
        }
    }

    return clientList
}

// notifyOtherClients waits for notify events and broadcasts the message
// to all clients except the one who sent the message.
func (m *Manager) notifyOtherClients() {
    for {
        select {
        case e := <-m.notifyChan:
            otherClients := m.otherClients(e.client)

            for _, c := range otherClients {
                c.writeChan <- e.message
            }
        }
    }
}

// addClient adds a new client to the manager's client list.
func (m *Manager) addClient(client *Client) {
    m.Lock()
    defer m.Unlock()

    m.clients[client] = true
}

// removeClient removes a client from the manager's client list and
// closes the WebSocket connection.
func (m *Manager) removeClient(client *Client) {
    m.Lock()
    defer m.Unlock()

    if _, ok := m.clients[client]; ok {
        client.connection.Close()
        delete(m.clients, client)
    }
}

// ServeWS is an HTTP handler that upgrades the HTTP connection to a
// WebSocket connection and registers the new client with the manager.
func (m *Manager) ServeWS(w http.ResponseWriter, r *http.Request) {
    log.Println("New Connection")

    conn, err := websocketUpgrader.Upgrade(w, r, nil)
    if err != nil {
        log.Println(err)
        return
    }

    client := NewClient(conn, m)
    m.addClient(client)

    go client.readMessages()
    go client.writeMessages()
}

Our WebSocket service is powered by a Go file that relies on the gorilla/websocket library. It begins with setting up an Upgrader from the library, which transitions an HTTP connection to the WebSocket protocol, allowing for real-time communication.

The code defines a NotifyEvent type that encapsulates messages from a client (an active WebSocket connection) and a Client type that keeps track of individual connections and their communication with the server through channels.

The Manager type oversees all Client instances, managing incoming and outgoing messages with the help of ClientList, a map that keeps track of all active connections.

Clients are created with a unique ID and are tied to their WebSocket connections and the managing system. They have two main routines: readMessages listens for incoming messages and forwards them to the manager, and writeMessages sends outgoing messages from the server to the client's WebSocket.

The Manager is also responsible for broadcasting messages to all clients except the sender, ensuring that messages are propagated in real-time to all connected users.

Lastly, ServeWS is a function that handles new WebSocket connections by upgrading HTTP requests to WebSocket and initiating the message reading and writing routines.

This structure allows us to manage multiple clients and their interactions efficiently, forming the backbone of our WebSocket-based real-time communication service.

main.go

main.go, in this file we need to serve our index.html and some websocket stuff. ill use net/http package for this.

package main

import (
    "log"
    "net/http"

    "github.com/doganarif/wasm/2/pkg/socket"
)

func main() {
    setupAPI()

    // Serve on port :8080, fudge yeah hardcoded port
    log.Fatal(http.ListenAndServe(":8080", nil))
}

// setupAPI will start all Routes and their Handlers
func setupAPI() {
    manager := socket.NewManager()

    // Serve the ./public directory at Route /
    http.Handle("/", http.FileServer(http.Dir("./public")))
    http.Handle("/ws", http.HandlerFunc(manager.ServeWS))
}

Our web service has a simple starting point, a file where everything kicks off. It sets up our web server and tells it to listen for visitors on port 8080 (the classic spot for web development testing).

We call a setupAPI() function that does a bit of behind-the-scenes work. It taps into our socket package to create a Manager, which is like the director of a play, coordinating all our WebSocket connections.

We also set up a special /ws route for WebSocket connections, where our Manager starts handling the real-time chat.

wasm/wasm.go

package main

import (
    "context"
    "fmt"
    "log"
    "syscall/js"

    "nhooyr.io/websocket"
)

// Conn wraps a WebSocket connection.
type Conn struct {
    wsConn *websocket.Conn
}

// NewConn establishes a new WebSocket connection to a specified URL.
func NewConn() *Conn {
    c, _, err := websocket.Dial(context.Background(), "ws://localhost:8080/ws", nil)
    if err != nil {
        fmt.Println(err, "ERROR")
    }

    return &Conn{
        wsConn: c,
    }
}

func main() {
    // Channel to keep the main function running until it's closed.
    c := make(chan struct{}, 0)

    println("WASM Go Initialized")
    // Establish a new WebSocket connection.
    conn := NewConn()

    // Register the onButtonClick function in the global JavaScript context.
    js.Global().Set("onButtonClick", onButtonClickFunc(conn))

    // Start reading messages in a new goroutine.
    go conn.readMessage()

    // Wait indefinitely.
    <-c
}

// onButtonClickFunc returns a js.Func that sends a "HELLO" message over WebSocket when invoked.
func onButtonClickFunc(conn *Conn) js.Func {
    return js.FuncOf(func(this js.Value, args []js.Value) interface{} {
        println("Button Clicked!")
        // Send a message through the WebSocket connection.
        err := conn.wsConn.Write(context.Background(), websocket.MessageText, []byte("HELLO"))
        if err != nil {
            log.Println("Error writing to WebSocket:", err)
        }
        return nil
    })
}

// readMessage handles incoming WebSocket messages and updates the DOM accordingly.
func (c *Conn) readMessage() {
    defer func() {
        // Close the WebSocket connection when the function returns.
        c.wsConn.Close(websocket.StatusGoingAway, "BYE")
    }()

    for {
        // Read a message from the WebSocket connection.
        messageType, payload, err := c.wsConn.Read(context.Background())

        if err != nil {
            // Log and panic if there is an error reading the message.
            log.Panicf(err.Error())
        }

        // Update the DOM with the received message.
        updateDOMContent(string(payload))

        // Log the message type and payload for debugging.
        log.Println("MessageType: ", messageType)
        log.Println("Payload: ", string(payload))
    }
}

// updateDOMContent updates the text content of the DOM element with the given text.
func updateDOMContent(text string) {
    // Get the document object from the global JavaScript context.
    document := js.Global().Get("document")
    // Get the DOM element by its ID.
    element := document.Call("getElementById", "text")
    // Set the innerText of the element to the provided text.
    element.Set("innerText", text)
}

Our wasm.go file, once compiled into a .wasm executable, is the client-side counterpart to the WebSocket service we discussed earlier. It's set to run within the web browser, interfacing directly with the DOM.

The file’s responsibility starts with establishing a WebSocket connection to the server endpoint we've set up—ws://localhost:8080/ws. This is the communication channel our web service listens to, we already detailed.

In parallel, we're attentive to the server’s response. Incoming messages from our WebSocket server are caught and used to update the webpage dynamically.
This is where the full circle of client-server interaction completes—our Go code receives server messages, then reflects changes directly in the user's view (<p id="text").

Makefile

.PHONY: build buildwasm run clean serve

# Variables
WASM_DIR := ./wasm
PUBLIC_DIR := ./public
SERVER_FILE := main.go
WASM_SOURCE := $(WASM_DIR)/wasm.go
WASM_TARGET := $(PUBLIC_DIR)/main.wasm
GOOS := js
GOARCH := wasm

# Default rule
all: run

# Run the server
run: serve

# Serve the project
serve: buildwasm
    go run $(SERVER_FILE)

# Build the WebAssembly module
buildwasm:
    GOOS=$(GOOS) GOARCH=$(GOARCH) go build -o $(WASM_TARGET) $(WASM_SOURCE)

# Clean the built artifacts
clean:
    rm -f $(WASM_TARGET)

Conclusion

We've gone through the ropes of creating a real-time application using Go and WebAssembly. The project's setup is simple: a WebSocket server in Go, a basic HTML page, and some Go code compiled to WebAssembly that runs in the browser. We've covered how to send and receive messages through WebSockets and how to reflect those messages in the web page by updating the DOM.

From setting up the project structure to handling real-time communication between the client and server, the provided code examples aim to give you a clear starting point for building your own applications with Go and WebAssembly. It's a straightforward setup, but it lays the foundation for more complex projects.

Hope this helps you kickstart your own adventures in WebAssembly with Go. Keep building, keep learning, and most importantly, keep it simple.

Github Repo : https://github.com/doganarif/go-wasm-socket

Dockerizing Django with Numpy and Gunicorn

arif — Thu, 14 Jan 2021 10:19:47 +0000

Hi all.

Actually theres lot of tutorial about dockerizing django on web. But at the project i working on we using libraries like Numpy and Pillow and alpine linux cannot compile this because missing dependencies.

Our Stack;

Django
Numpy
Pillow
Postgresql

First of all let create basic Dockerfile

FROM python:3.6.4-alpine

ADD . /app

WORKDIR /app

RUN pip install -r requirements.txt

EXPOSE 8000

CMD ["manage.py", "runserver"]

Its basically works if you not using c compiled libraries like Numpy.

The thing is we need to add compilers to alpine i have big layer on there because running this proccesses on one layer is better i think.

Full Dependency install Layer :

RUN apk add --no-cache jpeg tiff-dev openjpeg-dev postgresql-dev && \
    apk add --no-cache \
    --virtual=.build-deps \
    g++ zlib-dev freetype-dev lcms2-dev \
    gcc libc-dev linux-headers tk-dev tcl-dev harfbuzz-dev \
    fribidi-dev build-base py-pip rsyslog cython file binutils \
    musl-dev python3-dev cython && \
    ln -s locale.h /usr/include/xlocale.h && \
    pip install --upgrade pip && \
    pip install -r requirements.txt --no-cache-dir && \
    rm -r /root/.cache && \
    find /usr/lib/python3.*/ -name 'tests' -exec rm -r '{}' + && \
    find /usr/lib/python3.*/site-packages/ -name '*.so' -print -exec sh -c 'file "{}" | grep -q "not stripped" && strip -s "{}"' \; && \
    rm /usr/include/xlocale.h && \
    apk --purge del .build-deps

Compilers for Alpine

I installed some packages like g++ and linux-headers virtual and cleaned them after pip install because i just need this packages for post scripts of some dependencies and size is matter on container.

  -t, --virtual NAME    Instead of adding all the packages to 'world', create a new 
                        virtual package with the listed dependencies and add that 
                        to 'world'; the actions of the command are easily reverted 
                        by deleting the virtual package

What that means is when you install packages, those packages are not added to global packages. And this change can be easily reverted. So if I need gcc to compile a program, but once the program is compiled I no more need gcc.

I can install gcc, and other required packages in a virtual package and all of its dependencies and everything can be removed this virtual package name. Below is an example usage

apk add --virtual mypacks gcc vim
apk del mypacks

The next command will delete all 18 packages installed with the first command.

Postgres

Im running Postgresql on my local machine not docker. And i need to connect it. You can use host.docker.internal this mapping is directly routes your machines localhost

So i added this env variable

ENV DB_HOST=host.docker.internal

and used this on `settings.py`

Gunicorn

I decided to use Gunicorn for running processed with n workers. So i added gunicorn to my requirements.txt and added this two lines for run processes

ENV WORKER_COUNT=2

CMD ["sh", "-c", "gunicorn --bind :8000 --workers ${WORKER_COUNT} project.wsgi:application"]

Using WORKER_COUNT env variable for default value. Im overriding this value everytime i need.

Running

I decided to create makefile for run this container with n workers.
So i created Makefile with this command.


docker-run:
    docker run -e worker_count=${worker} -p 127.0.0.1:8000:8000 project${version}

Full Dockerfile

FROM python:3.6.4-alpine

ADD . /app
WORKDIR /app

# Need to Update at Settings.py
ENV DB_HOST=host.docker.internal

# Default Worker Count is 2
ENV WORKER_COUNT=2

# Install dependencies layer.
RUN apk add --no-cache postgresql \
                        jpeg tiff-dev openjpeg-dev \
                        libpq postgresql-libs postgresql-dev && \
    apk add --no-cache \
    --virtual=.build-deps \
    g++ jpeg-dev zlib-dev freetype-dev lcms2-dev \
    gcc libc-dev linux-headers tk-dev tcl-dev harfbuzz-dev \
    fribidi-dev zlib-dev build-base py-pip rsyslog cython file binutils \
    musl-dev python3-dev cython && \
    ln -s locale.h /usr/include/xlocale.h && \
    pip install --upgrade pip && \
    pip install -r requirements.txt --no-cache-dir && \ 
    rm -r /root/.cache && \
    find /usr/lib/python3.*/ -name 'tests' -exec rm -r '{}' + && \
    find /usr/lib/python3.*/site-packages/ -name '*.so' -print -exec sh -c 'file "{}" | grep -q "not stripped" && strip -s "{}"' \; && \
    rm /usr/include/xlocale.h && \
    apk --purge del .build-deps  


EXPOSE 8000

CMD ["sh", "-c", "gunicorn --bind :8000 --workers ${WORKER_COUNT} project.wsgi:application"]

I hope you found this article useful.

Thanks.

Im Looking For Product Manager For My Own Project

arif — Wed, 30 Dec 2020 22:07:38 +0000

Project Description

Basically Take-Away eCommerce Mobile Application and BackEnd Services.

Status

I only developed %50 of Back End Project with Django. For example developed pricing, basically customer logic etc.

Stack

Im developing Back-End Web Services with Django. Also for mobile apps im thinking develop with flutter or react-native.

Requests

I need one product manager for develop product and business. For now im funding this project myself. So i can share stock. If you can accept please message me :)