DEV Community

Umar Bilal
Umar Bilal

Posted on Originally published at codeatoms.ai

System design for physical AI: a canal ledger that sees the short tail

By Muhammad Umar. This is the engineering summary of an open reference architecture from CodeNinja Atoms for a self-improving AI system that allocates canal water, one of a series of self-improving systems that get better from what named people confirm, on hardware the operator owns, without training a model on the fly. The paper, its object model as JSON and the model register are free to reuse under CC BY 4.0: https://codeatoms.ai/canal-irrigation-pakistan/. DOI 10.5281/zenodo.24048587. The operator is described by class, never by name.

A provincial irrigation department in Pakistan takes its share of river water at barrages and head regulators and moves it through a main canal, distributaries and minors to outlets that farmers share by rotation. Every rotation it needs one question answered for every distributary on the main: did it receive its entitlement share, and if not, where, since when and why. The answer lives in pieces no system joins. Telemetry measures flow depth and velocity at the main heads every few minutes, the indent and release ledgers state what was asked for and released, the gauge reader diaries record what the field saw, and the complaint register is the first signal that a tail has gone short, usually after the crop is lost.

1. Three layers, and the records stay put

The design is one pattern in a fixed order: systems of record below, read in place and never replaced, modified or loaded; one governed object model in the middle as a projection over them, never a copy; applications and agents above, with the surfaces where engineers confirm or reject each draft. The telemetry system and the paper ledgers remain the legal record, so the platform never operates a gate and never writes into a system of record. The single write path into the model is a named person's confirmation at a surface.

Telemetry and ledger data never leave the operator's network, and no third party model service is called. No agent operates a gate, which stays a hand operation on telephone instructions. The design chose against cameras (the canals are watched through flow instruments and people), against edge serving at canal heads, and against labelling and retraining; gate actuation, new field telemetry, satellite imagery analytics and historic data migration are out of scope.

2. Two adapter families, eight sources

Eight read only sources feed the design, split into two paths. The live path is the telemetry system. The paper path holds the rest, among them the barrage and head regulator gauge records that baseline the telemetry, the indent and release ledgers that fix entitlement, the rotation schedules per distributary that define each outlet's share, the gauge reader diaries that carry field evidence between telemetry points, the complaint register, and the revenue crop records that give indents their demand basis.

Two adapter families are the sole path in. The telemetry adapter performs a direct historian read, polling the telemetry store without ever writing to it. The integration adapter runs an OCR pipeline over the paper sources, and every batch passes a human verification stage with a 2 percent failure gate before anything enters the object model, because a wrong entitlement baseline would corrupt every downstream comparison. Every item enters stamped with source system, adapter, timestamp and verification state. Reads are idempotent, so a replayed batch cannot double count a reading or a ledger entry, and one clock reconciles telemetry timestamps, diary dates and ledger entries before any entitlement versus delivery join is attempted.

Apache Kafka 4.3.x in KRaft mode is the backbone, ordered per partition keyed on station identity so readings from one station arrive in order even when stations interleave. Delivery is at least once with idempotent keys, and broker retention buffers a dropped link. TimescaleDB holds the telemetry as time series, Harbor holds artifacts and images as an air gapped registry, Keycloak 26 issues identity on site, and Prometheus 3.13 LTS, Grafana 13.3 and Loki observe the tier.

3. Sixteen objects, one walk

The object model holds sixteen objects in five kinds with thirteen typed links on Hyper Ontology v0.9. Five assets carry the hydraulic structure: barrage, head regulator, reach, gate and outlet. Two sites locate the network: main canal and distributary. Three events unfold in time: rotation, shortfall and season. Four records are the paper trail: indent, release, complaint and crop area. Two measures are readings: telemetry reading and gauge reading. The shortfall is the pivot, with a status vocabulary that runs open, cause drafted, cause confirmed, corrected.

A barrage commands head regulators; a head regulator reports telemetry readings and sets gates; a rotation entitles outlets; a release instructs a gate and is measured at a head regulator; a shortfall is judged against an indent; a complaint names a distributary; an indent draws on crop area. So the question of which outlets were short of share for two consecutive rotations while the main's head met its indent is one traversal, complaint to distributary to rotation to outlet, then shortfall to indent and release to telemetry reading, every step carrying its provenance. A document store can hold the same fields, but it cannot walk the edges.

4. Two servers in one room

All inference runs in one place, the department's circle server room, on hardware it owns inside Pakistan. There is no edge tier and nothing is served at a canal head. A 48 GB PCIe class site server carries the small model tier: embedding models for retrieval over ledger text and telemetry annotations, OCR models for the handwritten ledgers, and forecasting models for shortfall detection. Beside it sits the frontier class node for the three drafting agents and the command center, one node of eight 141 GB HBM cards, about 1,128 GB of aggregate memory.

That node is held arithmetic, not a filed footprint. No open weight model was verified live on this run, so the register holds classes and no weights are committed until it verifies one. The operator names GLM 5.3, and the plan holds its FP8 arithmetic: published FP8 weights of about 756 GB fit one node of eight 141 GB cards, chosen against GLM 5.3 at BF16 on two nodes or a mid size model. The 141 GB class sits above the US advanced computing export threshold for a D:2, D:3 and D:4 destination, so the export licence determination runs first in phase one, with an in country procurement route or colocation fallback named before any order.

The latency budget follows human cadences. Shortfall detection must land inside one telemetry cadence, so a developing head shortfall is visible on the first of the three days it takes to breach. Indent drafts must be ready ahead of the ten day submission window. OCR batches must clear verification before the rotation they record closes. When a telemetry feed drops, the system marks the gap and holds the last good state for that reach rather than extrapolating through it. When power fails, the live loop pauses and canal operation continues by hand, because the ontology is only a projection.

5. Agents draft, engineers confirm

Six services (network ontology, ledger digitization, agents and memory, command center, sovereign hosting and the season pilot) carry the work onto three surfaces. Three agents draft: shortfall, indent and dispute. The controlling engineer confirms or rejects the drafted cause and correction before any gate instruction is drafted, the indent draft is approved before it goes to the authority that sets the provincial share, and the dispute account is drafted from cited records for the officer to send, presenting entitlement versus delivery with sources, never blame. Every confirmed cause and accepted adjustment is kept as a reason coded record the next season reads, captured at the decision rather than reconstructed afterwards, so the memory stays with the department when an engineer moves post. Nothing is trained into weights.

Appendix A prices the conditional shape at about 636,000 US dollars to own over three years (606,000 to 662,000): about 408,000 for the eight card node, about 26,000 for the site server with its 48 GB card, support at 8 to 12 percent of hardware value a year, and power taken as a ceiling at the uniform tariff for general services. Renting the same eight 141 GB cards and one 48 GB card on the deepest three year commitment in the nearest AWS region that offers them is about 773,000, so ownership costs about four fifths of it; on demand it is about 2.05 million, against which ownership costs a little under a third. Renting is a price reference only: AWS lists no region in Pakistan, and every rented option would carry telemetry and ledgers across the border.

The rollout is two phases carrying 17 scope items, 16 and 1, each closed by a gate, never a date. Phase one runs six workstreams on one main canal and its distributaries for one cropping season, and its gate is the ontology itself: barrage, main, distributary, reach, gate, outlet, rotation, indent, release and shortfall must reconcile one real week of telemetry against the ledgers before any agent is allowed to draft. Phase two, the extension to the other canals, opens only through that gate. Manual and telemetry baselines are reconciled in the open in the first month, before any improvement is claimed. The headline measure is the percent of tail outlets receiving their rotation share, nominal 85 percent, with days from shortfall onset to correction and complaints per distributary beside it.

Made with CodeNinja Praxis, the CodeNinja Atoms platform for designing physical AI systems. The object model imports into Hyper Ontology. Paper, ontology and register: https://codeatoms.ai/canal-irrigation-pakistan/

One of the CodeNinja Atoms worked examples of physical AI system design: systems of record read and never written, a governed ontology joined by typed links, open weights on hardware the operator owns in its own country, and agents that draft while named people approve, the shape every physical AI design in the series is published in, with its object model as hyper-ontology/1 JSON under CC BY 4.0.

Top comments (0)