The field app does not ask a public chatbot whether the job is done. It runs detectors on our workers, then a self-hosted LLM gateway on our GPU cluster, because the photos are customer sites. The agent sees, detects, reads the ticket plus the crew’s notes, then closes or sends them back. Policy still decides. A human does not sit on closeout.
Pass → the agent closes the job. Reject → recapture, with one line on what to fix. The crew never sees Nano arguing with Qwen. They see a closed ticket or a camera again.
Why the models stay inside
Field cabinets, GPS pins, and work notes do not go to a vendor chat API. Vision and reasoning call an internal OpenAI-compatible gateway. Weights for PPE, fiber, and GPON live on the worker. CLIP embeddings live next to Postgres. Data protection here is a network boundary, not a privacy policy PDF.
| Model | Where | Job |
|---|---|---|
| YOLOv8-seg PPE | worker | boots, gloves, goggles, helmet, person, vest |
| YOLO fiber subject | worker | is this fiber work? on the pass gate |
| YOLO quality classifier | worker |
not_fiber → excellent bands; not the pass gate |
| YOLO GPON / V-SOL | worker | find the ONT, read PWR / PON / LOS |
CLIP vit-base-patch32
|
worker | similar-install retrieval into context |
| Nemotron-3-Nano | self-hosted gateway | stage 1: ticket + notes context, no image |
| Qwen (vision) | self-hosted gateway | stage 2: blind rubric on the photo; thinking disabled |
| Nemotron-3-Super | self-hosted gateway | stage 3: pass or recapture, agreement vs YOLO |
Qwen is the eyes. Nano is the briefing. Super is the referee’s clerk. Python is the referee.
Three UML views. Component for the tape. Component for the models. State for pass-close vs recapture.
Overview
The new app is a field web client, an API, a worker, and Postgres. Source MySQL is read-only. The GPU gateway is the only place pixels become language. The worker never writes the source database.
API hydrates tickets. Worker owns models. Gateway never sees a write to MySQL.
The ML stack
PPE default: two distinct required classes. Person is informational. Fiber is two models: subject detection is on the pass gate; the quality band is telemetry. GPON is a side path: find the ONT, ignore map-watermark lamps, then Qwen names PWR / PON / LOS.
Capture metadata: new, and near the site
The agent does not trust a JPEG from last week’s gallery as “I am on this ticket.”
Work cannot start without a current GPS check-in. Every PPE and inspection save carries live latitude, longitude, accuracy, and captured_at. The ticket already has site coordinates from the source request. That pair is the near-site check: the stamp on the capture versus the pin on the job. A photo with no fresh on-site stamp is not evidence; it is a file.
“New” is the capture timestamp, not EXIF we hope WhatsApp left intact. EXIF is used to orient the frame. Location of record is the browser geolocation taken at save time. Camera vs gallery is tagged on the upload. Blurry, dark, or tiny frames die at the gate before Nano or Qwen ever run.
The agent
Detect. Read context. See. Pass and close, or reject and recapture. No human on the close button.
The crew writes work notes during the session and inspection notes on the capture (required, 4000 characters). Nano’s only job is text: expected work, risks, the diary of what they already did (“replaced the drop, LOS was red, waiting for light”). It does not look at pixels. CLIP neighbors above 0.7 are prior examples. The prompt says do not copy their outcome.
Qwen then scores seven rubric ids with the photo only: routing, bend/strain, connectors, labels, slack, cleanliness, maintainability. Status is pass, advisory, fail, or not_visible. not_visible is not fail.
Super proposes pass or recapture and whether it agrees with YOLO. It cannot escalate to a supervisor to finish the job. Policy either closes or asks for a recapture. Disagreement and uncertain are recaptures, not a human closeout queue. Admins may still audit after the fact. They do not stamp the ticket shut.
The worker claims the durable job, runs the three stages, writes the receipt, and marks the job complete. Stale jobs recover after 30 minutes. On policy pass, the ticket closes. On recapture, the crew gets the one-line fix and the camera. Nobody fills a closeout form to bless a pass.
How a recommendation is born
Policy overrules Super. Any unavailable stage → recapture. Detector not fiber → recapture. Visual confidence under 0.50 → recapture. Required criterion below 0.50, fail, or not visible → recapture. Labels may be advisory. No dBm, no OTDR, no certification sentence.
On pass, the rec is optional polish from an advisory criterion, and the ticket is already closing. On recapture, the rec is the blocking fix or a framing retake — that is the only thing the crew must do. We keep one. User APIs do not dump the model argument; they dump a closed job or a recapture.
Thursday, the wide tray
A crew shot the whole cabinet at 14:10. Detector said fiber. Classifier said good. Qwen marked connectors not_visible. Super wanted to pass. Policy recaptured. The rec was: closer on the patch panel.
Dispatch asked why a “good” photo failed. Because good is not on the gate. The missing close-up is. GPS was on site. The photo was new. The tray was too wide. That is the scar we kept.
How to improve accuracy
Measure admin overturns, not Super’s self-confidence. Gold set = the disagreement queue.
Retrain the fiber detector if subject is wrong. Leave the quality classifier as a dashboard.
Calibrate 0.50 against reviewers, especially bend_strain and connectors. Labels are already non-blocking.
Write notes the agent can use. “Done” is noise. “Replaced drop, LOS flashed red, coiled slack on the left” is context Nano can actually brief. Put the session work-note diary in that briefing every time, not only the inspection textarea.
Keep the GPS/timestamp contract. A model cannot fix a photo taken in the yard. Reject stale captured_at and pins that are not the ticket site before Qwen spends GPU on the wrong cabinet.
Fix capture SOP: overview plus close-up of the fault. Tune blur/luma from rejected field files.
Stop CLIP from copying old passes. Filter neighbors by work type; drop ones without a human verdict.
Map the blocking fail to the same action string we use on advisory pass. Prompt version is ticket-vision-decision-v5. Diff it on the gold set.
Do not send these frames to a public model to “get a better LLM.” The accuracy loss you cannot see is the data leaving the site.
PPE: raise min detected items only after glove/goggle recall in real cabinets.
Self-hosted eyes. Named models. One referee. One rec. No certification from a JPEG.
Top comments (0)