A draft surveyor weighs a bulk cargo by reading the painted draft marks on a hull, and the sea never holds still across them. I built DraftMark for the Meta AI Developer Hackathon to do two things: time-average that moving waterline from a short video, and let a model write the per-ship survey program. This post is about the second part, because the interesting problem is not "can a model write the program" but "why would you believe it".
Live: https://draftmark.edycu.dev · Repo: https://github.com/edycutjong/draftmark
The problem with a per-ship program
After the draft is read, a survey converts it to displacement through that ship's hydrostatic booklet: interpolate a table, apply trim corrections, correct for water density, subtract ballast and fuel. Every booklet states its own conventions. The usual answer is a spreadsheet per vessel, built by hand.
I gave Muse Wonder 1.0 a public trim and stability booklet (a Ro-Ro/passenger ship, not a bulk carrier) and asked for the survey program. A program written by a model is a claim. I wanted an exam it could not talk its way through.
The exam is the booklet's own printed examples
The booklet prints 22 worked loading conditions with the displacement it expects. So the acceptance rule is: a program passes a row if it lands within 0.1 % of the printed number, and it is accepted only if at least 20 of 22 rows pass, both out-of-range probes raise, and 200 randomized cases agree with a reference program within 0.05 %.
verdict = "PASS" if (n >= 20 and pc >= math.ceil(0.9 * n) and all(probes) and len(probes) == 2
and fails == 0 and dmax is not None and dmax <= 0.05) else "FAIL"
The program runs in a sandbox: an AST gate, a python -I -S subprocess, resource limits, an audit hook, no network. The model's code never runs in the server process.
What happened
| Program | Printed conditions reproduced |
|---|---|
| Generic textbook template | 0 of 22 |
| The booklet's section 3.1 procedure implemented verbatim | 14 of 22 |
| Muse Wonder 1.0, 3rd attempt (2 regenerations) | 22 of 22 |
| Our own reference program | 22 of 22 |
The attempt log is more instructive than the result. Attempt 1 was rejected by the static gate (it used globals). Attempt 2 ran and passed 14 of 22, failing at the same intermediate step on eight rows (conditions 2.2, 3.1, 3.2, 6.2, 7.2, 8.2, 9.1, 9.2: a second trim correction). Those failing ids were fed back to Wonder, and attempt 3 passed 22 of 22. The three calls cost about $0.48 at list price.
The verbatim procedure scoring 14 of 22 is the point. Section 3.1 omits the second trim correction, and without it the high-trim rows miss by up to 0.38 %. The mark formulas divide by 140 although Lpp is 150, because the marks sit 140 m apart. The MTcm column is per degree by its heading although the legend says per cm. Following the text exactly is not enough (our verbatim implementation scores 14 of 22); graded against the printed numbers, Wonder found the missing convention on its third attempt.
On condition 4.1 the finished survey gives 17825.8 t against the printed 17827.7 t, a difference of -0.0106 %. That row is one of the 22 used to accept the program, so it is an in-sample check, not a held-out one.
The rest of the pipeline, briefly
SAM 3.1 tracks the painted numerals ("white number") and the sea ("water") on a 2x crop of the mark column; the crop and scale are needed because 18 px numerals on a full frame returned nothing. On a real third-party clip it found the marks in 60 of 60 frames, giving a draft of 12.761 m with a repeatability uncertainty of 0.086 m. Muse Spark 1.3 reads which box is which label, and five code checks (monotone, pitch, fit residual, two-frame agreement, a wide-glyph anchor) must pass before any label is used.
Where it refuses
Both real-ship clips certify as REFUSE OUT_OF_TABLE (a third clip, eeLL, was refused earlier by the label checks). The only booklet I have covers drafts of 0.5-9.3 m, and those ships sit at 12.8 m and 20.8 m. The program raises rather than extrapolate, and the certificate says so. An edited measurement is refused as unsigned. I think a tool that refuses to guess is the better default for a document that decides what a buyer pays.
Honest limits
- The weight check is on a rendered synthetic barge. It validates the arithmetic and the error model, not the models, and SAM does not detect rendered water.
- The real clips are third-party test footage with no ground truth, so they show repeatability and refusal, not accuracy.
- The test booklet is not a bulk carrier.
- Uncertainty is wave-averaging repeatability only. One clip reads one mark; the other positions are entered by hand.
- It is a prototype, not a surveyor's document.
Every figure above is in evidence/ or docs/DX-REPORT.md in the repo, and 124 tests pass. If you want to see it, the self-test re-runs live on the site with no login.
Top comments (0)