$87,000 vs roughly $1,000. Those were our first estimates for processing a corpus of 6.5 million semantic atoms with Sonnet or training and running a smaller model ourselves. I was ready to build the smaller model.
Our job was narrow. We needed to read financial text and record who said what, what they claimed, and how certain the source was. A wrong speaker or an invented certainty would become a record that other parts of the system trusted. Sonnet was our quality anchor. We would use it to label examples, train a Qwen2.5-7B-Instruct student, and run the student over the corpus. Kepler, my AI data scientist agent, built the training harness. Owning the model felt like an answer to the price problem and, eventually, an advantage we could keep improving.
Then Tycho, our FinOps and science agent, pushed us to test models we could rent directly. The first bake-off used a 23-case, hand-built set of difficult examples:
| Model | Score | Role in the test |
|---|---|---|
| Claude Sonnet 5 | 21/23 | Proposed teacher and quality reference |
| DeepSeek V3.2 | 18/23 | Lower-cost API candidate |
| Qwen3-Next-80B-A3B | 8/23 | Open-weight candidate, distinct from our Qwen2.5 student |
At first, that table strengthened the case for distillation. DeepSeek missed more hard cases than Sonnet. A student trained only on Sonnet's answers would inherit a different problem: the teacher itself missed two cases, so its labels could not teach the correct answer there. We expected to add our own corrections as we found failures.
What the test was missing
The extraction rubric had been treating uncertainty as something to overcome. Sometimes the source does not support a definite speaker or a definite claim. We added Unknown as a legitimate output and used the unknown cases to examine where our instructions were pushing a model past the evidence. That gave us a cheaper experiment than another training run: change the rubric, run fresh hard cases, and check that the model had not learned to answer Unknown to everything.
In a later evaluation, DeepSeek V4.1 Flash with the corrected rubric passed 46/46 cases across three separate held-out sets. The six clear control cases stayed clean. Those 46 cases are separate from the original 23-case bake-off, and V4.1 Flash is a different model from V3.2. The table above and the later result cannot be subtracted to calculate how much improvement came from the rubric alone. The later tests showed that an available API model could clear our bounded hard-case bar under the revised instructions. They did not prove performance across all 6.5 million atoms.
We had built a student baseline, but we never benchmarked that student on the hard-case set or served it over the full corpus. There is no honest student score to put beside DeepSeek's later result.
The bill for the next correction
Our modeled cost for training and running the student was roughly $1,000 in GPU time. The optimized DeepSeek route also modeled at roughly $1,000 for this extraction stage. Neither number is an invoice. The API estimate depends on caching, and the student estimate leaves out the engineering and operations required to serve it. Once the compute estimates were close, the decision moved to something we had barely priced: how long it would take to incorporate the next correction.
With an API model, we could revise the rubric and test the next batch. With our student, a correction intended to live in the weights would require new training, evaluation, and deployment. We might still do that someday. For this workload, we had no reason to pay that cycle before testing whether the instructions were the source of the failure.
I had been thinking of the trained model as the asset. It is an asset, but it captures what we knew at training time. The rubric and the test cases are assets too; they can also be wrong or go stale. The advantage I care about now is the rate at which we find an error, inspect the source, correct the judgment, and test that correction against new cases and unaffected controls. We thought we were building that learning into a model we owned. The faster route, at least for this job, was to keep the learning loop outside the weights.
About the cost estimates
The $87,000 was our initial Sonnet estimate; correcting the number of calls reduced the all-Sonnet projection to about $58,000. The student and optimized DeepSeek figures are modeled costs for the extraction stage, not the whole pipeline. The current uncached Fireworks batch projection for DeepSeek is about $2,151. The student was not served over the corpus.
This is the engineering companion to my original note at InflectAI.
Top comments (0)