I got to measurements and got a number that forced me to reshuffle the whole plan.
One agent step: a context of about 6 thousand tokens, an answer of about 350 tokens: 77.5 seconds. Of those, 48 seconds went to prompt processing alone; generation was not even the main cost.
Now multiply. An agent rarely solves a task in one iteration; ten iterations is 13 minutes before the tests run. Add the tests and a second pass.
An unpleasant consequence followed: the whole value of the future stages, the queue, the isolation, the sandbox, rested on the assumption that a local model can drive an agent loop. And nobody had checked that assumption.
The decision: put a separate short experiment on five to ten real tasks in front of the construction. Not "I'll try and see", but with thresholds written down in advance: what percentage of tasks must be solved, in what time, how many broken answers are acceptable.
Top comments (0)