ML engineer focused on cost-efficient AI infrastructure. I benchmark GPU clouds and share practical guides for running LLMs in production.
📬 Telegram: https://t.me/qisuancloud
The fair comparison here is not output tokens alone, but the full session bill: with roughly 1,160 input tokens added per call, short replies can still lose on brief exchanges. That suggests measuring savings per task, not per reply, especially when cached context already dominates the input. For routine fixes the scannability gain alone may justify it, but for debugging an unfamiliar failure I would want the fuller reasoning the terse mode deliberately removes.
Agreed, per task is the fair way to measure it, since the added input can eat the saving on short exchanges. For debugging an unfamiliar failure I'd turn it off too.
ML engineer focused on cost-efficient AI infrastructure. I benchmark GPU clouds and share practical guides for running LLMs in production.
📬 Telegram: https://t.me/qisuancloud
Exactly — the per-task framing is what makes the comparison honest. Per exchange flatters the mode, because a short exchange hides the input overhead; per task you see the whole ledger, input included, and that's where short exchanges go underwater.
The boundary I'd draw on the debugging side is familiarity, not severity. For a failure I've seen before, terse output is fine — I already have the mental model, I just need the diff. For an unfamiliar failure, the reasoning trace and the surrounding context are the debugging material itself, so the mode goes off. The interesting middle case is a failure that's new but small: there I'd want terse output with full error context kept, rather than all-or-nothing.
Familiarity is a better line than severity, I agree. For the new but small case, keeping errors and commands verbatim while trimming the prose is close to how the mode already behaves.
ML engineer focused on cost-efficient AI infrastructure. I benchmark GPU clouds and share practical guides for running LLMs in production.
📬 Telegram: https://t.me/qisuancloud
Agreed — familiarity is the cleaner test, because severity alone doesn't tell you what the reader still needs. A severe error I've seen before is mostly a lookup; a small unfamiliar one can still hide the real cause.
One way to make that line operable is to ask whether I've seen this exact error-and-command combination before. If yes, trim the prose and keep the error and command verbatim — that's all I need to act. If no, keep the full context around it, even if the failure looks small, because that context is what lets me diagnose it instead of guessing.
The check I'd use is whether first-attempt fixes get better or worse after trimming, and whether retries go up. If the trimmed version still fixes familiar cases on the first try without extra rounds, the line is in the right place; if retries climb on the unfamiliar ones, we've trimmed too far.
Asking whether I've seen that exact error and command together is a cleaner test than how bad the failure looks. Watching retries after trimming is a good check too, since more retries on the unfamiliar ones would show I trimmed too far.
We're a place where coders share, stay up-to-date and grow their careers.
The fair comparison here is not output tokens alone, but the full session bill: with roughly 1,160 input tokens added per call, short replies can still lose on brief exchanges. That suggests measuring savings per task, not per reply, especially when cached context already dominates the input. For routine fixes the scannability gain alone may justify it, but for debugging an unfamiliar failure I would want the fuller reasoning the terse mode deliberately removes.
Agreed, per task is the fair way to measure it, since the added input can eat the saving on short exchanges. For debugging an unfamiliar failure I'd turn it off too.
Exactly — the per-task framing is what makes the comparison honest. Per exchange flatters the mode, because a short exchange hides the input overhead; per task you see the whole ledger, input included, and that's where short exchanges go underwater.
The boundary I'd draw on the debugging side is familiarity, not severity. For a failure I've seen before, terse output is fine — I already have the mental model, I just need the diff. For an unfamiliar failure, the reasoning trace and the surrounding context are the debugging material itself, so the mode goes off. The interesting middle case is a failure that's new but small: there I'd want terse output with full error context kept, rather than all-or-nothing.
Familiarity is a better line than severity, I agree. For the new but small case, keeping errors and commands verbatim while trimming the prose is close to how the mode already behaves.
Agreed — familiarity is the cleaner test, because severity alone doesn't tell you what the reader still needs. A severe error I've seen before is mostly a lookup; a small unfamiliar one can still hide the real cause.
One way to make that line operable is to ask whether I've seen this exact error-and-command combination before. If yes, trim the prose and keep the error and command verbatim — that's all I need to act. If no, keep the full context around it, even if the failure looks small, because that context is what lets me diagnose it instead of guessing.
The check I'd use is whether first-attempt fixes get better or worse after trimming, and whether retries go up. If the trimmed version still fixes familiar cases on the first try without extra rounds, the line is in the right place; if retries climb on the unfamiliar ones, we've trimmed too far.
Asking whether I've seen that exact error and command together is a cleaner test than how bad the failure looks. Watching retries after trimming is a good check too, since more retries on the unfamiliar ones would show I trimmed too far.