We build a consulting product. Not an API, not a chatbot — a thing you describe a business
problem to, and 81 seconds later a structured, CEO-ready document comes back that you could
put in front of a board.
That number came from a deliberate measurement, not marketing. Last night I ran the smallest
possible request through our pipeline and timed it end to end:
POST /v1/consulting/tasks { query: "Say hi in one sentence.", mode: consulting_report }
→ 11s running
→ 21s running
→ 71s running
→ 81s succeeded answer length: 2,178 characters artifacts: 1
→ balance charged, usage row written
A trivial prompt, and it still took 81 seconds. That is the honest cost of a multi-stage
pipeline: reframe the question, decompose it into sub-tasks, route each sub-task to a
different model, synthesise the results, then assemble a report. Anyone who tells you their
LLM product answers a strategic question in two seconds is either caching, or returning a
single model's stream-of-consciousness and calling it a deliverable.
What the deliverable is made of
The output is not a paragraph. It is a document with a fixed skeleton:
- Executive summary — the answer, in the form you could read aloud in a meeting
- Context and goals — what we understood the problem to be
- Key findings — the parts that surprised us, or that constrain the answer
- Recommended design — the actual proposal, with options and trade-offs
- Risks and mitigations — what would make this fail, and what we'd do about it
The interesting engineering problem is not generating text. It is making five sections that
agree with each other, when each was produced by a different model with a different context
window. That is where most of the 81 seconds goes, and it is why a single-model answer reads
fluent but incoherent if you check one section against another.
The boring parts nobody shows you
Three things we had to fix before the pipeline was worth shipping:
1. A dead upstream must not kill the whole run. Our router used to pick models from a
fixed per-domain list. When one provider's account ran out of credit, every request died with
HTTP 403 用户额度不足 — and the failure surfaced as a content string inside a 200 response.
Now: failures are detected as failures, the run does not get billed, and the router fails over
to a provider that is actually healthy.
2. A thinking model leaks its thinking. Some upstreams stream the entire answer in a
reasoning_content field and leave content empty. Our first fix was to copy it into
content so clients saw something; the second fix was to buffer it and only emit it if no
real content ever arrives. Streaming should give you the answer, not the rehearsal.
3. Never promise a channel you cannot deliver on. The product page used to say "leave and
we'll email you the report." For one day our transactional email quota was exhausted, which
meant leaving the page meant losing the report. The copy now says the report stays on the page,
and email is a if available extra. Two words change, one broken promise avoided.
What it's not
It is not a replacement for a consultant who knows your industry. It is the thing you do
before hiring one: getting a structured initial position, in a minute, so that the expensive
conversation starts from a draft instead of a blank page.
And it costs tokens, not retainer hours — which is exactly why it runs on a token-metered
pipeline rather than a per-seat subscription.
If you want to see the shape of the output, describe a problem you actually have. The report
appears in the page, usually under two minutes.
Top comments (0)