CloudveilElenor12

Posted on Sep 21

Authenticated Web App Chatbot API: Candidate Scoring with Bounded Telemetry

#ai #backend #observability

TL;DR: For an authenticated property-management chatbot that scores job candidates, the simplest useful backend API is a thin server-owned streaming boundary, not a browser-to-model connection or a thick SDK wrapper. Authenticate before opening the stream, validate one versioned rubric, emit a small stable event vocabulary, and persist the final scoring record separately from token events. Choose the runtime only after replaying a fixed evaluation set against two budgets: scoring quality and end-to-end latency. This keeps substitution possible, but the more immediate benefit is better evidence with fewer telemetry bytes.

The hard requirement is not displaying tokens. It is letting a hiring manager ask, "How does this candidate match our assistant property manager rubric?" while keeping authorization, rubric versions, retries, and audit evidence coherent. Streaming improves perceived responsiveness, yet a partial explanation must never become the official score.

What should an authenticated web app chatbot backend API own?

The browser should send a candidate reference, a job reference, and the user's message to an application endpoint. The server resolves those references inside the authenticated tenant, selects the active rubric, constructs the model request, and streams display-only progress back. It records the final structured result only after validation. Credentials remain on the server. Authorization also remains there; possession of a candidate identifier is not proof that the signed-in user may read that candidate.

That separation is small, but consequential. An SDK can be replaced. Tenant isolation cannot be delegated to whichever model client happens to be convenient this quarter.

Keep it boring.

Use a narrow event contract. For example, status can say that rubric evidence is being assessed, delta can carry provisional prose, result can carry the validated score object, and error can terminate the attempt. Four event names are enough. Do not turn each internal stage, model family, or rubric dimension into a new event name; that vocabulary will leak into clients and telemetry labels.

A request might look like this from a signed-in web session:

curl --no-buffer 'https://app.example.test/api/candidate-score' \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Accept: text/event-stream' \
  --header 'Cookie: session=REDACTED' \
  --header 'Idempotency-Key: 018f-example-attempt' \
  --data '{
    "candidate_id": "cand_4821",
    "job_id": "job_apm_17",
    "message": "Score this candidate and cite rubric evidence."
  }'

The example uses Server-Sent Events because the interaction is predominantly server-to-browser and ordered. The wire choice is less important than the invariant: only a terminal, schema-valid result is eligible for persistence as a hiring artifact. A disconnected stream may be restarted under the same idempotency policy, but it must not create two official assessments.

Why isn't token streaming the main latency metric?

A fast first token can conceal a slow or unusable decision. Track at least three durations: request acceptance to first meaningful event, request acceptance to validated result, and validation time after generation. The second is the operational latency budget. The first describes user perception. Mixing them rewards chatty runtimes that begin quickly and finish late.

Quality also needs a concrete definition. Build a frozen evaluation set from synthetic or properly governed candidate profiles, each paired with a rubric version and expected evidence boundaries. Useful checks include schema validity, whether every awarded rubric point cites supplied evidence, whether absent evidence is treated as absent, and whether protected or irrelevant attributes affect the score. Prompt-writing guidance can improve instructions, but prompt prose is not an evaluation method.

Run the same set against every candidate runtime configuration. Record distributions rather than a single average: completion latency can have a long tail, and a hiring workflow feels that tail when several profiles are reviewed together. Do not invent a universal threshold. Set one from the product's interaction budget and the organization's review process, then publish the acceptance rule before testing.

Quality wins the tie. A configuration that produces an early stream but violates the rubric is not a faster solution to the same problem. It is defective behavior. Once configurations clear the quality floor, latency can decide among them.

Embeddings may help retrieve rubric guidance or relevant approved material, but vector similarity does not establish that a candidate meets a criterion. Retrieval supplies context; the scoring record still needs explicit evidence and a rubric version. Keep those responsibilities separate.

Count telemetry before choosing a runtime

Observability cost is mostly a multiplication problem. Let A be scoring attempts per day, E the average events retained per attempt, B the average stored bytes per event after enrichment, and D the retention days. A first-order storage estimate is A x E x B x D. Index overhead, replication, and query processing sit outside that estimate, so it is a floor rather than a bill forecast.

Consider a planning example, not a benchmark: 20,000 attempts per day, 18 retained events per attempt, 1,200 stored bytes per event, and 30 days of retention produce 12.96 GB of raw event payloads. Retaining 600 token-level events instead would produce 432 GB before indexes or replicas. The model response did not become more correct. The telemetry merely became noisier.

Keep low-cardinality dimensions as indexed attributes: environment, outcome, rubric version, stream transport, and coarse latency bucket. Keep request IDs, candidate IDs, user IDs, trace IDs, free-form prompts, and error text out of metric labels. They belong in access-controlled records or sampled traces when policy permits. A label with one value per request creates a series per request; retention math then stops being a simple log-volume concern and becomes a cardinality problem.

Trace context should cross the application-to-runtime boundary, but candidate data should not be placed in trace headers. Record payload size, event counts, and timing without recording the payload by default.

Sampling needs two lanes. Retain aggregate counters for every attempt, including outcome and latency histograms. Sample detailed traces, while retaining policy-approved traces for rare failures or schema rejections. Head sampling is cheap but cannot know the eventual outcome; tail sampling can make outcome-aware decisions but requires buffering and additional collector capacity. The choice should follow the questions operators must answer.

Signal	Keep broadly	Sample or restrict	Reason
Attempt count and outcome	Yes	No	Establishes rates without candidate content
First-event and final-result latency	Yes, as histograms	Raw exemplars	Preserves distributions with bounded dimensions
Rubric version	Yes, if version count is controlled	Retired versions after policy window	Connects behavior to a scoring contract
Token events	No	Short-lived diagnostic traces	High event volume, weak audit value
Prompt and candidate evidence	No	Authorized audit record	Sensitive content needs separate access and retention
Request and trace identifiers	No metric labels	Logs or traces	Nearly unique values cause cardinality growth

The audit record answers why a score was accepted and under which rubric. Operational telemetry answers whether the system is healthy. Copying complete conversations into both increases exposure and retention cost without creating independent evidence.

Less is evidence.

Compare boundaries, not client libraries

A selection exercise should compare deployment boundaries after the contract and evaluation set exist. A direct hosted inference API minimizes infrastructure ownership, but the application team still owns authentication, tenant authorization, validation, retries, audit storage, and error normalization. A managed gateway can centralize routing and policy, at the cost of another service boundary and telemetry surface. A self-hosted runtime offers control over models and scheduling, while transferring capacity planning, upgrades, and tail-latency work to the team. These are operating models, not rankings.

The thin streaming boundary has a real limitation: it is not suitable when the product needs bidirectional, low-latency audio, client-originated events throughout generation, or long-lived collaborative state. A duplex transport and a stateful session service fit those cases better, even though they add connection management and operational cost. The direct hosted option is also a poor fit when policy requires inference inside a controlled network; self-hosting may then be necessary despite its capacity burden. Conversely, self-hosting is hard to justify for a team that cannot staff model serving and on-call response. This trade-off should be decided from constraints, not familiarity with a client library.

Score each option on the same evidence. Can it preserve cancellation through the application server? Does it expose usage and finish metadata in a stable terminal response? Can timeouts be set independently for connection, first event, and completion? Can the team replay the evaluation set in staging? What bytes will be emitted at full traffic?

The last question belongs in the selection worksheet. Estimate retained events per attempt and bounded label values before load testing. Then verify the estimate during a replay. An integration that looks simple at ten requests can become expensive when every streamed fragment is logged at three layers.

Avoid treating protocol resemblance as full portability. Similar JSON request shapes do not guarantee identical tokenization, tool behavior, safety behavior, streaming termination, or error semantics. Put a small adapter behind the application's own contract, and test behavior there. The adapter should translate, not decide hiring policy.

Roll out with a reversible scorecard

Start with shadow evaluation using non-production or properly governed records. No streamed output should influence a hiring decision during this phase. Compare rubric adherence, unsupported claims, schema rejection rate, time to first meaningful event, time to validated result, and telemetry bytes per attempt.

Next, expose the chatbot to a limited authorized cohort while keeping the result advisory and visibly tied to a rubric version. Alert on missing terminal events, duplicate idempotency keys, authorization failures, and shifts in validated-result latency. Review sampled traces under the same access policy as the underlying candidate data.

Then expand only when the predeclared quality floor and latency budget both hold. Keep the previous adapter available during the observation window, but do not split one official assessment across two runtimes. A retry must either recover the same attempt according to the contract or begin a clearly identified new attempt.

The durable asset is the decision record, not the stream. A modest backend contract, a fixed evaluation corpus, and intentionally sparse telemetry make runtime changes testable. They also keep the property-management team focused on the real question: whether the score is supported, timely, and reviewable.

DEV Community