StarspireGavren48

Posted on Sep 24

Fintech Moderation Text Summarization API: Portable Chat Completions JSON Output

#ai #node #observability

The least complex useful Node.js design is a two-stage text summarization API: send each moderation report through a chat request, validate its JSON output, then classify the compact record before human review. The expensive surprise is often outside the model call. If every prompt, response, retry, and intermediate chunk becomes an indexed log event, retained telemetry can outweigh the application data you meant to keep.

Short answer: define one provider-neutral summary contract, validate every response, and record measurements rather than report text. Keep the original report in the system of record under its own access and retention policy. For observability, retain request identifiers, timings, status, token counts when supplied, payload byte counts, schema version, and a bounded result label. Sample sanitized diagnostic bodies briefly and deliberately. This preserves portability and enough evidence to operate the pipeline without turning sensitive fintech reports into a second, loosely governed archive.

What is the bill actually made of?

Start with bytes retained, not the number of API calls. A useful planning equation is:

monthly stored bytes = events per day × bytes per event × retained days × index/replica multiplier

The final multiplier depends on the telemetry system, so measure it rather than borrowing a generic ratio. The first three terms are already enough to expose the dominant choice. Consider an illustrative workload of 100,000 reports per day. If the service emits four 12 KB events per report and retains them for 30 days, the raw monthly event volume is 144 GB before indexing, replication, or metadata. Reducing those events to four 600-byte records yields 7.2 GB under the same assumptions. This is capacity math, not a benchmark or a vendor price claim.

The change that moves the dominant term is straightforward: do not place the source report, chunk bodies, or generated summary in routine logs. Payload size is multiplicative. Retention tuning helps, but shrinking the event first improves every retention tier and every replica.

Cardinality is the second bill. A label such as outcome=accepted has a bounded set of values; report_id, raw error text, a prompt hash, or a free-form category can create a new time series or index term for nearly every request. Keep unique identifiers in trace fields or logs only where lookup requires them, and keep them out of metric dimensions. OpenTelemetry's attribute guidance makes the same distinction practical: attributes are useful context, but sensitive data and unbounded values require care.

I would budget the telemetry before choosing a provider. The planning table is intentionally small:

Signal	Keep routinely	Retention decision
Metrics	Request count, latency histogram, bounded outcome, schema version	Long enough to compare releases and seasonality
Traces	Stage timings, retry count, request correlation, no body	Sample successes; retain errors more aggressively
Logs	Validation failures, status, byte counts, redacted error class	Short operational window
Bodies	Original report in the governed system of record	Follow the report's legal and review policy

Do not copy a report merely because a tracing SDK makes recording an attribute convenient. In fintech moderation, the copied text may contain account details, allegations, or other material with a narrower audience than the engineering log store.

Which JSON contract survives a provider change?

Portability begins at the application boundary. Ask for a compact object whose fields describe the human-review job, not one provider's response envelope. For example:

{
  "schema_version": "1",
  "summary": "Customer reports repeated pressure to bypass an account control.",
  "category": "social_engineering",
  "urgency": "high",
  "evidence": [
    { "start": 418, "end": 503 }
  ]
}

The evidence entries are source offsets, not generated quotations. They let the review interface retrieve the authoritative text without duplicating it into the model result or telemetry. Offsets also fail visibly if preprocessing changes the source, which is preferable to presenting an untraceable paraphrase as evidence.

Keep the provider adapter thin: translate a local request into a chat-style request, obtain one response, and return only the candidate JSON plus usage metadata. Validation belongs after the adapter. Require the schema version, cap summary length, enumerate category and urgency values, reject additional properties, and verify that every evidence range is ordered and within the normalized source. A syntactically valid object can still be operationally false.

Here is the shape of a portable request using curl. The endpoint and model are deployment configuration, while the contract remains application-owned:

curl --fail-with-body --silent --show-error \
  --request POST "${AI_BASE_URL}/chat/completions" \
  --header "Authorization: Bearer ${AI_API_KEY}" \
  --header "Content-Type: application/json" \
  --header "traceparent: ${TRACEPARENT}" \
  --data @- <<'JSON'
{
  "model": "${AI_MODEL}",
  "messages": [
    {
      "role": "system",
      "content": "Summarize this moderation report for human review. Return one JSON object with schema_version, summary, category, urgency, and evidence source offsets. Do not add facts."
    },
    {
      "role": "user",
      "content": "REPORT_TEXT_INSERTED_BY_THE_CALLER"
    }
  ]
}
JSON

Environment-variable substitution does not occur inside the quoted heredoc; a production caller should serialize the request with its JSON library. The block shows the wire contract, not a shell templating technique. Do not build JSON by concatenating untrusted report text.

Chat-shaped endpoints are widespread, but their structured-output controls and usage fields are not guaranteed to be identical. Treat those as adapter capabilities. The local validator remains authoritative, and the application should map a missing token count to unknown instead of guessing.

How should a text summarization API use chat completions for long reports?

Measure before chunking. If a normalized report fits the configured input budget with instructions and output headroom, send it once. Chunking every report multiplies latency, failure opportunities, and telemetry volume.

For an oversized report, split on stable structural boundaries such as paragraphs or message turns, attach source offsets, and summarize chunks into the same narrow intermediate schema. A final reduction pass should receive those intermediate records, not the entire original text again. The reducer can merge duplicate categories and select evidence offsets, but it must never invent a range absent from its inputs.

There is a sharp trade-off here. Larger chunks preserve context but increase retry cost and make diagnostic samples more sensitive. Smaller chunks isolate failures, yet they can separate a coercive request from the sentence that makes it coercive. The correct boundary is therefore an evaluation result, not a fashionable token count. Build a fixed test corpus containing long conversations, repeated boilerplate, contradictory statements, Unicode text, empty sections, and reports whose decisive evidence crosses a proposed boundary.

Track classification agreement and evidence-range validity separately. A summary may read well while pointing reviewers to the wrong passage.

For backfills, an asynchronous batch facility can be appropriate because the work is not on the reviewer interaction path. Keep that decision behind the same adapter, and correlate each batch item with an opaque local identifier. Interactive reports still need explicit deadlines, bounded retries with jitter, and an idempotency strategy at the job layer so a timeout does not create two review records.

Telemetry that answers questions without retaining reports

A useful trace has spans for normalization, each summarization call, validation, reduction, classification, and persistence. W3C Trace Context defines the traceparent mechanism for propagating trace identity across process boundaries. Propagation does not imply that report text belongs in the trace.

Record a compact event after validation:

{
  "event": "moderation_summary_completed",
  "schema_version": "1",
  "input_bytes": 28614,
  "output_bytes": 742,
  "chunk_count": 2,
  "attempt_count": 1,
  "duration_ms": 1840,
  "outcome": "accepted",
  "usage_source": "provider_reported"
}

Those numbers are an example event shape, not observed performance. In a real event, duration and byte counts come from instrumentation, and usage_source distinguishes provider-reported usage from an unavailable value. Avoid pretending that a local character count is an exact token count across tokenizers.

Sampling needs two independent controls. First, trace sampling decides how many executions retain detailed timing. Second, diagnostic-content sampling decides whether a sanitized body is retained at all. Conflating them is dangerous: raising trace sampling during an incident should not silently raise the amount of sensitive content stored.

Use a tiny, access-controlled diagnostic corpus with an explicit expiration only when body-level debugging is justified. Sample by a stable hash of an opaque report identifier so repeated attempts make the same decision, but never expose that identifier as a metric label. Errors deserve a higher trace sampling rate, though even an error is not permission to retain its input.

The deliberate omission is full prompt and response logging. When an incident depends on exact wording, this choice makes retrospective debugging harder. You may know that validation failures rose after a schema change without possessing every rejected body. Compensate with reproducible redacted fixtures, versioned prompts, schema hashes, bounded error classes, and a controlled reprocessing path against the governed source. The cost is slower diagnosis for rare semantic failures. The benefit is that the observability platform does not become an accidental moderation-report database.

Deployment rules for a review-critical pipeline

Release the prompt, schema, normalization rules, and adapter configuration as separately identifiable versions. Route a small portion of eligible traffic to a candidate version, but compare it against a frozen evaluation set before expanding. Online metrics can detect latency and validation regressions; they cannot establish that the new summaries preserve the facts reviewers need.

Retries should be selective. Retry transport interruptions, throttling, and transient server failures within a deadline. Do not retry the same malformed response indefinitely. One constrained repair attempt can be reasonable if it receives the validation errors and no new source material; after that, place the job in a reviewable failure state. Record the error class, not the raw response.

Provider portability should be exercised, not inferred. Run the same conformance corpus through each configured adapter and require the same local invariants. Compare schema-valid rate, evidence validity, category agreement, tail latency, and telemetry bytes per completed report. Usage units may differ across providers, so retain their provenance and avoid combining unlike counters into a deceptively precise metric.

Stop keeping intermediate chunk text, raw model responses, and unique identifiers in high-cardinality metric labels. Keep the governed source, the accepted review artifact, bounded operational signals, and enough version metadata to reproduce the path. That is a smaller evidence trail, by design, and its limits should be part of the incident runbook rather than discovered during an investigation.

DEV Community