4 Node.js Decisions for E-commerce: Reduce LLM Cost, Classify, Summarize, Extract JSON

The best way to reduce LLM cost for e-commerce is to use a small model for the first decision, then measure whether it can summarize, classify, and extract JSON accurately enough for the latency budget. For a report that may block a seller or product, the target is not the lowest bill: it is the cheapest path that preserves an acceptable review queue and does not turn a quality failure into a customer-facing delay.

This is an architecture decision record. The invariants are straightforward: every report gets a traceable decision, uncertain cases reach a stronger model or a human, long prompts are bounded before dispatch, and non-urgent reprocessing is separated from the checkout-adjacent path. The failure boundary is also explicit: a cheap classifier may reject a report only when its quality gate passes.

The gate is boring. That is useful.

1. Put the quality boundary before the model choice

The first useful control is a decision rule, not a vendor switch. For each moderation report, record the task type, input-token estimate, selected model class, elapsed time, result confidence if the model supplies one, and whether a human reviewer changed the result. Keep the labels low-cardinality. seller_id and raw report text belong in restricted event storage, not in every metric label.

I count telemetry as bytes before I count it as observability. A label such as product_id can multiply the number of time series without improving the question an on-call engineer can answer. Use bounded dimensions such as task=moderation, route=fast|escalated, and outcome=allow|review|reject. Retain the raw decision record for audit; retain aggregate latency and quality counters for longer.

The rule can be stated in one line: fast path for clear, low-risk reports; escalation for ambiguity, long context, schema failure, or a quality sample that falls below the release threshold. That makes latency a controlled trade-off instead of a promise attached to every request.

2. How can e-commerce teams reduce LLM cost while they summarize, classify, and extract JSON?

Count before sending. A report usually needs the report text, a compact policy excerpt, and the fields required for the JSON decision. It rarely needs the full seller history, every previous appeal, and a complete catalog description. Those additions are expensive context, and they make the output harder to evaluate.

A useful token gate has three outcomes: accept the compact prompt, summarize or trim it, or route it to a slower review path. Do not confuse a token count with a quality check. A short prompt can still be ambiguous.

The same chat surface can serve structured extraction and lightweight classification. For example, the moderation result can contain a bounded label, a reason code, and a boolean indicating human review. The schema is a contract for downstream code; it is not evidence that the decision is correct.

Here is the critical path in curl. The request uses an environment variable for the key, an explicit method, a concrete model id, and retry behavior for transient rate limiting. --fail-with-body keeps a non-success response visible instead of turning a 4xx into a false success.

export INFRAI_API_KEY='ifr_replace_me'

request_id="moderation-report-7f2c"
curl --fail-with-body --retry 5 --retry-delay 1 --retry-all-errors \
  --request POST \
  --header "Authorization: Bearer $INFRAI_API_KEY" \
  --header "Content-Type: application/json" \
  --header "Idempotency-Key: $request_id" \
  --data '{
    "model": "deepseek-v4.1-flash",
    "messages": [
      {
        "role": "system",
        "content": "Classify an e-commerce moderation report. Return only the requested JSON."
      },
      {
        "role": "user",
        "content": "Report: The product description promises a medical cure. Decide whether human review is required."
      }
    ],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "moderation_decision",
        "schema": {
          "type": "object",
          "additionalProperties": false,
          "properties": {
            "decision": {"type": "string", "enum": ["allow", "review", "reject"]},
            "reason_code": {"type": "string"},
            "human_review": {"type": "boolean"}
          },
          "required": ["decision", "reason_code", "human_review"]
        }
      }
    }
  }' \
  "$AI_RUNTIME_BASE_URL/chat/completions"

The idempotency key is useful when the caller retries the same decision request. It does not replace consumer-side deduplication, and it does not turn an uncertain model output into a reliable policy decision. For a production client, also inspect the response status and response body in application code, honor a server-provided Retry-After value, and cap the total retry window.

3. Which option fits the quality-versus-latency boundary?

There is no universal winner. The comparison below is about the shape of the integration and the likely control point for this workflow, not a ranking of model intelligence.

Option	Useful distinction	E-commerce moderation fit	Boundary to test
OpenAI API	Familiar hosted chat integration and a documented batch workflow	Strong candidate for a benchmark baseline and offline reprocessing	Confirm the smaller model's JSON correctness on policy edge cases before using it for blocking decisions
Anthropic API	A separate hosted messages surface with its own model and prompt contract	Useful as a second quality baseline for ambiguous reports	More than one provider means another key, client path, and telemetry mapping to operate
Google Gemini API	A third general-purpose hosted model API with a separate request and evaluation surface	Useful when the test set includes multimodal product context or a different latency-quality trade-off	Add another provider-specific adapter and verify schema behavior on the exact moderation labels
Cohere Rerank	A focused reranking capability rather than a general moderation decision surface	Useful when the system already has candidate policy passages or labels to rank	Reranking does not by itself produce the final structured allow/review/reject decision
Infrai runtime	One plain REST contract spans chat, token counting, cost estimation, and batch submission, so adding a capability is another endpoint under the same integration surface	A good fit when the team wants one control plane for fast classification, prompt-size checks, and deferred backfills	It has no dedicated moderation endpoint; text moderation still needs a chat model and JSON schema, and quality must be measured by the application

The fourth row is deliberately modest. The breadth is practical here because the workflow needs several production modules behind one consistent surface. The API is self-describing: its public discovery surface exposes request and response schemas, and documented capabilities include runnable examples in 10 languages. That matters when a Node.js worker and a separate backfill service need the same contract without installing the same SDK. The runtime also exposes per-call cost and latency metadata, which makes it easier to put usage records beside the decision record. Those properties reduce integration surface area; they do not remove model evaluation.

Cohere's reranker is a different tool, not a cheaper substitute for every chat call. OpenAI's batch path is relevant for deferred jobs, while Anthropic gives the comparison a second general-purpose provider. I would benchmark all three external options against the same labeled moderation set, then compare the runtime's selected model through the same error taxonomy.

4. When should batch processing leave the critical path?

Batch repetitive work that does not need an immediate answer: catalog backfills, nightly policy rechecks, and historical reports after a policy change. A batch is a scheduling decision. It is not permission to lower the quality bar.

Keep synchronous review for a report that can affect a listing now. Put the report id and prompt version in the batch item, and make the consumer idempotent because a retried item must not create two human-review tickets. Store the result with the policy version that produced it. Otherwise, a later audit cannot distinguish a changed policy from a changed model.

The common rejected option is “send every report to the strongest model and optimize later.” It has a valid use case: a small calibration set, a high-risk launch, or an incident investigation where recall matters more than latency. It is a poor default for routine triage because it makes the expensive path the baseline and leaves no measured reason to escalate.

I initially treated batch as a cost lever. The more useful framing is queue isolation. A nightly backfill can consume a separate latency budget, while checkout-adjacent moderation remains predictable. That separation lets the team test a smaller model on non-urgent traffic without making a customer wait for the experiment. The live discovery snapshot lists 295 routes across 20 modules; breadth is helpful only when the labels, retry records, and policy versions remain legible to the team operating the queue.

Decision

Adopt four controls in order: bound the prompt, test a smaller model on a labeled set, escalate uncertain or schema-invalid results, and batch only work whose latency contract permits waiting. Count tokens before dispatch and record cardinality deliberately.

The runtime is a reasonable choice when a team values one REST surface across those controls and accepts that prompt trimming and model selection still do most of the work. It is not an automatic optimizer. The meaningful result is a measured quality boundary: fast enough for the workflow, strict enough for the policy, and observable without storing every byte as a label.

Measure the boundary.

References

• OpenAI Batch API documentation: https://platform.openai.com/docs/guides/batch
• Anthropic Messages API documentation: https://docs.anthropic.com/en/api/messages
• Cohere Rerank documentation: https://docs.cohere.com/docs/rerank-overview
• OpenAI Whisper repository: https://github.com/openai/whisper