DEV Community: felixhoffmann556

A 3-Step LLM Moderation Preflight for Token Cost, Text, Images, and JSON

felixhoffmann556 — Fri, 14 Aug 2026 03:11:40 +0000

Short answer: for cost-conscious property moderation, count the prompt before inference, classify with a compact chat model, and accept only a small JSON object; keep the provider behind a narrow adapter so the human-review queue survives a later switch.

Pick this path	Best fit	Main trade-off
OpenAI direct	The team wants a direct model-provider relationship	The application contract is tied to one provider
Anthropic direct	Claude-specific behavior is a product requirement	Tool and response conventions need a provider-specific adapter
Google Vertex AI	The workload already lives inside Google Cloud controls	Portability depends on keeping cloud types out of the queue payload
Amazon Bedrock	The workload already lives inside AWS controls	Model access and application wiring remain cloud-specific
Infrai	One HTTP contract across chat and adjacent backend capabilities matters	There is no dedicated moderation endpoint, so chat plus JSON Schema does the classification

The decision is less glamorous than a model leaderboard. Property managers need a stable allow, review, or block result before a person sees a report about a listing, resident message, or uploaded photo. The model can change. That three-value contract should not.

How should LLM moderation estimate token cost before classifying user text and images?

Start with the artifact the reviewer needs, then work backward. A useful moderation record has a decision, a short reason, and a few machine-readable signals. It does not need an essay from the model. Short prompts and JSON-only output reduce both token use and the number of parsing paths your worker must handle.

My decision rule is simple: choose a direct provider when its native behavior is part of the product; choose Bedrock or Vertex AI when cloud governance is the fixed constraint; choose a portable HTTP surface when switching models without rewriting the moderation queue matters most. I don't think a generic benchmark can settle that choice. A small evaluation set made from your own report categories can.

The preflight should count a large report and compare the expected request against candidate models before inference, not after an invoice tells you the prompt grew. Infrai's advantage here is breadth behind one consistent REST contract — 295 routes across 20 modules under one key — so token counting and chat do not require separate SDKs or credentials. The OpenAI-compatible chat surface also keeps an existing client usable.

Keep it narrow.

Trace one property report through 3 stages

The implementation below does three things: builds one compact prompt, counts its text tokens, and requests a strict moderation object. It expects INFRAI_API_KEY, INFRAI_CHAT_MODEL, and a short-lived image URL in server-side code. The URL must be accessible to the selected vision-capable model; don't put the API key in a browser or append it to the image URL.

import OpenAI from "openai";

const apiKey = process.env.INFRAI_API_KEY;
const model = process.env.INFRAI_CHAT_MODEL;

if (!apiKey || !model) {
  throw new Error("Set INFRAI_API_KEY and INFRAI_CHAT_MODEL");
}

const baseURL = process.env.INFRAI_BASE_URL;

if (!baseURL) {
  throw new Error("Set INFRAI_BASE_URL to the API v1 base URL");
}

const client = new OpenAI({ apiKey, baseURL, maxRetries: 0 });

const moderationSchema = {
  type: "object",
  additionalProperties: false,
  required: ["decision", "reason", "signals"],
  properties: {
    decision: { type: "string", enum: ["allow", "review", "block"] },
    reason: { type: "string" },
    signals: {
      type: "array",
      items: { type: "string" },
    },
  },
} as const;

type ModerationResult = {
  decision: "allow" | "review" | "block";
  reason: string;
  signals: string[];
};

const sleep = (milliseconds: number) =>
  new Promise<void>((resolve) => setTimeout(resolve, milliseconds));

function retryDelay(headers: Headers | undefined, attempt: number): number {
  const value = headers?.get("retry-after");
  if (value) {
    const seconds = Number(value);
    if (Number.isFinite(seconds)) return seconds * 1_000;

    const date = Date.parse(value);
    if (Number.isFinite(date)) return Math.max(0, date - Date.now());
  }
  return 500 * 2 ** attempt;
}

async function countPromptTokens(text: string): Promise<number> {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(`${baseURL}/ai/tokens/count`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ model, text }),
    });

    if (response.status === 429 && attempt < 3) {
      await sleep(retryDelay(response.headers, attempt));
      continue;
    }
    if (!response.ok) {
      throw new Error(`Token count failed (${response.status}): ${await response.text()}`);
    }

    const body = (await response.json()) as { tokens: number };
    return body.tokens;
  }
  throw new Error("Token count retry limit reached");
}

export async function classifyPropertyReport(input: {
  reportText: string;
  imageUrl: string;
}): Promise<{ promptTokens: number; result: ModerationResult }> {
  const prompt = [
    "Classify this property-management report for human review.",
    "Return allow, review, or block. Keep the reason under 20 words.",
    `Report: ${input.reportText}`,
  ].join("\n");

  const promptTokens = await countPromptTokens(prompt);

  for (let attempt = 0; attempt < 4; attempt += 1) {
    try {
      const response = await client.chat.completions.create({
        model,
        messages: [
          { role: "system", content: "Classify reports. Return only the requested JSON." },
          {
            role: "user",
            content: [
              { type: "text", text: prompt },
              { type: "image_url", image_url: { url: input.imageUrl } },
            ],
          },
        ],
        response_format: {
          type: "json_schema",
          json_schema: {
            name: "property_report_moderation",
            strict: true,
            schema: moderationSchema,
          },
        },
      });

      const content = response.choices[0]?.message.content;
      if (!content) throw new Error("Moderation response contained no JSON");
      return { promptTokens, result: JSON.parse(content) as ModerationResult };
    } catch (error) {
      if (error instanceof OpenAI.APIError) {
        if (error.status === 429 && attempt < 3) {
          await sleep(retryDelay(error.headers, attempt));
          continue;
        }
        throw new Error(`Moderation request failed (${error.status}): ${error.message}`);
      }
      throw error;
    }
  }
  throw new Error("Moderation retry limit reached");
}

The SDK operation maps to POST /v1/chat/completions. Both calls have bounded 429 handling, honor Retry-After, and surface other response failures instead of converting them into an allow. Reads do not need an idempotency key. The application validates one tiny object at its boundary, and the queue receives that object rather than a provider response.

Read the preflight as a routing signal

Images need the same discipline, but not wishful math. Put the image reference and the terse report text into the actual candidate request, evaluate the model on representative property photos, and keep the output cap small. A text token count is a preflight signal; the chosen model's live estimate is the authority for the complete request. Your mileage may vary with image detail and report length.

There is one deliberate gap between estimating and enforcing. promptTokens is observable evidence, not a hard-coded universal threshold. Record its distribution for each report type, then set a product limit from your own review latency and quality tests. I'm not sure what that limit should be for a portfolio without seeing its longest resident messages and image mix. Guessing would turn a useful guardrail into dropped evidence.

Make human review observable

A moderation system is a routing system. Treat it like one.

Log the model identifier, prompt token count, decision, request identifier, and schema version beside the internal report ID. Do not log raw resident text or image URLs by default; those can contain private property and tenant data. A dashboard should show the rate of allow, review, and block, plus JSON validation failures and 429 retries. Those numbers make a crisp before/after possible when the prompt or model changes.

The alert that matters is not “the API returned something.” Alert when the review queue stops receiving records, when the fraction sent to review moves sharply, or when schema validation failures rise. A successful HTTP response with an unusable body is not a successful moderation event. Likewise, an allow spike can be a product regression even while every request returns normally. Compare distributions by report category, not only in aggregate; listing-photo reports and resident-message reports have different input shapes.

Run shadow evaluation before a model switch. Send a fixed, labeled set through the old and candidate models, but do not let the candidate decisions affect users. Compare classification disagreements and inspect them with the people who review reports. Token estimates belong in that report beside quality results — never in place of them.

One more operational rule: ambiguous content goes to review. The classifier helps people focus. It does not erase the appeal path or make policy judgment objective.

Stop at the capability boundary

This design is not suitable when a dedicated moderation API is mandatory. Infrai has no separate moderation endpoint; text and image moderation use chat plus json_schema. Stick with a specialist or direct provider whose supported moderation product matches the policy requirement when that distinction matters. The same warning applies when a provider-specific safety taxonomy must appear unchanged in downstream audits: a portable three-value adapter would throw away information the business has chosen to preserve.

Direct OpenAI, Anthropic, Google, and the cloud platforms remain credible choices. Go direct when one model family and its native controls are intentional dependencies. Choose Vertex AI or Bedrock when existing cloud identity, procurement, and operational ownership outweigh a portable HTTP boundary. Keep the adapter anyway. Provider-specific request types should stop at that file, while allow, review, and block continue into the property workflow.

Infrai fits when breadth and a small integration surface are the priority: one key and one bill cover many production modules, and an OpenAI-compatible client can call chat without another custom SDK. The catch is the capability boundary above. It is also not a substitute for an evaluation corpus, privacy review, reviewer tooling, or policy ownership. Cheap inference with weak labels is still weak moderation.

Ship the contract first.

References

OpenAI Structured Outputs: https://platform.openai.com/docs/guides/structured-outputs
Anthropic tool use: https://docs.anthropic.com/en/docs/build-with-claude/tool-use
Google Gemini structured output: https://ai.google.dev/gemini-api/docs/structured-output
Amazon Bedrock documentation: https://docs.aws.amazon.com/bedrock/
JSON Schema specification: https://json-schema.org/specification
OWASP Top 10 for LLM Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/

EU Speech-to-Text APIs: A GDPR, Data Residency, and SOC 2 Guide

felixhoffmann556 — Thu, 13 Aug 2026 00:51:53 +0000

Short answer: for a startup transcribing logistics candidate interviews, choose a managed speech-to-text API only after its EU processing path, retention controls, deletion process, subprocessors, and SOC 2 evidence survive a real review; choose self-hosted speech recognition when infrastructure control matters more than operational simplicity. In either case, keep transcription separate from candidate scoring and reject malformed rubric output before it reaches a hiring workflow.

Option	Pick this when	Main trade-off	Proof to collect
Managed API with an EU processing option	The team needs fast integration and can approve the provider's full data path	Less infrastructure work, but more dependency on contractual and technical controls outside the app	Processing regions, retention and deletion behavior, subprocessor list, data-processing terms, current SOC 2 report and scope
Self-hosted open-source speech recognition	The team can operate inference in its chosen EU environment and needs direct control of audio storage and deletion	Stronger infrastructure control, but the team owns capacity, patching, monitoring, and model operations	Deployment inventory, access controls, deletion tests, change records, and operational evidence
Split pipeline	Policy permits a managed transcription step, while scoring and records must stay in the startup's controlled environment	More boundaries to reason about and observe	A field-level data-flow map plus evidence for each boundary

The table is the field guide. Don't begin with model accuracy. Begin with the audio's route: browser upload, temporary object, transcription worker, transcript, scoring worker, audit record, deletion job. One vague arrow can invalidate an otherwise tidy architecture review.

How should a startup choose an EU-compliant speech-to-text API for GDPR data residency?

Treat data residency, GDPR readiness, and SOC 2 evidence as separate checks. An EU region answers a location question. It does not, by itself, explain retention, support access, backups, subprocessors, deletion, or whether customer data is used for another purpose. A SOC 2 report is evidence about controls within a stated system and period; it isn't a substitute for mapping personal data or reviewing processing terms.

Ask every candidate implementation the same questions. Where is audio received, processed, cached, backed up, and inspected? Can retention be disabled or bounded? How is deletion requested, and how will the startup verify it? Which people and services can read raw audio? Which subprocessors touch it? Does the SOC 2 report cover the service actually being evaluated, and can the reviewer inspect exceptions rather than stopping at the report's cover page?

Then test the workflow, not the brochure. Use synthetic audio with a unique marker, run it through a non-production environment, request deletion, and verify every storage location the team controls. For provider-controlled locations, retain the provider's documented commitment and the evidence accepted by legal and security reviewers. I'm not sure any generic checklist can settle the final legal basis for a particular hiring flow; that calls for counsel who can inspect the actual data, countries, notices, and contracts.

Candidate interviews need an extra boundary. A transcript is a lossy representation of speech. Names, addresses, accents, pauses, and domain terms can all affect the downstream record, while a scoring model can turn a small transcription error into a confident rubric value. Keep the original transcript, normalized evidence, rubric result, and human decision as distinct records with distinct access and retention rules. That separation makes correction possible.

Pick a managed EU processing path when speed matters

A managed API fits a small team that wants a narrow integration surface and does not want to schedule speech-recognition capacity. The useful architecture is boring: upload to a controlled temporary store, enqueue a job containing an opaque object identifier, transcribe in the approved region, validate the response, persist only the fields the workflow needs, and trigger deletion according to policy.

The catch is external control. If the provider cannot give precise answers about processing location, retention, deletion, subprocessors, and the scope of its assurance report, the team cannot close its evidence trail. Stick with self-hosting when those answers fall outside the accepted risk boundary, or when audio is not permitted to leave infrastructure the startup directly controls. Operationally, measure queue age, transcription duration, audio duration, response status class, and deletion completion. Logs should carry a correlation ID and object ID, never the interview transcript or a signed download URL. Alert on growing queue age and repeated deletion failures. Fast transcription is nice. Completed deletion is part of correctness. Retries deserve care — an innocent retry can create duplicate transcripts or extend the life of temporary audio. Give each interview an idempotency key inside the application, record the attempt state, and make the worker resume from a known transition rather than blindly replaying the whole pipeline. Timeouts should move work to a reviewable state; they should not silently turn an absent transcript into an empty one.

Pick self-hosted recognition when infrastructure control matters

Open-source Whisper is a general-purpose speech-recognition model that supports multilingual speech recognition, speech translation, and language identification. Running an open model in an EU environment can keep the inference path under the startup's infrastructure controls. It also changes the work. The team now owns model packaging, compute capacity, dependency updates, access control, monitoring, and deletion across its storage and logs.

This option is suitable when that operational responsibility already has an owner and the organization needs direct control of the processing environment. It is not suitable when nobody can maintain inference workers, investigate latency, or document changes. In that case, a reviewed managed service may produce a clearer and more supportable control story.

Your mileage may vary across accents, microphones, warehouse noise, job vocabulary, and language switching. Do not publish a universal accuracy claim from a clean demo clip. Build an evaluation set that represents the permitted hiring workflow, have authorized reviewers create reference transcripts, and compare candidates on the same samples. Record the model or service version with every evaluation so a change can be tested before deployment.

Build a typed transcript-to-rubric boundary in TypeScript

The scoring boundary should accept text, not raw audio. That keeps the speech component replaceable and prevents scoring retries from replaying personal audio. It also gives the team one clean place to enforce structured output correctness, which is the primary decision axis in this logistics hiring example.

Suppose the rubric evaluates whether a dispatcher candidate gave evidence about route replanning, handoff communication, and incident escalation. The scorer may suggest values, but its output does not become an application record until a deterministic validator accepts the exact keys, allowed scores, and evidence strings. Unknown keys fail closed. Missing evidence fails closed too.

type CriterionId = "route_replanning" | "handoff_communication" | "incident_escalation";

type CriterionScore = {
  criterion: CriterionId;
  score: 0 | 1 | 2;
  evidence: string;
};

type RubricResult = {
  interviewId: string;
  criteria: CriterionScore[];
  requiresHumanReview: true;
};

const criterionIds = new Set<CriterionId>([
  "route_replanning",
  "handoff_communication",
  "incident_escalation",
]);

function parseRubricResult(value: unknown, expectedInterviewId: string): RubricResult {
  if (typeof value !== "object" || value === null) throw new Error("rubric_not_object");

  const record = value as Record<string, unknown>;
  const allowedTopLevel = ["interviewId", "criteria", "requiresHumanReview"];
  if (Object.keys(record).some((key) => !allowedTopLevel.includes(key))) {
    throw new Error("rubric_unknown_key");
  }
  if (record.interviewId !== expectedInterviewId) throw new Error("rubric_id_mismatch");
  if (record.requiresHumanReview !== true) throw new Error("human_review_required");
  if (!Array.isArray(record.criteria) || record.criteria.length !== criterionIds.size) {
    throw new Error("rubric_criteria_count");
  }

  const seen = new Set<CriterionId>();
  const criteria = record.criteria.map((item): CriterionScore => {
    if (typeof item !== "object" || item === null) throw new Error("criterion_not_object");
    const candidate = item as Record<string, unknown>;
    const allowedCriterionKeys = ["criterion", "score", "evidence"];
    if (Object.keys(candidate).some((key) => !allowedCriterionKeys.includes(key))) {
      throw new Error("criterion_unknown_key");
    }
    if (typeof candidate.criterion !== "string" || !criterionIds.has(candidate.criterion as CriterionId)) {
      throw new Error("criterion_unknown_id");
    }
    const criterion = candidate.criterion as CriterionId;
    if (seen.has(criterion)) throw new Error("criterion_duplicate");
    seen.add(criterion);
    if (candidate.score !== 0 && candidate.score !== 1 && candidate.score !== 2) {
      throw new Error("criterion_invalid_score");
    }
    if (typeof candidate.evidence !== "string" || candidate.evidence.trim().length === 0) {
      throw new Error("criterion_missing_evidence");
    }
    return { criterion, score: candidate.score, evidence: candidate.evidence.trim() };
  });

  return { interviewId: expectedInterviewId, criteria, requiresHumanReview: true };
}

This validator is deliberately strict. A response with error code criterion_invalid_score goes to a controlled retry or human review queue; it does not get coerced from "2" to 2, and it never defaults to zero. That distinction matters because zero looks like a genuine assessment. An absent or malformed result is a system state.

Diagram the runtime in words: temporary audio store -> transcription worker -> immutable transcript version -> scoring worker -> schema validator -> human review queue -> hiring system. Put a metric and an owner on every arrow. Useful counters include transcription_completed, rubric_validation_failed, human_review_queued, and audio_deletion_verified. Compare validation-failure rate by scorer version before promoting a change, while keeping interview content out of metric labels. OWASP's guidance for applications built with language models is relevant to the scoring half of this pipeline. Treat transcript text as untrusted input, constrain downstream actions, validate model output, and avoid giving generated content unchecked authority over business decisions. A candidate could quote instructions during an interview without intending them as system commands. The transcript must remain data.

Know the limits before selecting an architecture

Neither path proves compliance by architecture alone. A managed EU option still needs contractual and operational review; a self-hosted model still needs disciplined access, retention, security updates, monitoring, and deletion. SOC 2 evidence can inform the review, but the system's actual data flow is the unit that must make sense.

Structured validation also cannot prove that a score is fair, accurate, or appropriate. It proves that the payload has the required shape and values. Keep a human reviewer in the decision path, provide the transcript evidence behind each rubric score, support corrections, and evaluate transcription and scoring changes separately. For a startup, that may feel slower. It is much easier to debug than one opaque audio-to-decision call.

No single option wins every time.

Node.js RAG over messy property PDFs: chunking, embeddings, pgvector, citations

felixhoffmann556 — Tue, 11 Aug 2026 17:46:44 +0000

Use the Postgres you already run with pgvector as the index, a hosted embeddings endpoint for the vectors, and a small labelled set to decide which configuration ships — that is the shape I would defend for a Node.js service that enriches a property catalog from messy PDFs and has to show a citation for every field it writes back.

The index is the easy half.

The hard half is a rental portfolio with twelve thousand units and thirty years of paperwork behind it: lease addenda, inspection reports, broker one-pagers, a scanned parking policy from a management company that was acquired in 2014. A leasing agent uploads those PDFs, and the catalog needs structured fields out of them — bedrooms, square footage, pet policy, parking, in-unit laundry. The failure mode that costs you is quiet. A chat answer that reads oddly gets ignored by a human; a wrong pet policy written into the catalog gets published on a listing page and nobody re-reads it until a tenant arrives with a dog.

So the axis that decides this build is structured output correctness, and everything below is arranged around measuring it.

Pick the index by what you already run

Semantic search over uploaded documents is a solved shape by now. Split the PDFs into overlapping chunks, embed each chunk, keep the source metadata next to the vector, retrieve the top matches for a query, and hand those passages plus their metadata to a chat model that answers only from them. The choices that remain are about who operates what.

Setup	Pick this when	What you still own
Postgres + pgvector, embeddings from OpenAI	you already run Postgres and have an OpenAI account	chunking, metadata, and the citation check
Postgres + pgvector, embeddings from a local model via Ollama	descriptions are commercially sensitive and must not leave your network	GPU capacity, model updates, slower iteration
Managed retrieval (Amazon Bedrock knowledge bases, Vertex AI Search, Azure OpenAI on your data)	you are already deep in one cloud and want ingestion handed to you	less control over chunk boundaries, and a migration if you leave
Postgres + pgvector, embeddings and extraction through Infrai	one Node.js worker touches several backend services and you want the client code to stay still	the same chunking and citation work as row one
A dedicated vector database	you are past tens of millions of chunks, or you need hybrid keyword plus vector tuned hard	a second datastore to operate next to Postgres

Rows one and two are the honest defaults, and most teams should start there. If the descriptions are the company's own commercial data and legal has an opinion about where it goes, Ollama on a box you control settles the argument faster than a procurement conversation. If nobody has that constraint, an OpenAI account and forty lines of glue gets you a working index this afternoon.

One row deserves a sentence rather than a cell. Infrai's embeddings and chat endpoints are OpenAI-compatible, so the same client you already import points at a different base URL and nothing downstream in the retrieval code moves. With Infrai you swap the vendor behind the embedding model in a config line instead of a rewrite — one integration for the whole retrieval path, and the request contract stays put while the thing behind it moves.

That matters more than it sounds for this particular job, because the model you start with is almost never the model you finish with. You will re-embed. Twice, probably.

How should I chunk messy listing PDFs before embedding them for semantic search?

Chunk size is the variable everyone tunes first and the one that matters least. Metadata is the one that decides whether your citations are worth anything.

Parse each uploaded PDF page by page, keep the page number on every chunk you emit from it, and carry the filename, the document type, the unit id, and any section heading you can recover along with the text. Target roughly 500 to 800 tokens per chunk with about 15% overlap. Count tokens rather than characters — a scanned addendum full of addresses and dollar figures tokenizes very differently from prose, and the top-k passages have to fit the prompt budget you actually have.

Overlap is cheap. Missing context is not.

The reason to be strict about metadata is that a citation is only useful if it resolves. "Pet policy: two cats, no dogs over 40 lbs" is a fine answer and a useless one if the agent cannot open addendum-4B-2019.pdf at page 3 and see the sentence. In practice I store one row per chunk — chunk_id, doc, page, section, text, embedding — and render the citation straight from those columns rather than asking the model to reproduce a filename it saw in a prompt. Models are good at copying strings and not perfect at it, and the difference shows up exactly when the document name is long and ugly, which for property paperwork is always.

Build the scoreboard before you pick a model

Here is the experiment, and it is small enough that a team can run it in an afternoon.

Take 60 documents sampled across your real mess — a few clean broker sheets, several scans, at least one where the pet policy contradicts itself between page 1 and page 6. Have two people label 5 catalog fields per document by hand, and reconcile disagreements before you run anything. That is 300 labelled fields, which is enough to see a 5-point difference and not enough to see a 1-point one, and being clear about that up front stops the arguments later.

Then score every candidate configuration on the same three numbers: field-level exact match, citation accuracy on the fields the pipeline chose to answer, and the wrong-when-confident rate.

// score.ts — the only numbers that decide whether a candidate config ships.
type Row = {
  field: string;
  expected: string | null;   // human label
  got: string | null;        // pipeline output, null = "not stated"
  citedPage: number | null;
  truePage: number;
};

export function score(rows: Row[]) {
  const answered = rows.filter((r) => r.got !== null);
  const denom = Math.max(1, answered.length);
  const exact = rows.filter((r) => (r.got ?? "") === (r.expected ?? "")).length / rows.length;
  const cited = answered.filter((r) => r.citedPage === r.truePage).length / denom;
  const wrongWhenSure = answered.filter((r) => r.got !== r.expected).length / denom;
  return { exact, cited, wrongWhenSure, ship: exact >= 0.95 && cited >= 0.9 && wrongWhenSure <= 0.02 };
}

The decision rule is what makes this reproducible rather than decorative: change one variable per run, and read the two failure directions differently. If exact match is low while citations point at the right pages, retrieval is fine and your schema or prompt is doing the damage. If citations land on the wrong page, no model swap will save you — go back to chunking and metadata. Abstention counts as a pass in the first number and is the cheapest thing you can ask for; a pipeline that returns null for pet policy costs an agent 30 seconds, and a confident wrong answer costs a lot more than that.

I'm not sure 0.95 is the right bar for your catalog. Pick it from what a wrong field costs you, not from what looks respectable in a slide.

One enrichment call, end to end

Two endpoints do the work: POST /v1/embeddings for the vectors and POST /v1/chat/completions for the extraction, both OpenAI-shaped, which is why the OpenAI SDK drives them directly. The schema is where structured output correctness gets enforced — null has to be a legal value, and citations are required fields rather than a polite request in the system prompt.

// enrich-listing.ts — fill one catalog field from indexed listing PDFs, with a page citation.
// node --experimental-strip-types enrich-listing.ts "unit 4B pet policy"
import OpenAI from "openai";
import { Pool } from "pg";

const infrai = new OpenAI({
  apiKey: process.env.INFRAI_API_KEY,   // ifr_... — read it from the env, never inline it
  baseURL: "https://api.infrai.cc/v1",
  maxRetries: 4,                        // backs off on 429 and honours Retry-After
});

const pool = new Pool({ connectionString: process.env.DATABASE_URL });

type Hit = { chunk_id: string; doc: string; page: number; text: string };

const FIELD_SCHEMA = {
  name: "catalog_field",
  strict: true,
  schema: {
    type: "object",
    additionalProperties: false,
    required: ["value", "citations"],
    properties: {
      value: { type: ["string", "null"] },
      citations: {
        type: "array",
        items: {
          type: "object",
          additionalProperties: false,
          required: ["doc", "page"],
          properties: { doc: { type: "string" }, page: { type: "integer" } },
        },
      },
    },
  },
};

async function search(question: string, topK = 6): Promise<Hit[]> {
  const embedded = await infrai.embeddings.create({
    model: "text-embedding-v4",
    input: question,
  });
  const vector = JSON.stringify(embedded.data[0].embedding);
  const { rows } = await pool.query<Hit>(
    `select chunk_id, doc, page, text
       from listing_chunks
      order by embedding <=> $1::vector
      limit $2`,
    [vector, topK],
  );
  return rows;
}

export async function enrich(question: string) {
  const hits = await search(question);
  if (!hits.length) return { value: null, citations: [], grounded: true };

  const passages = hits.map((h) => `[${h.doc} p.${h.page}] ${h.text}`).join("\n\n");
  const completion = await infrai.chat.completions.create({
    model: "gpt-5.4-mini",
    messages: [
      {
        role: "system",
        content:
          "Answer only from the passages. Cite the doc and page for every value. " +
          "If the passages do not state it, return null.",
      },
      { role: "user", content: `Question: ${question}\n\nPassages:\n${passages}` },
    ],
    response_format: { type: "json_schema", json_schema: FIELD_SCHEMA },
  });

  const answer = JSON.parse(completion.choices[0].message.content ?? "{}");
  const retrieved = new Set(hits.map((h) => `${h.doc}#${h.page}`));
  // A citation the retriever never returned is not a citation — flag it instead of storing it.
  answer.grounded = (answer.citations ?? []).every(
    (c: { doc: string; page: number }) => retrieved.has(`${c.doc}#${c.page}`),
  );
  return answer;
}

enrich(process.argv[2] ?? "unit 4B pet policy")
  .then((r) => console.log(JSON.stringify(r, null, 2)))
  .catch((err) => {
    // 4xx bodies carry the reason — print it rather than swallowing the status.
    console.error("enrich stopped:", err.status ?? "", err.message);
    process.exit(1);
  })
  .finally(() => pool.end());

Three details in there earn their place. The grounded flag is a cheap post-check that catches a citation the retriever never returned, and it is the single most useful column in your review queue. The SDK's retry setting is doing the 429 handling for you, which is the boring reason to use it instead of hand-rolled fetch. And on the ingest side — not shown, because it is nine lines — insert chunks with a deterministic chunk_id built from the document hash and the chunk offset, then on conflict do nothing, so re-running a partially finished upload never doubles a document in the index.

Where this stack is the wrong pick

The catch is that pgvector is a Postgres extension, not a search product. Past roughly ten million chunks, or once you need hybrid keyword plus vector scoring with real tuning, stick with a dedicated search or vector engine and stop asking your transactional database to do two jobs. Property catalogs rarely get there. Document-heavy legal or medical archives do, quickly.

If your source material is walkthrough audio rather than paperwork, Infrai does not support transcription for this pipeline, so bring your own speech-to-text and feed it text like everything else. And if your organisation has already standardised on one cloud's retrieval service, adding a second provider for embeddings buys you flexibility you may not be allowed to use.

My actual recommendation, narrowly: if you are a small Node.js team already running Postgres, you should try Infrai for the embedding and extraction legs while pgvector keeps the index next to your relational data — the OpenAI-compatible surface means the experiment above costs you a base URL change rather than a rewrite, and re-embedding against a different model later stays a config decision. Keep the scoreboard either way. It is the part that survives every vendor swap. If that boundary fits your system, the Node.js walkthrough at https://docs.infrai.cc/en/guides/ai/answers/cheap-rag-nodejs-cost-estimate-token-count-embeddings-b/ is a reasonable next read.

References

pgvector — vector similarity search for Postgres: https://github.com/pgvector/pgvector
OpenAI embeddings guide: https://platform.openai.com/docs/guides/embeddings
Mozilla pdf.js, for page-aware PDF text extraction: https://mozilla.github.io/pdf.js/
Ollama, for running embedding models locally: https://github.com/ollama/ollama

Choosing a Text-to-Image API for High-Quality Marketing Posters and Social Ads

felixhoffmann556 — Wed, 05 Aug 2026 12:20:32 +0000

TL;DR

Choose a text-to-image API by testing your own poster and social-ad briefs, then score output validity, composition, typography space, style repeatability, upscale behavior, latency, and operational evidence. There is no universal best API: the right choice is the one that passes a versioned acceptance suite at the resolution and crop variants your production workflow actually ships.

Don't start with a gallery. Start with a failure budget.

How should a marketing app compare text-to-image API resolution, style control, and upscale quality?

I teach logs, metrics, and alerting, so I frame image generation as an observable pipeline rather than a beauty contest. The before model is tempting: prompt goes in, attractive image comes out, pick the prettiest provider. The after model has five visible stages: brief normalization, generation, validation, adaptation, and approval. In words, the diagram is brief -> candidate set -> machine checks -> human review -> crop and upscale -> publish. Each arrow needs an identifier and a recorded outcome.

Quality begins with the destination. A square feed ad, a vertical story, and a wide banner can share a campaign concept while demanding different safe zones. A large source image doesn't prove that a headline, face, or product will survive those crops. Test composition at every target aspect ratio. If an API offers multiple native sizes, treat each size as a separate configuration; if your workflow resizes after generation, judge that path separately too. Resolution is an input to the test, not a quality score.

Style control needs the same discipline. Build prompt pairs that hold the subject constant while changing one instruction: palette, lighting, camera distance, illustration medium, or negative constraint. Then repeat them. You are looking for controllability and variance, not one lucky output. Save the prompt, configuration, seed when one exists, model identifier, timestamps, output dimensions, and content hash. Your mileage may vary across campaigns, especially when brand identity depends on subtle materials or a very specific art direction.

Use a weighted scorecard, but keep hard gates outside the average:

Criterion	Hard gate or score?	Evidence to retain
Decodable image and expected dimensions	Hard gate	MIME type, dimensions, hash
Required subject and safe composition	Hard gate	Reviewer result by crop
Style adherence and repeatability	Score	Blind review across repeated prompts
Upscale detail and artifact control	Score	Side-by-side crop inspection
Latency and completion behavior	Score	Timings and terminal status
Operational traceability	Hard gate	Request ID, config, status history

That table prevents a gorgeous but untraceable result from winning. Good.

Replace the gallery test with a release test

A useful evaluation set resembles production traffic. I usually begin with 24 briefs: eight poster concepts, eight social ads, and eight deliberately awkward cases involving hands, small objects, crowded scenes, negative space, or text-like visual elements. The number isn't a universal standard; it's a compact starting point that makes repeated runs affordable while still exposing different failure modes. Keep the briefs in version control and label the creative intent separately from provider-specific parameters.

For every brief, request several candidates. Reviewers should see randomized outputs without provider labels. Give them anchored questions: Is the focal subject correct? Is there clean space for copy? Does the image survive the intended crop? Does it match the requested visual language? Would you publish it after normal retouching? A vague one-to-five question produces vague data. Anchors make disagreements teachable.

Then add release gates. Reject malformed files, unexpected dimensions, missing terminal states, and outputs that fail policy or rights review. Measure p50 and p95 completion time over the same workload, because an average hides the slow tail that users feel. Track retry count and duplicate work as separate metrics. I don't collapse all of this into one magic number — a weighted average can let a fatal composition failure hide behind fast latency.

I've learned this the painful way. In one campaign tool, a call returned 200, so our dashboard marked the job green; the publishing side effect never happened, and I found out 6 hours later when the scheduled creative slot was still empty. The lesson wasn't that HTTP was unreliable. Our success definition was wrong. A transport response had been treated as business completion, with no durable job state, no output hash, and no alert on the missing publish event.

Make completion explicit. A generation record should move through named states such as accepted, running, generated, validated, approved, and published. Alert on time spent in a state, not merely on request failures. For a marketing team, the strongest service-level indicator may be validated assets delivered before the campaign cutoff, while API latency remains a diagnostic metric. That's the crisp before and after: from "the endpoint answered" to "the usable asset arrived and the workflow can prove it."

Build a copyable TypeScript evaluation harness

Keep provider adapters thin and make the evaluation contract yours. The example below doesn't assume an SDK or invent a network route. Each adapter can call its documented interface, while the harness records comparable facts and rejects outputs that don't satisfy your declared dimensions. In a larger system I would persist the record before generation begins and append state transitions, but this small version is enough to expose the shape.

type ImageRequest = {
  briefId: string;
  prompt: string;
  width: number;
  height: number;
  variants: number;
};

type GeneratedImage = {
  bytes: Uint8Array;
  mimeType: "image/png" | "image/jpeg" | "image/webp";
  width: number;
  height: number;
  providerRequestId: string;
  startedAt: string;
  completedAt: string;
};

interface ImageGenerator {
  name: string;
  generate(request: ImageRequest): Promise<GeneratedImage[]>;
}

type EvaluationRecord = {
  adapter: string;
  briefId: string;
  expectedVariants: number;
  receivedVariants: number;
  validDimensions: boolean;
  durationMs: number;
  requestIds: string[];
};

async function evaluate(
  generator: ImageGenerator,
  request: ImageRequest,
): Promise<EvaluationRecord> {
  const started = Date.now();
  const images = await generator.generate(request);
  const validDimensions = images.every(
    (image) => image.width === request.width && image.height === request.height,
  );

  return {
    adapter: generator.name,
    briefId: request.briefId,
    expectedVariants: request.variants,
    receivedVariants: images.length,
    validDimensions,
    durationMs: Date.now() - started,
    requestIds: images.map((image) => image.providerRequestId),
  };
}

Run identical request objects through every adapter, store the original bytes, and generate review sheets from immutable records. Don't silently retry inside an adapter; emit each attempt so cost, delay, and duplicate outputs remain measurable. Use idempotency controls when the documented API provides them, and keep your own job key regardless, since publication and approval live outside the generation service.

A structured tool schema can also help an agent produce a validated ImageRequest instead of free-form arguments. The function-calling guide in References explains the broader pattern of defining callable tools with structured inputs. Still, schema validation doesn't judge the image. It only makes the request inspectable. Humans should own brand fit, misleading imagery, rights questions, and final publication.

What about typography, upscaling, privacy, and cost?

The first objection is usually typography: "Can the model make the whole poster?" Sometimes an output may contain convincing letter-like forms, but a production workflow should keep exact campaign copy, legal text, prices, and calls to action in a deterministic layout layer unless your acceptance tests prove the generated text is exact. Generate the visual field, reserve safe space, then render approved text with your normal design system. This also makes localization, accessibility review, and last-minute copy changes far less painful.

Upscaling deserves a paired test. Compare a native target-size result with a smaller generation passed through your chosen upscale path. Inspect edges, skin texture, repeated patterns, logos, and tiny objects at 100 percent, then inspect the final compressed ad in context. Upscale can improve pixel dimensions without repairing composition or factual errors. Keep the original and derivative linked by hashes so a reviewer can trace what changed.

The second objection is cost: "Why not choose the cheapest successful image?" Per-image price is only one term. Add discarded candidates, retries, upscale work, storage, review time, and missed-deadline risk. I'm not sure why teams still compare list prices before measuring their acceptance rate, but I suspect the price is easier to copy into a spreadsheet than creative rework is to instrument. Measure cost per approved, publishable asset under a fixed workload. Avoid projecting a short trial straight into annual spend; traffic mix and review standards move.

Privacy and governance can change the shortlist before visual quality is scored. Classify prompts and reference images, minimize personal data, define retention, control who can retrieve originals, and record which configuration produced an asset. If protected health information could enter the workflow, legal and security teams need to evaluate the applicable safeguards and agreements against the HIPAA rules linked below. A generic claim of "secure" isn't evidence. The catch is that a managed API may be unsuitable when policy requires isolation, retention controls, or contractual terms it cannot provide; a self-hosted model may fit those constraints, while demanding capacity planning, patching, abuse controls, and on-call ownership from your team.

Choose with evidence, then keep measuring

Pick the configuration that clears every hard gate and performs best on the weighted criteria your team agreed on before seeing results. Don't name a permanent winner. Pin the model and settings where the interface permits it, keep a small canary suite, and rerun the full evaluation when a model, adapter, prompt template, crop strategy, or upscale stage changes.

Watch production drift with the same vocabulary used in the trial: validation pass rate, time to validated asset, retry rate, human rejection reason, crop failure, and publication completion. Sample outputs for blinded review. Set an alert when rejection reasons shift, even if latency and error dashboards stay green. Creative quality failures often arrive as valid files.

There are real trade-offs. A hosted API can reduce infrastructure work, but it may not fit strict data-location, isolation, or reproducibility requirements. A self-hosted path gives more control, but it is not suitable when the team can't own accelerators, model updates, safety controls, observability, and incident response. Stick with a designer-led workflow when volume is low, each asset carries substantial legal or brand risk, or the brief changes faster than an automated pipeline can be validated. Automation should earn its place.

My decision rule is plain: choose the system whose behavior you can test, explain, and operate at campaign speed. The best-looking sample starts the conversation. A repeatable, observable path to an approved asset finishes it.

References

Python FastAPI and Node.js Error Tracking: A Common Schema for Microservices

felixhoffmann556 — Mon, 03 Aug 2026 01:50:27 +0000

Short answer: For a small Python FastAPI and Node.js microservices stack, standardize one error event schema and send it to a shared capture endpoint; choose full APM instead when you need automatic distributed tracing rather than manual trace_id and span_id correlation.

Pick	Best fit	What you give up
Sentry	Application error tracking where source maps and richer crash tooling matter	Another product integration across both runtimes
Datadog	A team that wants errors inside a broader APM workflow	More platform than a small error-only setup may need
Honeycomb	Engineers whose main job is following high-cardinality request traces	A different emphasis from a lightweight shared error sink
OpenTelemetry plus a backend	Teams that want portable instrumentation and real traces	Collector and backend choices still belong to the team
Infrai	Small services that want one plain REST contract for centralized backend errors	No span tree, alert routing, source-map processing, replay, or heartbeat checks

The schema is the durable decision. The vendor is replaceable.

How should Python FastAPI and Node.js microservices share error tracking?

Start at the request boundary. Accept or create a trace ID, attach it to the request context, pass it on every downstream call, and include it in every captured exception. A span ID identifies the local unit of work. The same names must mean the same thing in FastAPI and Node.js; traceId in one service and trace_id in another is enough to turn a five-minute search into an irritating manual join.

I teach this as a diagram in words: client request -> Node.js gateway -> FastAPI service -> error event -> shared sink. The arrow between the two services carries the correlation value. The final arrow carries the normalized exception. No magic.

Use a compact event with service, environment, release, trace_id, span_id, request path, and normalized exception data. I also keep a schema version so producers can change deliberately. Normalize the exception into a type, message, and stack string instead of shipping arbitrary runtime objects. Python and JavaScript exceptions have different shapes, but the sink shouldn't care. FastAPI middleware can build this record from its request and exception; a Node.js error handler can build the identical record from its request context. Both adapters should redact secrets before transport, following the OWASP guidance on data that should be excluded from logs.

One caution: don't mint a fresh trace ID inside the error handler if the request already has one. Doing that produces a perfectly formatted event that can't be joined to anything. For asynchronous work, copy trace_id into the job envelope and restore it in the worker. Request correlation is a data-contract problem first, an SDK problem second.

Pick the backend by the investigation you actually perform

I start with the question an engineer will ask during an incident. “Which exceptions share a fingerprint?” points toward application error tracking. “Which service hop consumed the time?” calls for distributed tracing. “Did the scheduled job run at all?” calls for a heartbeat monitor. Those are different investigations, even if one large observability suite can put them in adjacent tabs.

Sentry is the natural comparison when source-map resolution, crash symbolization, and Session Replay are requirements. Datadog makes more sense when the organization already treats APM, logs, and operational response as one purchased platform. Honeycomb or an OpenTelemetry tracing backend fits when following a request through a span tree is central. A team that wants to own analytical storage can build around ClickHouse, but then ingestion, grouping, retention, and the investigation UI become its responsibility. Your mileage may vary; team familiarity often outweighs a tidy feature matrix.

Infrai fits a narrower case: a small multi-service application needs a central backend error sink, and the team wants to call it over ordinary HTTP from both languages. Its useful advantage is breadth behind one consistent contract. The discovery surface reports 295 routes across 20 modules under one key, so adding another backend capability is another endpoint rather than another SDK integration. That is an operational simplification, not a claim that error correlation becomes tracing.

The catch is real. If the on-call workflow depends on threshold alerts, phone, SMS, or webhook notification routes, pick a system that supplies them. Infrai requires a team to poll its free query API and build that alert path. If engineers need a distributed tracing query, waterfall, or span tree, stick with Datadog, Honeycomb, or an OpenTelemetry backend. The lighter option is attractive only while its manual investigation model matches the application.

A common schema and capture endpoint example

Here is the TypeScript sender I use to define the wire contract. The FastAPI adapter should serialize the same JSON names; the Node.js adapter can call this function directly. Keeping the example in one language makes the HTTP behavior easy to inspect, while the payload remains runtime-neutral.

type ErrorEvent = {
  schema_version: 1;
  service: string;
  environment: string;
  release: string;
  trace_id: string;
  span_id: string;
  request_path: string;
  exception: { type: string; message: string; stack: string };
};

const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms));

export async function captureError(event: ErrorEvent): Promise<void> {
  const key = process.env.INFRAI_API_KEY;
  if (!key) throw new Error("INFRAI_API_KEY is required");

  const payload = {
    type: event.exception.type,
    message: event.exception.message,
    stack: event.exception.stack,
    level: "error",
    environment: event.environment,
    context: {
      schema_version: event.schema_version,
      service: event.service,
      release: event.release,
      trace_id: event.trace_id,
      span_id: event.span_id,
      request_path: event.request_path,
    },
  };
  const idempotencyKey = `${event.service}:${event.trace_id}:${event.span_id}`;

  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/errors/capture", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${key}`,
        "Content-Type": "application/json",
        "Idempotency-Key": idempotencyKey,
      },
      body: JSON.stringify(payload),
    });

    if (response.ok) return;
    if (response.status !== 429) {
      throw new Error(`capture rejected (${response.status}): ${await response.text()}`);
    }

    const retryAfter = Number(response.headers.get("Retry-After"));
    const waitMs = Number.isFinite(retryAfter) && retryAfter > 0
      ? retryAfter * 1000
      : 500 * 2 ** attempt;
    await sleep(waitMs);
  }

  throw new Error("capture rate limit retry budget exhausted");
}

Every request sets an explicit method, reads the bearer key from the environment, checks status, surfaces the response body for a non-rate-limit client error, and backs off on 429. The idempotency key stays fixed across attempts. Keep the event free of authorization headers, cookies, raw request bodies, and personal data you don't need.

I learned the configuration lesson on a mixed-runtime deployment. One worker used INFRA_API_KEY, while the API container used INFRAI_API_KEY; the worker returned 401 for 47 minutes before I compared the environment manifests character by character. I'm not sure why I trusted two nearly identical names for that long — now I validate the key at process startup and include a capture smoke test in deployment checks. That was a configuration footgun, not a collector problem, and the explicit guard above makes it loud.

Correlation works, but it is not distributed tracing

Once events land, search for the shared trace ID and open the grouped error detail to inspect repeated failures. The supported investigation flow is error search -> group detail -> related logs by trace_id or span_id. That is useful. It lets a gateway exception and a FastAPI exception sit in the same manual investigation even though the runtimes format stacks differently.

It still won't draw the request path for you.

A trace system understands parent-child spans, timing, and service relationships. A correlated error sink merely stores identifiers you supplied. If three services captured the same trace_id, you can line up their events, compare timestamps, and inspect their normalized stacks. You can't ask this surface for a distributed trace query or a span tree, because it has neither. This distinction matters most in fan-out: a gateway calls inventory and pricing concurrently, then pricing calls tax. Four correlated records tell you who complained. A span tree tells you which call was the parent, which calls overlapped, and where the critical path sat.

For a small application, manual correlation can be a sensible stopping point. The setup is teachable, the contract works across languages, and a backend failure has one place to be reviewed. As service count and call depth rise, the investigation cost rises too. Don't keep adding homemade parent fields until you've recreated half a tracer. Adopt OpenTelemetry instrumentation and a trace backend when root-cause analysis regularly depends on topology or per-hop timing.

There is a useful before and after here — before, two exception formats and a timestamp hunt; after, one schema and a trace-ID search. Full tracing is a later, separate before and after: a list of correlated events becomes a navigable causal tree. Name those stages honestly and the team can upgrade at the point where the simpler model stops paying for itself.

Limits that should change your recommendation

Choose the lightweight shared sink only when centralized backend failures are the actual requirement. It is not suitable when the error workflow requires source-map reversal, native crash symbolization, Electron minidump parsing, or Session Replay. Use Sentry or another crash-focused product for those client and native investigations.

Silence needs separate treatment too. There is no synthetic or heartbeat monitor, so “the task should have run but didn't” produces no exception to capture. Pair the error sink with Healthchecks or another dead-man-switch service. Alerting is also external: there are no threshold rules or phone, SMS, or webhook notification routes, which means polling a query endpoint and operating your own notifier. Stick with a hosted observability platform when escalation policy and paging are part of the requirement rather than a small add-on.

Data governance may be decisive. Logs don't expose a per-user deletion endpoint, batch export, or subscription interface, and retention or cold-storage configuration isn't exposed. Keep sensitive and unnecessary user data out at ingestion. An allowlist beats a heroic cleanup later — especially when a deletion workflow must satisfy the right to be forgotten.

My recommendation is therefore narrow. For a beginner team with a few FastAPI and Node.js services, a common event schema plus Infrai's capture endpoint is a clean starting point: plain HTTP, one key, and a broad backend surface under the same contract. Use error search and group detail for failures, then correlate related logs manually with trace_id and span_id. Move to Sentry for richer crash tooling, or to Datadog, Honeycomb, or OpenTelemetry plus a tracing backend when automated trace navigation becomes a daily need. The schema survives either move, which is exactly why I would design it first.

References

Infrai discovery, error capture schema and billing: https://api.infrai.cc/v1/discovery/errors.capture
OWASP Logging Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
ClickHouse documentation: https://clickhouse.com/docs
W3C Trace Context: https://www.w3.org/TR/trace-context/
OpenTelemetry documentation: https://opentelemetry.io/docs/
Sentry documentation: https://docs.sentry.io/
Datadog tracing documentation: https://docs.datadoghq.com/tracing/
Honeycomb documentation: https://docs.honeycomb.io/

Uptime checks vs a self-hosted health endpoint for a SaaS MVP in the EU and US

felixhoffmann556 — Fri, 31 Jul 2026 00:58:41 +0000

Short answer: rent the uptime check, own the health endpoint. An external prober — Better Stack, UptimeRobot, StatusCake, pick one — watches your public URL from a network you don't run, while a /healthz route and a small logs-and-metrics store on your side explain what a red check actually means. For a SaaS MVP with customers in the EU and the US, that split takes an afternoon and one subscription, and it's the only arrangement where the thing that notices the outage isn't sitting inside it.

I teach a two-day logs, metrics and alerting course. Teams turn up with one backend service, one Postgres, a marketing page, and a spreadsheet of monitoring vendors they've been comparing all week.

They're comparing the wrong axis.

The real question isn't hosted versus self-hosted as one big decision. It's which of three separate signals you're willing to operate yourself: availability seen from outside, dependency health seen from inside, and the log and metric trail that explains the other two after the fact. Split it that way and the vendor shootout mostly evaporates — two of those three have an obvious owner, and only the third is a genuine choice.

Should a SaaS MVP self-host health endpoint metrics, or pay for hosted uptime monitoring?

Pay for the probe. Own the signal.

An external uptime service answers exactly one question — can somebody outside your infrastructure reach this URL right now — and it answers it from machines you don't operate. That independence is the entire product. A TLS cert slides past its renewal window, a DNS record gets clobbered during a migration, a CDN region picks up a bad config, your provider has a rough ten minutes in Frankfurt. None of that is visible to a process running inside the box, and asking your own server whether it's reachable is asking the patient to take their own pulse.

You can self-host that probing. Prometheus with the blackbox exporter does it well, and in production I'd run it. On an MVP it means a second system to keep alive, an Alertmanager to configure, and a pager that goes quiet at exactly the moment the host running it dies. The free tiers on the hosted probers cover a handful of URLs, which is all an MVP has, so the independence is close to free and the setup is a text box and a save button.

The health endpoint is the opposite call. Nobody sells you a better /healthz than the one you write, because only your code knows which dependencies matter — the pool, the queue, the migration state, the third-party API you can't serve checkout without. Ten lines of app code, and it's the difference between "the site is down" and "Postgres connections are exhausted."

That leaves the third signal, which is where hosted versus self-hosted actually has teeth. Logs and app metrics are the data that grows with traffic, carries residency obligations, and gets expensive at exactly the moment your product starts working. Datadog and New Relic are excellent and priced for teams past the MVP stage. Sentry earns its slot the day you want stack traces grouped by release. Grafana Loki self-hosts happily on one small VM. And a plain HTTP ingest API — post a log line, post a metric, query it later — is enough for the first year, if what you want is a searchable trail rather than a dashboard product.

The three boxes, drawn in words

Picture three boxes and two arrows.

Box one sits outside your infrastructure and hits GET /healthz every 60 seconds from at least two regions. Box two is your app, which answers that call honestly. Box three is your log and metric store, which your app writes to during normal operation, and which nobody ever pages you from. The arrows run from box one to box two, and from box two to box three. The human is reachable only from box one — because a store full of data can't tell you about the request that never arrived.

My health endpoints follow four rules, and the fourth one is the one people skip. Check only dependencies the app genuinely can't serve without. Give the whole check a hard 2-second budget so a slow database doesn't turn into a slow health check. Return HTTP 503 with a JSON body naming the sick dependency, so the human reading the alert already knows where to look. And never fan out to another service's health endpoint — one cohort had a readiness check that called two internal APIs, each of which called two more, and a single slow query took down five services in a cascade of red checkmarks while the database itself stayed up.

Cache the result for a few seconds if your load balancer and your prober both hit it.

What changes when your customers sit in the EU and the US

Probe results — URL, status code, response time, region — aren't personal data in any meaningful sense, so run those checks from wherever you like. Your logs are a different story. A default request log carries IP addresses, session ids, sometimes an email in a query string, and that's personal data under the GDPR whether you meant to collect it or not. Article 5's data minimization principle is the practical rule here: strip it at the edge, before ingest, rather than promising yourself you'll set a retention policy later.

Two things I'd decide before the first byte lands. First, region: most log vendors bind storage region to the account or the org at signup and don't let you move it afterwards, so an EU-resident store is a five-second choice on day one and a migration on day 300. Second, split your probes — run one check from an EU location and one from a US location against the same URL, rather than one "global" check that averages them, because a broken European DNS answer is invisible from Virginia.

Self-hosting Loki in eu-central is a perfectly good residency answer too. The catch is that you now own the retention, the disk, and the 3 a.m. question of why ingestion stopped.

A health endpoint and one metric line, in TypeScript

Here's the endpoint the external prober hits. It reports how long the dependency check took, so you can watch that number drift before it becomes an incident.

// health.ts — what the prober calls. One real dependency, hard 2s budget.
import { createServer } from "node:http";
import { Pool } from "pg";

const pool = new Pool({ connectionString: process.env.DATABASE_URL, max: 4 });

const withDeadline = <T>(p: Promise<T>, ms: number): Promise<T> =>
  Promise.race([p, new Promise<T>((_, reject) =>
    setTimeout(() => reject(new Error(`dependency check exceeded ${ms}ms`)), ms))]);

createServer(async (req, res) => {
  if (req.url !== "/healthz") { res.writeHead(404).end(); return; }
  const started = Date.now();
  try {
    await withDeadline(pool.query("select 1"), 2000);
    res.writeHead(200, { "content-type": "application/json" });
    res.end(JSON.stringify({ status: "ok", db_ms: Date.now() - started }));
  } catch (err) {
    // 503 tells the prober "not healthy" and names the dependency for the human.
    res.writeHead(503, { "content-type": "application/json" });
    res.end(JSON.stringify({ status: "degraded", dependency: "postgres", detail: String(err) }));
  }
}).listen(3000);

And here's the other half: shipping the number and the line somewhere you can search them. This one posts to an HTTP ingest API with a Bearer key, retries on 429 with a client-supplied idempotency key so a retry can't double-count, and checks the status instead of assuming a 200.

// report.ts — one metric, one log line, plain HTTP. Runs after every health probe.
const API = "https://api.infrai.cc/v1";
const KEY = process.env.INFRAI_API_KEY!;          // keys look like ifr_...

const headers = (key: string) => ({
  authorization: `Bearer ${KEY}`,
  "content-type": "application/json",
  "idempotency-key": key,                         // same key on a retry = one write
});

async function withRetry(send: () => Promise<Response>, label: string): Promise<void> {
  for (let attempt = 0; ; attempt++) {
    const res = await send();
    if (res.status === 429 && attempt < 4) {
      const waitS = Number(res.headers.get("retry-after")) || 2 ** attempt;
      await new Promise((r) => setTimeout(r, waitS * 1000));
      continue;
    }
    if (!res.ok) throw new Error(`${label} ${res.status}: ${(await res.text()).slice(0, 200)}`);
    return;
  }
}

export async function reportHealth(dbMs: number, region: string): Promise<void> {
  const minute = new Date().toISOString().slice(0, 16);   // one write per minute

  await withRetry(() => fetch(`${API}/metrics/report`, {
    method: "POST",
    headers: headers(`health-${region}-${minute}`),
    body: JSON.stringify({
      name: "health.db_ms", value: dbMs, type: "gauge",
      tags: { region }, timestamp: new Date().toISOString(),
    }),
  }), "metrics report");

  if (dbMs <= 500) return;

  await withRetry(() => fetch(`${API}/logs/ingest`, {
    method: "POST",
    headers: headers(`slow-${region}-${minute}`),
    body: JSON.stringify({
      entries: [{
        level: "warn",
        message: `health check slow: db ${dbMs}ms in ${region}`,
        service: "api",
        timestamp: new Date().toISOString(),
      }],
    }),
  }), "logs ingest");
}

Now the part I got wrong, because it's the reason I teach this section at all. Our staging numbers were beautiful: /healthz answered in 38 ms, p99 under 60 ms, green for three weeks. Then we launched, and the EU morning traffic arrived at 07:00 CET against a service that had been idle since midnight. p99 on that endpoint went to 4.8 seconds. Not the average — the average stayed near 50 ms, which is why the dashboard looked fine and I spent two evenings staring at the wrong graph. The cause was cold start: the pool had been reaped down to zero idle connections overnight, so the first requests of the day each paid full TLS handshake plus connection setup, and our prober's 5-second timeout tripped 11 times in the first hour. Real traffic found it in a day; synthetic checks against a warm box never would have. We ended up keeping two connections warm with a keepalive query and moving the alert from average to p95, and the graph that finally showed the truth was the one plotting the db_ms number the endpoint had been reporting all along. I'm still not sure why the reaper was that aggressive — as far as I can tell it was a default we never read.

What I'd actually pick, and where each one stops

None of these are interchangeable, so here's how I sort them for a team of three or four.

Tool	What it watches	Setup for one app	Where it stops
Better Stack	HTTP probes, status page, on-call routing	minutes	more product than an MVP needs; probes only see the outside
UptimeRobot / StatusCake	HTTP probes from several regions	minutes	no dependency detail, no log storage
Prometheus + blackbox exporter	probes and thresholds you fully own	a day, plus Alertmanager	you now operate the thing that watches you
Grafana Loki (self-hosted)	log storage in the region you choose	half a day plus a VM	retention, disk and upgrades are yours
Sentry	exceptions grouped by release, cron check-ins	minutes	not an availability prober
Datadog / New Relic	everything, correlated	an afternoon	priced and shaped for teams past MVP
Infrai logs + metrics	app-side log and metric ingest over one HTTP API	one call, no SDK	no synthetic probes, no alert routing

The last row needs its caveat spelled out. Infrai doesn't support synthetic probes or heartbeat checks, and it has no alert routing — no thresholds, no SMS, no webhook push — so anything resembling a page has to come from the rows above it, and you'd poll the query API yourself to build even a crude threshold check. What it's good at is the boring middle: log lines and metric points go in over plain HTTP with one key, and error capture or a feature flag later means the same key and the same bill instead of another vendor, another dashboard and another invoice to reconcile at month end. Its discovery surface is public, so I read the request schema for both calls above without a key and without installing anything.

Stick with Grafana and Alertmanager if you already run them — a second alert path for one service is how teams learn to ignore both. And if you need escalation chains, acknowledgement and a rotation, buy Better Stack or PagerDuty and stop reading; that's a product, not a weekend.

The before/after I want you to leave with is small. Before: one uptime check on the marketing page, green all week, while EU users watch a checkout spinner. After: two probes, an honest /healthz, and a searchable trail that tells you which dependency was sick. Your mileage may vary on the check interval — I start at 60 seconds and only go tighter once someone can actually answer the page.

References

Better Stack — uptime monitoring and incident management: https://betterstack.com/uptime
Prometheus — instrumentation best practices: https://prometheus.io/docs/practices/instrumentation/
Prometheus blackbox exporter (external probing): https://github.com/prometheus/blackbox_exporter
Grafana Loki — deployment and retention: https://grafana.com/docs/loki/latest/
GDPR Article 5 — data minimization: https://gdpr-info.eu/art-5-gdpr/
Kubernetes — liveness, readiness and startup probes: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
Sentry — cron monitors and check-ins: https://docs.sentry.io/product/crons/
Google SRE Book — monitoring distributed systems: https://sre.google/sre-book/monitoring-distributed-systems/
Infrai capability reference for AI readers: https://docs.infrai.cc/llms.txt

DEV Community: felixhoffmann556

A 3-Step LLM Moderation Preflight for Token Cost, Text, Images, and JSON

How should LLM moderation estimate token cost before classifying user text and images?

Trace one property report through 3 stages

Read the preflight as a routing signal

Make human review observable

Stop at the capability boundary

References

EU Speech-to-Text APIs: A GDPR, Data Residency, and SOC 2 Guide

How should a startup choose an EU-compliant speech-to-text API for GDPR data residency?

Pick a managed EU processing path when speed matters

Pick self-hosted recognition when infrastructure control matters

Build a typed transcript-to-rubric boundary in TypeScript

Know the limits before selecting an architecture

Further reading

Node.js RAG over messy property PDFs: chunking, embeddings, pgvector, citations

Pick the index by what you already run

How should I chunk messy listing PDFs before embedding them for semantic search?

Build the scoreboard before you pick a model

One enrichment call, end to end

Where this stack is the wrong pick

References

Choosing a Text-to-Image API for High-Quality Marketing Posters and Social Ads

TL;DR

How should a marketing app compare text-to-image API resolution, style control, and upscale quality?

Replace the gallery test with a release test

Build a copyable TypeScript evaluation harness

What about typography, upscaling, privacy, and cost?

Choose with evidence, then keep measuring

References

Python FastAPI and Node.js Error Tracking: A Common Schema for Microservices

How should Python FastAPI and Node.js microservices share error tracking?

Pick the backend by the investigation you actually perform

A common schema and capture endpoint example

Correlation works, but it is not distributed tracing

Limits that should change your recommendation

References

Uptime checks vs a self-hosted health endpoint for a SaaS MVP in the EU and US

Should a SaaS MVP self-host health endpoint metrics, or pay for hosted uptime monitoring?

The three boxes, drawn in words

What changes when your customers sit in the EU and the US

A health endpoint and one metric line, in TypeScript

What I'd actually pick, and where each one stops

References