DEV Community: Hassan

Your Extraction Agent Is 94% Accurate and Still Ships Wrong Quantities

Hassan — Thu, 05 Nov 2026 06:09:27 +0000

Document-level accuracy hides the errors that cost money. Score fields, weight them by consequence, and gate every prompt change in CI.

A team demos an agent that reads supplier quotes and customer RFQs and turns them into structured line items. The PDF-to-JSON accuracy is reported as 94%. Two weeks into the pilot, a customer receives a quote for 120 units that should have been 12. The description field was perfect. The quantity was off by a digit, and no dashboard showed it, because the metric averaged everything together.

This is the most common gap we see in LLM extraction projects: the model is good enough, and the measurement is not. If you cannot say how often the quantity field is wrong on scanned PDFs from your top five suppliers, you do not know whether the system works.

Averaged accuracy rewards the wrong behavior

Extraction outputs have fields of very different weight. A mangled product description gets noticed by a human in seconds. A wrong unit (m2 versus m3, piece versus pack of ten) or a shifted decimal in a quantity flows straight into a quote, an order, or an ERP posting.

A single accuracy number blends these together. Say a document has 8 fields per line, and the model nails 7 of them on every line. You report 87.5% field accuracy, and the one wrong field per line is the quantity. The number looks fine. The system is useless.

Document format makes it worse. Native PDFs, scanned faxes, Excel sheets with merged cells and hand-edited emails behave like different models. An aggregate over all of them tells you nothing about the scans, which is where the failures cluster. In the past month, at least four DACH-funded startups announced agents for legacy-heavy workflows: procurement, B2B trade, travel and building materials (per our market tracking, sourced from Tech.eu weekly recaps). Their domains differ and the documents are messy in the same way.

What we run on our own pipeline

Our internal sales pipeline uses LLM agents to research companies, score leads and draft outreach. None of that output is trusted on its own. Each agent run is checked against acceptance criteria: required fields present, a justification attached to every score dimension, a source URL on every claim. Before any first email leaves, a person approves it. The model drafts; deterministic code and a human decide what ships.

That setup is small, and it taught us the lesson that applies to client work: validation and review catch more than prompt tweaking does. When a field fails a check, the cheapest fix is almost never a longer prompt. It is a rule, a lookup table or a review step.

Build the eval harness before the second prompt revision

The harness is not complicated, and it should exist before anyone rewrites the prompt for the first time. We build these with the client's domain expert, because only they can label the gold set correctly.

Start with a gold set of 50 to 100 real documents, stratified by source: native PDF, scan, spreadsheet, email body. A domain expert labels every line item. Fifty documents with 20 lines each is already 1,000 labeled lines, enough to see patterns. Synthetic documents are not a substitute, because the failures come from real-world mess.

Score each field with its own match rule. Use exact match on article numbers and quantities, canonicalized match on units, fuzzy match on descriptions. Lines must be aligned first, by article number where it exists and by position otherwise. Alignment bugs in the harness produce phantom regressions, so test the aligner on its own.

type Field = 'article_no' | 'quantity' | 'unit' | 'description';
const CRITICAL: Field[] = ['article_no', 'quantity', 'unit'];

function scoreLine(pred: Line, gold: Line): Record<Field, boolean> {
  return {
    article_no: norm(pred.article_no) === norm(gold.article_no),
    quantity:   pred.quantity === gold.quantity,            // Decimal, never float
    unit:       canonicalUnit(pred.unit) === canonicalUnit(gold.unit),
    description: similarity(pred.description, gold.description) >= 0.85,
  };
}

// CI gate: fail the build if any critical field regresses on any stratum
for (const stratum of strata) {
  for (const f of CRITICAL) {
    if (recall(results, stratum, f) < baseline[stratum][f] - 0.01) {
      console.error(`regression: ${stratum}/${f}`);
      process.exit(1);
    }
  }
}

Run the gate on each document type separately. A prompt change that lifts native PDFs by two points and drops scans by six should fail, and an average would let it through.

Pin model versions and record every result. Store each eval run with the prompt hash, model identifier and per-field numbers. When a provider updates a model behind an alias, you will see the shift in the next scheduled run instead of in a customer complaint. Treat prompts as versioned artifacts with the same discipline as code; we wrote about that separately in our post on prompt versioning.

Route to humans with deterministic checks, not model confidence

Self-reported confidence from an LLM is poorly calibrated. A model can return 0.97 on a field it got wrong. Use signals you can verify instead:

Schema validation with a strict parser (Zod in TypeScript, Pydantic in Python). Reject, do not coerce, anything outside the enum of known units.
Cross-field arithmetic. If quantity times unit price does not equal the line total within rounding, flag the line.
Two-pass disagreement. Extract twice with different prompts or models and compare critical fields. Disagreement goes to review. This roughly doubles token cost on the extraction step, so apply it to critical fields only if cost matters.
Plausibility against history. A quantity of 120 where the same customer has ordered 10 to 15 for two years deserves a second look.

Flagged lines go into a review queue in Postgres with the document image next to the extracted values. Every correction a reviewer makes is stored and, after a spot check, promoted into the gold set. After a few months the gold set reflects your real traffic, and the harness gets sharper without extra labeling projects.

Staffing this

The work splits into three parts: the extraction service (Python or Node.js), the eval and review tooling (TypeScript for the reviewer UI, Postgres underneath), and the labeling process with a domain expert. The first two are ordinary backend engineering. Teams with strong domain knowledge and few backend engineers often skip the harness because it looks like overhead, then spend the pilot debugging by anecdote. A backend engineer who has built one before can stand up a first version quickly, and that costs far less than one wrong quote reaching a customer.

Key Takeaways

Report accuracy per field and per document type. A blended number hides the failures that matter.
Weight fields by consequence. Quantities, units and identifiers get exact-match rules and a CI gate; descriptions get fuzzy matching.
Gate every prompt, model and parser change against a gold set of 50 to 100 real documents, and pin model versions.
Route to human review using schema checks, arithmetic and cross-pass disagreement. Do not rely on the model's own confidence.
Feed reviewer corrections back into the gold set so the harness reflects real traffic.

SifrVentures builds dedicated engineering teams for tech companies. Based in Berlin. Learn how we work | Read more on our blog

Replayable Credit Decisions: The Backend Pattern Auditors and Engineers Both Want

Hassan — Thu, 29 Oct 2026 06:08:49 +0000

If you cannot re-run a lending decision from last March and get the same answer, you do not have an audit trail. You have a log file.

A merchant asks why their credit line was declined in March. Compliance wants the answer by Friday. The engineer who wrote the underwriting rules left in June, the rules have been edited eleven times since, and the bureau response was never stored. All that remains is a row with status = 'declined'.

This is the normal state of lending backends that grew from a prototype. Underwriting starts as an if block in a service. It becomes a config file, then a feature-flagged config file, then a set of rules that product managers adjust directly. Each step is reasonable. Together they destroy your ability to explain a past decision.

A decision is a function of inputs and rules, and you are storing neither

Most schemas store the outcome. Some store the score. Almost none store the exact inputs the rules saw, or the exact version of the rules that ran. Both are required to reproduce a decision.

The gap matters more now than it did three years ago. The CJEU's SCHUFA ruling (C-634/21, December 2023) held that producing a credit score can itself count as an automated decision under GDPR Article 22 when a lender relies on it heavily. For natural persons, including sole traders, that brings explanation and human-review obligations. The EU AI Act lists creditworthiness assessment of natural persons as a high-risk use case in Annex III. And DORA (Regulation 2022/2554, applicable since January 2025) pushes financial entities and their ICT providers toward demonstrable control over change management and incident reconstruction.

None of these regulations prescribe a schema. All of them are easier to satisfy if any past decision can be reconstructed on demand.

What we see when lending teams hit this wall

In the fintech backends we have worked on or reviewed, the failure mode is consistent. Rules live in three places: code, a database table that ops edits, and a third-party decisioning tool. Nobody can say which version ran for a given application. Bureau and bank-data responses are fetched at decision time and discarded after the score is computed. When a regulator, an auditor or a funding partner asks for reconstruction, an engineer spends days piecing it together from application logs, and the answer is "probably this."

The second symptom shows up in engineering velocity. Teams become afraid to change rules because they cannot tell what a change would have done to historical applications. A rules change is shipped on a Friday with a prayer instead of a diff.

Both symptoms have the same cause, and the fix is not exotic.

Make the evaluator pure, snapshot the inputs, version the rules

Three decisions give you replayability. None of them need a new platform. PostgreSQL and a disciplined module boundary are enough.

First, the evaluator is a pure function. It takes a rule spec and an inputs object and returns an outcome with reason codes. No clock reads, no HTTP calls, no database access, no randomness. Everything the decision needs arrives as an argument.

export interface DecisionInputs {
  applicationId: string;
  evaluatedAt: string;          // passed in, never Date.now()
  applicant: ApplicantFacts;
  bureau: BureauSnapshot;       // stored response, not a live call
  bankData: BankDataSnapshot;
}

export function evaluate(spec: RuleSpec, inputs: DecisionInputs): Decision {
  const reasons: ReasonCode[] = [];
  for (const rule of spec.rules) {
    const hit = rule.test(inputs);
    if (hit) reasons.push(rule.reason);
  }
  return {
    outcome: reasons.some(r => r.blocking) ? 'declined' : 'approved',
    reasons,
  };
}

The side effects happen before the call. An orchestrating NestJS service fetches the bureau data and bank data, writes them to storage, then hands the snapshot to evaluate. The orchestration is messy and allowed to be. The evaluator is not.

Second, rules are immutable, content-addressed versions. A rule change creates a new row. Nothing is edited in place.

CREATE TABLE rule_versions (
  id           uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  name         text NOT NULL,
  version      int  NOT NULL,
  content_hash text NOT NULL,   -- sha256 of the canonical spec
  spec         jsonb NOT NULL,
  created_by   text NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now(),
  UNIQUE (name, version)
);

CREATE TABLE decisions (
  id              uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  application_id  uuid NOT NULL,
  rule_version_id uuid NOT NULL REFERENCES rule_versions(id),
  inputs          jsonb NOT NULL,   -- full snapshot, incl. bureau response
  outcome         text NOT NULL,
  reasons         jsonb NOT NULL,
  decided_at      timestamptz NOT NULL DEFAULT now()
);

REVOKE UPDATE, DELETE ON decisions, rule_versions FROM app_role;

The REVOKE is the point. An audit trail the application can rewrite is a suggestion. Add a trigger that rejects updates for the roles that still hold the privilege, and keep the migration role separate from the runtime role.

If rules must be code rather than data, the equivalent is a git SHA stored on every decision row, plus a build that refuses to deploy rule changes without it.

Third, retention is a decision made explicitly. Storing full bureau responses indefinitely conflicts with GDPR storage limitation. Pick a retention window per data class with your DPO, encrypt the snapshot column, and delete on schedule. The decision row can keep its hash of the inputs after the payload expires, which proves integrity even when replay is no longer possible.

Replay turns rule changes from a gamble into a diff

Once decisions are replayable, you get something more useful than compliance: a test suite built from production history.

Before a new rule version ships, run it against the last few thousand stored inputs and compare outcomes to what actually happened:

$ pnpm replay --candidate underwriting@v14 --baseline underwriting@v13 --last 5000

outcomes changed: 212 / 5000 (4.2%)
  approved -> declined: 31
  declined -> approved: 181
top reason codes driving change: CASHFLOW_VOLATILITY (140), SECTOR_CAP (52)

A product manager can see that a "small tweak" would have flipped 181 declines, and decide whether that is the intent. Risk gets a number before release instead of a surprise after it. The same harness works as a regression test in CI: pin a golden set of inputs and fail the build if the outcomes change without an explicit approval.

One caveat to state plainly. Replay reproduces the rules engine, not the world. If a decision depends on a model, pin the model artifact and its version as part of the rule spec. If it depends on an external call, the snapshot is the record and the live source is irrelevant.

Where a small team goes wrong

Retrofitting is cheap when the service already has a clean seam between data fetching and rules. It gets expensive when rules are tangled with I/O, because the real work is untangling them. Retrofits tend to stall at this point, since the feature roadmap does not wait.

This is the kind of work where an embedded engineer with backend depth in PostgreSQL and TypeScript pays off. It is unglamorous, it touches the core of the product, and it needs someone who will stay long enough to see the replay harness adopted by the team that owns the rules. We hire engineers for that specific profile and embed them in the client's team, so the refactor is owned by someone who knows the codebase, not delivered as a document.

Key Takeaways

Store the inputs and the rule version with every decision. The outcome alone cannot be explained later.
Keep the evaluator pure. Fetch external data first, snapshot it, then evaluate.
Make rule versions immutable and revoke UPDATE and DELETE from the runtime database role.
Decide retention per data class up front, so replayability does not collide with GDPR storage limitation.
Use replay against historical inputs as the release gate for every rules change.

SifrVentures builds dedicated engineering teams for tech companies. Based in Berlin.

Draft created 2026-10-22 by the CTO agent. Pending partner review before publishing as a LinkedIn article.

SifrVentures builds dedicated engineering teams for tech companies. Based in Berlin. Learn how we work | Read more on our blog

Your AI Agent Will Retry. Your Booking System Will Not Notice.

Hassan — Thu, 29 Oct 2026 06:08:38 +0000

Letting an agent write to a legacy system of record takes an intent log, not a better prompt.

The agent calls create_booking. The legacy API takes 31 seconds and the HTTP client times out. The orchestrator retries the step. The second call succeeds, and the customer now holds two reservations and a confused confirmation email. The prompt was fine. The model was fine. Nothing in the stack knew that the first call had actually gone through.

Every agent pilot we look at passes its demo on read-only tools: search, lookup, summarize. The trouble starts with the first tool that changes state. Booking, order entry, ticket creation, ERP postings. That is the write path, and it needs the same engineering you would give a payment flow.

An agent has three retry layers, and the legacy system sees all of them

The first layer is the HTTP client. Most SDKs retry on connection errors and 5xx responses by default. The second is the orchestrator. Whether you use LangGraph, Temporal, a queue worker, or a loop you wrote yourself, a failed step gets re-run, and a redelivered message gets processed twice. At-least-once delivery is the default contract of nearly every queue you will use.

The third layer is unique to agents. The model reads a timeout or an ambiguous response, decides the action probably failed, and calls the tool again. This retry is not byte-identical to the first. The arguments may differ in small ways: "adults": 2 versus "adults": "two", a date in a different format, an extra optional field. An idempotency key derived from the raw request body will not catch it.

Stripe popularized client-supplied idempotency keys for exactly this problem, and they work well when the downstream system honors them. Many systems of record do not. SOAP interfaces, batch-file ERPs, and older reservation backends often have no idempotency concept at all. Some will happily create a duplicate and return a fresh ID each time.

A timeout is not a failure, and your agent needs to be told that

The most damaging bug in these systems is a tool wrapper that turns a timeout into the string "Error: request failed". The model reads that as a failure and does what a helpful assistant does: it tries again.

The honest state after a timeout is unknown. The request may have been rejected, may still be queued, or may have committed. Treat it as a fourth outcome next to success, rejection, and error. The agent should receive "outcome pending, do not retry" and a reconciler should find out what happened.

We apply the same discipline to our own internal tooling. Our outreach pipeline runs agents that draft and schedule emails, and the sender records a marker per lead and per email step, checked before every send. The agent never decides whether an email already went out. A file on disk does. The mechanism is almost embarrassingly low-tech, and it has survived crashed runs and re-triggered cron jobs because the check lives outside the model. Production systems need the same property with a stronger store behind it.

Put an intent log between the model and the legacy API

The pattern is small. The agent never calls the legacy system directly. It calls a tool wrapper that writes an intent row to Postgres and returns. A separate worker executes intents against the downstream system.

CREATE TABLE tool_intents (
  intent_key   text PRIMARY KEY,
  run_id       uuid NOT NULL,
  tool         text NOT NULL,
  args         jsonb NOT NULL,
  state        text NOT NULL
               CHECK (state IN ('pending','sent','confirmed','rejected','unknown')),
  external_ref text,
  created_at   timestamptz NOT NULL DEFAULT now(),
  updated_at   timestamptz NOT NULL DEFAULT now()
);

The key is computed from business intent, not from the request:

const parsed = BookingArgs.parse(rawArgs);        // zod: "two" becomes 2, dates become ISO
const intentKey = sha256([run.id, 'create_booking', stableStringify(parsed)].join('|'));

const res = await db.query(
  `INSERT INTO tool_intents (intent_key, run_id, tool, args, state)
   VALUES ($1, $2, 'create_booking', $3, 'pending')
   ON CONFLICT (intent_key) DO NOTHING
   RETURNING intent_key`,
  [intentKey, run.id, parsed],
);

if (res.rowCount === 0) return loadPriorResult(intentKey); // same intent, same answer

Schema validation does the normalization work. Once the model's arguments have been coerced into a strict type, two retries that mean the same thing hash to the same key, and the second insert hits the primary key and returns the earlier result. If a run can legitimately perform the same action twice, add the plan step or line item ID to the hash so the two intents differ on purpose.

The worker claims rows with SELECT ... FOR UPDATE SKIP LOCKED, which lets several workers drain the table without double-claiming. Writing the intent and enqueuing the work is one transaction, which is the transactional outbox pattern. Before it calls the legacy API, it writes the intent key into whatever free-text reference field the system offers. Almost every booking or ERP interface has one.

Reconcile unknowns by reading, never by writing again

When the worker's call times out, the intent moves to unknown. A reconciler job then queries the legacy system for a record carrying that reference. If it finds one, the intent becomes confirmed with the external ID. If nothing appears after a defined window, it becomes rejected and can be retried safely under the same key, because the reconciler has verified that nothing exists.

Meanwhile the agent sees a structured result such as {"status":"pending","intent":"..."}. A model told that a booking is pending will tell the customer it is being processed. A model told that a call failed will try something else.

Split the dangerous tools in two

Give the model hold_booking and confirm_booking instead of one create_booking. A hold carries a TTL and expires on its own, so a wrong hold costs nothing. Confirmation is the irreversible step, and it can sit behind a policy check, an amount threshold, or a human approval click. Our own outreach pipeline works this way: agents draft freely, and a person approves before the first email to a new contact goes out.

This split also gives you a clean place to enforce limits. The model can propose a thousand holds. Only a bounded number can ever become confirmations.

What it takes to build

None of this needs an ML team. It is Postgres, a worker process, schema validation, and a reconciler, which makes it a solid fit for one senior backend engineer working in NestJS or Python and owning the integration boundary. The unglamorous parts, such as the reconciler and the monitoring on unknown intents, decide whether the agent can be trusted with real customers. Teams under funding pressure tend to postpone them until the first duplicate order arrives.

Key Takeaways

Agents retry at three layers: HTTP client, orchestrator, and the model itself. Only the first two are visible in your logs.
Treat a timeout as unknown, tell the model "pending, do not retry", and resolve the state by reading the downstream system.
Derive idempotency keys from validated, normalized business arguments, not from raw request bodies or LLM-generated call IDs.
Route every state-changing tool through an intent table in Postgres, with a worker and a reconciler between the model and the system of record.
Split irreversible actions into a cheap, expiring hold and a gated confirmation.

SifrVentures builds dedicated engineering teams for tech companies. Based in Berlin. Learn how we work | Read more on our blog

Platform Engineering After Series A: The Kubernetes Debt That Slows Every Engineer on Your Team

Hassan — Thu, 27 Aug 2026 05:58:43 +0000

Most SaaS teams hit a DevOps inflection point 12-18 months post-funding. By the time it's visible, one person holds all the infrastructure knowledge and every deploy carries risk.

At the moment a Series A closes, a startup's infrastructure is running exactly as well as it needs to. One engineer set up the Kubernetes cluster eight months ago. It hasn't broken. Deploys take 20 minutes but they work. The team is focused on product. Nobody is thinking about platform engineering.

Twelve months later, that cluster is the single biggest constraint on engineering velocity, and the problem doesn't appear in any sprint board.

This is not an edge case. It is the default trajectory for engineering teams between €5M and €30M in funding, across fintech, healthtech, and B2B SaaS in DACH. The pattern repeats because the incentives at seed stage reward individuals who ship, not teams who document. That's a rational tradeoff early on. It stops being rational once the team doubles.

Why the Founding DevOps Pattern Breaks Down

At seed, a single engineer owns all infrastructure. This is correct. There is nothing to run at scale, and the fastest path is one person who knows where everything lives. They install Kubernetes on a managed cloud provider (EKS or GKE, usually), write Helm charts or skip Helm and commit manifests directly, wire up a CI/CD pipeline to the main branch, and move on.

Three things go wrong quietly over the next year.

Staging drifts from production. Staging is lower-cost, so it doesn't get the same updates. Teams start shipping directly to production "just this once." Six months of "just this once" later, staging is an ornament. Nobody catches regressions before they reach users.

Infrastructure state ends up somewhere nobody planned for. Terraform state in a local file, or an S3 bucket without state locking. Secrets committed to environment variables in deployment YAML. A load balancer rule that was added manually in the cloud console and never codified. A Kubernetes namespace with resource limits last touched 14 months ago. None of this is visible until it breaks under pressure.

Then the bus factor arrives. The founding DevOps engineer leaves, moves to a different project, or takes parental leave. Nobody else fully understands the cluster. The next time something breaks at 2am, a backend engineer is SSHing into a jump host from a runbook that is nine months out of date. This is when leadership usually realizes they have an infrastructure problem, which is the worst possible time to discover it.

A Berlin Fintech Was Losing 60% of a Senior Engineer to Infrastructure Overhead

At a Berlin fintech we worked with, the lead backend engineer was spending three days a week on infrastructure. Not because they wanted to. Because they were the only person who could. Two of their five Kubernetes namespaces had no live documentation. GitHub Actions pipelines ran only on main branch merges, with no PR-level checks or preview environments. Terraform state lived in an S3 bucket without DynamoDB state locking, and two engineers had accidentally run concurrent applies on the same workspace in the same quarter. Both applied cleanly. Nobody knew which one won until they checked the cloud console.

Their engineering team was capable and the product was shipping. But 60% of one senior engineer's time had been redirected to infrastructure maintenance, and the team had stopped tracking it as overhead because it had become normal.

The cost wasn't visible in velocity metrics because nobody had a baseline to compare against. It showed up in conversations: "we need to hire another backend engineer." The real problem was that one backend engineer wasn't doing backend engineering.

The GitOps Migration: What Six Weeks Buys You

The fix is not glamorous, and it doesn't require a full-time platform engineering team. For a team running 6-20 backend services on Kubernetes, six weeks of focused work by one or two embedded platform engineers will address 80% of the operational risk.

The pattern that works consistently at this scale:

Start with ArgoCD for cluster state management. Every Kubernetes resource that should exist in the cluster lives in a Git repository. ArgoCD continuously reconciles that Git state to the actual cluster state. Nobody runs kubectl apply manually in production. Drift is detected in the ArgoCD UI before it causes an incident. "What is running in production right now?" becomes a Git blame command, not a 20-minute investigation.

Alongside ArgoCD, standardize on Helm. Running eight backend services without Helm usually means maintaining eight nearly-identical collections of deployment YAML. A base Helm chart template eliminates that duplication. Security patches, resource limit adjustments, and environment variable structure changes become a one-line update per service instead of eight separate PRs. For a team deploying twice a week, this compounds quickly.

Secrets need to leave environment variables in deployment YAML. External Secrets Operator with AWS Secrets Manager (or GCP Secret Manager) handles this: credentials are managed in the cloud provider's secrets service and synced into Kubernetes at deploy time. Rotating a credential becomes an AWS console action. It never touches Git history.

Terraform gets remote state and DynamoDB state locking: one workspace per environment, terraform plan output reviewed in CI before any apply is permitted. Concurrent apply accidents, the kind that silently overwrite each other, become structurally impossible. Infrastructure changes go through the same review process as application code.

The deployment pipeline itself: GitHub Actions on merge to main builds and pushes a Docker image to ECR, updates the image tag in the ArgoCD Git repository, and triggers sync. Staging runs the pipeline first, with a manual promotion gate before production. This is not sophisticated engineering. It is reliable, auditable, and understandable by any engineer on the team without three hours of context transfer.

For the Berlin fintech team above, this migration took one embedded DevOps engineer six weeks working alongside their existing backend team. By week eight, the engineer who had been spending three days a week on infrastructure was down to two hours a week on operational tasks. Four months later, they hired a second backend engineer, and the new hire was contributing to production within three weeks.

Four Signals That the Inflection Point Has Already Arrived

The right moment to address Kubernetes infrastructure debt is before a key person leaves, not after. These signals indicate the inflection point:

Deploy frequency has dropped over six months without a deliberate decision to slow down. More than one engineer raised infrastructure as a blocker in the last quarter. Your on-call rotation is one person. Nobody can confidently answer "what is deployed in production right now?" without SSHing somewhere.

The wrong solution at this stage is hiring a full-time senior DevOps engineer unless you already have four to six backend engineers generating infrastructure work. One platform hire at 10 engineers spends their first three months fixing accumulated problems and their next three months maintaining the fix, with no capacity for the proactive work that actually scales a team.

The pattern that works for teams between 8 and 25 engineers: one to two embedded platform engineers for a defined migration scope (12-16 weeks), with explicit knowledge transfer and runbooks before handoff. The goal is not a permanent platform team. The goal is an infrastructure that any backend engineer can operate without calling the one person who set it up.

Key Takeaways

Infrastructure debt at Series A compounds faster than application debt because it affects every engineer, not just the team working on the affected service. When your most experienced backend engineer becomes your de facto DevOps engineer, you've lost your most expensive resource to the wrong problem.
GitOps with ArgoCD pays back within eight weeks, not six months. The setup investment is real, and the operational improvement is faster than most teams expect.
The founding DevOps pattern is a rational seed-stage decision that needs active dismantling in the first 12 months post-Series A. The window before it becomes a crisis is shorter than it looks.

SifrVentures builds dedicated engineering teams for tech companies. Based in Berlin.

Published: 2026-08-27

SifrVentures builds dedicated engineering teams for tech companies. Based in Berlin. Learn how we work | Read more on our blog

Why Market Five Breaks Your Payment Backend

Hassan — Thu, 20 Aug 2026 05:46:04 +0000

The integration debt that compounds quietly across payment providers, and the architecture decision that stops it.

The first payment integration is never the problem. You pick Stripe for Germany, wire up the webhook endpoint, handle payment_intent.succeeded, payment_intent.payment_failed, and charge.refunded, build a nightly reconciliation job, and ship. It takes five weeks. Everything works.

Market two takes four weeks. You add a second provider, extend the webhook handler with a conditional block, duplicate the reconciliation logic with minor differences for the new export format. The provider uses different event names for the same lifecycle states. You normalize them manually. Still fine.

By market five, the PR review for a new provider integration takes four days before anyone approves it, because no one fully understands the webhook handler anymore. Each provider fails differently. Refund flows are all slightly wrong in different ways. The reconciliation job is a 700-line script that runs for three hours and occasionally produces negative discrepancies no one can explain without reading the original engineer's Slack messages from two years ago.

This is not a code quality problem. It is a structural one, caused by a decision made at integration one that does not show its cost until integration five.

The First Integration Sets a Pattern

The natural approach when building the first payment integration is to model the provider's API surface directly. Stripe gives you PaymentIntent, Charge, Refund, and Webhook. You build types that mirror those, write handlers that know they are talking to Stripe, and ship. The code is clean, specific, and correct.

The problem is that Stripe-shaped code only supports Stripe. Every subsequent provider has a different data model, different webhook vocabulary, different lifecycle semantics, different retry contracts, and different reconciliation export formats. When you add provider two, you have two options: abstract, or duplicate. Most teams duplicate, because abstracting at that point requires a refactor that feels risky for a provider that has not shipped yet.

That decision does not hurt immediately. It hurts when:

A provider sends a duplicate webhook for an event the others send once. Your idempotency logic was written for Stripe's guarantee, not this provider's behavior. Payments start double-counting in the settlement report.

A payment that fails at provider A needs retrying at provider B. There is no shared payment entity in your system — just provider-specific records with incompatible status enums. You build the retry logic as a special case, which becomes technical debt the moment you add provider three.

A compliance audit asks for a full payment lifecycle audit trail across all markets. You can produce it per-provider. You cannot produce a unified view without writing a query layer that joins across three separate data schemas authored at different times.

None of these are hypothetical. They are the standard second-year failures of a payment backend built integration-by-integration without a shared domain model.

What Happens Without the Abstraction

At one client operating across five European markets, each payment integration had been built at the time of market entry — different retry policies, different webhook validation approaches, different error categorizations. The reconciliation process required a separate script per provider because each had a different export format, and those formats had drifted as providers released API updates.

A new market entry, which should have been a contained backend task, required touching all five existing integrations to normalize the shared error taxonomy the new provider's SDK introduced. Engineers who had not written the original integrations were modifying them. Code review was slow. The timeline slipped.

The engineering team was not slow. The code was not poorly written. The problem was that five provider adapters had been built in isolation rather than against a shared interface. Every cross-provider operation required coordinating across five data models with no common language between them.

The rebuild took one backend engineer six weeks: a shared Payment domain model with a normalized lifecycle (created, authorized, captured, settled, failed, refunded), a provider adapter pattern, and reconciliation logic that ran once against the shared model instead of per-provider. Three additional market entries since then have required writing one adapter each. The reconciliation job has not been touched.

The Architecture That Survives Scale

The abstraction layer that makes multi-provider payment backends maintainable has three components.

A shared Payment domain model. Define the payment lifecycle in your domain vocabulary, not your first provider's vocabulary. PaymentState.Authorized is not PaymentIntent.requires_capture. It is a domain concept that maps to different provider states per context. Internal systems — reconciliation, reporting, refund orchestration, customer notifications — work against this model exclusively. They never import provider-specific types.

Provider adapters with explicit contracts. Each adapter handles translation for one provider: inbound webhook events to domain events, outbound commands to provider API calls. The adapter boundary contains the inconsistencies. Stripe's idempotency contract is handled inside the Stripe adapter. If Adyen's is different, that is the adapter's problem, not the reconciliation engine's.

A minimal TypeScript adapter interface covers the surface area:

interface PaymentProviderAdapter {
  initiatePayment(request: PaymentRequest): Promise<PaymentResult>;
  handleWebhook(raw: Buffer, signature: string): Promise<DomainEvent[]>;
  issueRefund(paymentId: string, amount: Money): Promise<RefundResult>;
  reconcile(from: Date, to: Date): AsyncIterable<ReconciliationRecord>;
}

Every provider implementation satisfies this contract. The reconciliation engine, retry scheduler, and refund handler call only the interface. Adding a new provider does not change those systems.

Idempotency at the domain level. Build idempotency around a client-generated key your system controls, not around provider-specific guarantees. Store the key with the payment record. Deduplicate incoming webhooks against it before processing. Provider idempotency contracts vary significantly between vendors and change between API versions. Depending on them makes your idempotency guarantees only as strong as your weakest provider's — which you discover during an incident.

When to Build It

The abstraction is not technically complex. The reason most teams skip it at integration one is that the engineering team is also building the core product, and the one-provider path is shorter. That is a rational call in the moment.

The teams that pay the rebuild cost earliest — at integration two or three, before the pattern is entrenched — do so because they had dedicated backend capacity for the refactor alongside the integration work. The teams that delay do so because the engineers who understand the existing integrations are maintaining them while also shipping the next one.

Senior backend engineers with payments domain experience in Germany take 38-54 days to hire at the junior end and considerably longer at senior level (Source: Stack Overflow Developer Survey, DACH 2025). For a fintech team entering two new markets per quarter, that hiring timeline conflicts with delivery if the refactor and the next integration are scheduled concurrently on the same people.

The abstraction is a two-sprint engineering project. The rebuild, once you have five providers, is a six-week one. The earlier it happens, the cheaper it is. That arithmetic is straightforward. What is less obvious is that it requires engineering capacity that is not already allocated to maintaining what you shipped.

Key Takeaways

Multi-provider payment backends accumulate structural debt when each integration is built against the previous provider's API surface rather than a shared domain model. The cost is invisible until integration four or five.
Define a Payment domain model before integration two. Internal systems should never import provider-specific types.
Provider adapters with a common interface contain inconsistencies at the integration boundary. Reconciliation, retry logic, and refund orchestration stay stable as providers change.
Build idempotency around a key your system controls. Provider idempotency guarantees are not consistent enough to be your foundation.
The abstraction rebuild is a two-sprint task at integration two and a six-week task at integration five. Scheduling it requires engineering capacity that is not already allocated to delivery — which means planning it ahead of the next market expansion, not alongside it.

SifrVentures builds dedicated engineering teams for tech companies. Based in Berlin. Learn how we work | Read more on our blog

Grafting AI Agents onto a Working SaaS Platform Is a Different Engineering Problem

Hassan — Thu, 13 Aug 2026 05:51:25 +0000

You have a reliable platform, 5,000 customers, and an AI roadmap. The engineering work ahead is not what you planned for.

Your platform works. It has been in production for years, serves enterprise customers, and your engineering team knows every corner of it. Then the Series C closes and the board mandate lands: add AI agents. The roadmap is clear. The engineering cost is not.

Adding an agentic layer to a working SaaS platform is architecturally different from building an AI-native product from scratch. The teams that discover this early ship faster. The ones that discover it in production spend Q3 doing things they thought were done.

Where Existing Platforms Break Under Agentic Load

A mature SaaS backend is built for request/response. A user action triggers a synchronous call, a service handles it, a response comes back. That contract is reliable because it is bounded: one input, one output, predictable latency.

AI agents break that contract in three specific ways.

Agent workflows are stateful and multi-step. An agent processing an invoice does not do it in a single call. It extracts data, calls a validation tool, queries an enrichment service, makes a classification decision, and writes back a result. Each step can fail independently. The state has to survive across steps. In a request/response system, there is no standard place for that state to live. Teams improvise, usually with a combination of Redis and database rows, and end up with consistency problems neither their existing monitoring nor their existing retry logic was built to catch.

Tool calls are synchronous by default in most LLM client libraries, but the tools themselves are often slow. An agent calling an internal ERP endpoint, a regulatory data provider, or a document processing service can block for two to thirty seconds per call. Synchronous tool calls on an existing service mesh, under the thread budget of a Java microservice or a Node.js worker, produce cascading timeouts that look like infrastructure problems but are actually concurrency budget problems.

Context windows are a new kind of resource to manage. A standard backend service does not manage prompt state. The agentic layer does. As workflows grow more complex and the number of tool calls per agent session increases, the total token cost per workflow grows. Teams that treat LLM API calls like database queries discover at scale that their per-request cost is variable in a way nothing else in their stack is. Rate limits hit in ways that do not resemble any failure mode their on-call runbook anticipated.

What We Have Seen in Practice

At one client running a financial document processing workflow, the existing backend handled thousands of transactions per day across a Java microservice architecture on Kubernetes. Adding an agentic review layer seemed like a contained engineering task. The agent extracted fields from documents, cross-checked them against existing database records, and flagged discrepancies.

Three weeks into production, two problems appeared that no load test had surfaced. First: agent sessions that processed large documents hit the context limit of the chosen model silently, producing partial outputs that passed schema validation but contained truncated reasoning. The LLM client swallowed the overflow error and returned a 200. The workflow logged success. The flagging was wrong.

Second: the internal enrichment service called by the agent had a rate limit of 60 requests per minute per API key. Under normal conditions, this was never a problem. The agentic layer sent three to five tool calls per document. At 20 concurrent agent sessions, the enrichment service started returning 429s. The retry logic in the agent client queued retries on the same thread, consuming the worker pool while new agent sessions waited for capacity that was not coming.

Both problems were solvable. The solutions were not complex: a tokenizer check before each API call that hard-rejected requests over the context limit, and a dedicated async worker queue for tool calls with a circuit breaker on the enrichment endpoint. What took time was diagnosing two failure modes that did not exist before the agentic layer arrived and did not resemble any failure mode the team had seen before.

The Engineering Patterns That Hold Up

Three structural decisions determine whether an agentic layer integrates cleanly or fights with the rest of the platform.

Keep agent orchestration out of the application service layer. LangGraph, Temporal, and similar workflow engines provide the state management, retry logic, and step visibility that agentic workflows need. Running orchestration inside an existing application service mixes two different execution models in one process: the bounded, synchronous request handler and the stateful, long-running workflow. They do not share resource assumptions well. A dedicated orchestration runtime isolates the failure domains.

Instrument tool calls as first-class spans, not log lines. Standard OpenTelemetry instrumentation covers infrastructure and application-layer latency. Tool calls are neither. Every call from an agent to an external or internal service should emit a trace span with the tool name, input size, latency, and result code. This makes it possible to answer the question "which tool call is slowing agent sessions?" in minutes rather than hours. Without it, the debugging path is log-searching across systems that were not designed to correlate.

Treat model version as a deployment artifact. Every team eventually has a production incident caused by a provider-side model behavior change. An alias like gpt-4o or claude-3-5-sonnet hides which exact version handled each request. When output quality shifts, there is no clean path from "when did this start?" to "what changed?" Pinning to exact model identifiers and treating upgrades as PRs with evaluation runs removes an entire class of hard-to-diagnose regressions.

The Capacity Question No One Plans for in Advance

The architectural problems are solvable, but they require engineering time that most teams have not budgeted when they scope the agentic feature.

An established platform team is already fully allocated to the platform. The engineers who know the data model and the service boundaries are the ones maintaining it, handling incidents, and shipping the next product quarter. Adding a net-new architectural pattern, with new failure modes and new monitoring requirements, on top of that team's existing capacity is the decision that delays most agentic rollouts by two to four months.

The teams that ship agentic features on schedule tend to add capacity specifically for the integration work, not because they are understaffed on the core platform but because the integration work requires people who are not already context-switching between production reliability and new development. The work of connecting an LLM orchestration layer to an existing event-driven backend, and making it observable and reliable, is full-time work for the period it takes to do it.

In Berlin's engineering market, a senior backend engineer with LLM integration experience commands EUR 90-120k annually, and takes four to six months to hire from first posting to first commit (Source: Glassdoor DACH 2025). For a company shipping on a Series C mandate, that timeline is the product risk. The agentic roadmap is not waiting for the hire to clear.

Key Takeaways

Request/response architectures do not extend cleanly to stateful, multi-step agent workflows. The failure modes are different and will not show up in your existing monitoring until they hit production.
Tool call concurrency and model context limits are resource constraints with no equivalent in standard backend services. They need dedicated instrumentation before you need it in an incident.
Run agent orchestration in a dedicated runtime rather than inside existing application services. The execution models are incompatible at scale.
Pin model versions exactly. Treat model upgrades like dependency upgrades: deliberate, in a PR, gated on evaluation results.
The engineers who know your platform are not free to build the agentic layer. Adding capacity specifically for the integration period is the decision that determines whether the agentic feature ships on the Series C timeline or slips into the next cycle.

SifrVentures builds dedicated engineering teams for tech companies. Based in Berlin. Learn how we work | Read more on our blog

Production LLM Observability: What You're Missing When Your AI Feature Goes Dark

Hassan — Thu, 06 Aug 2026 05:56:19 +0000

Three silent failure modes that standard APM misses, and the instrumentation layer that catches them before your users do.

Your AI feature passes load testing. Latency is under two seconds, error rate is under one percent, the product demo runs clean. You ship it. Three weeks later, a customer reports that the AI-powered output "stopped making sense." Your dashboard shows nothing wrong. The endpoint is returning 200. The logs are quiet.

This is the observability gap in production AI backends. It is different from anything traditional monitoring tools were built to catch.

Why Standard APM Fails for LLM-Backed Services

HTTP response codes, p95 latency, and error rates describe the transport layer. For a REST endpoint serving cached data or querying a relational database, that is enough. For an LLM-backed feature, a 200 response tells you almost nothing about whether the system is working correctly.

A 200 from OpenAI or Anthropic can mean: the model returned a valid response. It can also mean: the model silently truncated your input because it exceeded the context window and returned a coherent-but-incomplete answer. Or: the response is syntactically correct but output quality has shifted because someone updated the system prompt two weeks ago without an evaluation run.

The failure modes that hit AI-native backends in production do not show up as error spikes on a Grafana dashboard. They show up as a slow erosion of output quality that accumulates over days until a user complains.

The Three Failure Modes

Silent Context Truncation

Most LLM clients truncate inputs silently when they hit the model's context limit, or throw an exception that gets swallowed by a generic catch block. Your feature continues returning 200s, but the model is working from a partial view of the data. Depending on what gets cut, the output degrades in ways subtle enough to pass automated validation but wrong enough that attentive users notice. The truncation is systematic, predictable, and completely invisible to standard monitoring.

Rate Limit Cascades

Token-per-minute limits are enforced per API key. At low traffic you will never hit them. At 200 concurrent users, a single high-traffic hour can push you into 429s that your retry logic converts into a thread-blocking cascade. If you are not tracking cumulative token consumption per minute per model, you have no signal before you hit the wall. By the time 429s appear in your error logs, you are already degraded.

Prompt Regression

Output quality is a function of three things: your prompt, your model version, and your input data distribution. All three change over time. Model providers update default behavior in minor versions. Engineers edit prompts without evaluation runs. Real user traffic exposes edge cases that your staging environment never did. Without a structured evaluation pipeline, prompt regressions surface as customer complaints rather than CI failures.

What We've Seen

At one client running an AI-powered document processing feature, the backend was handling several hundred LLM calls per day across two providers. Standard monitoring showed nothing unusual. A code review surfaced a context overflow condition that had been silently truncating inputs for eleven days. No user had seen enough degraded outputs to raise a ticket, but the truncation was systematic for documents over a certain length.

The fix took two hours: a tokenizer check before each API call, plus a hard rejection of requests that would overflow the context window. What was missing before the fix was a dashboard showing the distribution of input token counts per endpoint. That would have flagged the problem on day one.

A second pattern: a provider-side model version bump. One team was pinned to gpt-4-turbo by alias, not by exact identifier. The provider's new default version had different formatting behavior for a structured JSON output schema the feature depended on. Schema validation failures started appearing in the retry logs. It took four hours to diagnose because there was no span attribute recording which exact model version had handled each call.

The Instrumentation Stack

Standard APM covers infrastructure observability. AI-native backends need a second layer on top of it. Four components cover the surface area.

1. Structured OpenTelemetry Spans on Every LLM Call

Every call to an LLM provider should emit an OpenTelemetry span recording: exact model identifier, input token count, output token count, latency, and provider HTTP status. This is the raw data for everything else.

// NestJS — LLM call wrapped in an OpenTelemetry span
import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('llm-service');

async function callLLM(prompt: string, model: string): Promise<string> {
  return tracer.startActiveSpan('llm.completion', async (span) => {
    span.setAttributes({
      'llm.model': model,
      'llm.prompt_tokens': estimateTokens(prompt),
    });
    try {
      const res = await openai.chat.completions.create({
        model,
        messages: [{ role: 'user', content: prompt }],
      });
      span.setAttributes({
        'llm.completion_tokens': res.usage?.completion_tokens ?? 0,
        'llm.total_tokens': res.usage?.total_tokens ?? 0,
      });
      span.setStatus({ code: SpanStatusCode.OK });
      return res.choices[0].message.content ?? '';
    } catch (err) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
      throw err;
    } finally {
      span.end();
    }
  });
}

Token count and exact model version per call flow into Prometheus via an OTEL collector. From there, you have the foundation for rate limit alerting and model version attribution.

2. Token Budget Alerts

Set a Prometheus alert at 70% of your per-key TPM limit. That gives you a two-minute window to queue requests, shed load, or page on-call before hitting 429s.

# Prometheus alert rule
- alert: LLMTokenBudgetHigh
  expr: |
    sum(rate(llm_tokens_total[1m])) by (api_key, model)
    > 0.70 * on(api_key) llm_tpm_limit
  for: 2m
  annotations:
    summary: "Token budget at {{ $value | humanizePercentage }} for {{ $labels.model }}"

The 70% threshold is not arbitrary. At 70%, you have runway to respond before degradation starts. At 90%, you are already mid-cascade.

3. Exact Model Version Pinning

Pin to exact model identifiers, not aliases. gpt-4o-2024-08-06 not gpt-4o. When you upgrade, do it in a PR with a structured evaluation run in CI. Treat model upgrades like dependency upgrades: deliberate, reviewed, gated on test results.

This is the lowest-effort, highest-leverage change most AI-native teams can make this week. Add one line to your model config, and the four-hour diagnosis of "why did output quality shift" becomes a one-minute log query.

4. Evaluation Pipeline in CI

For any feature where output quality matters — which is most LLM features — run structured evaluations against a held-out test set as part of your deployment pipeline. Langfuse integrates directly into most AI backends and can gate deployments on score thresholds.

// Langfuse trace for evaluation tracking (TypeScript)
const langfuse = new Langfuse({ publicKey, secretKey, baseUrl });

const trace = langfuse.trace({ name: 'document-summary' });
const generation = trace.generation({
  name: 'summarize',
  model: 'gpt-4o-2024-08-06',
  input: prompt,
});
generation.end({ output: result });

Your evaluation script queries Langfuse for the latest generation traces against your test set and fails the deployment if quality metrics drop more than 5% from baseline. Prompt regressions become CI failures, not support tickets.

The evaluation test set does not need to be large. Twenty to thirty representative inputs with labeled expected outputs catches the vast majority of regressions in our experience. The effort to build it is one day. The effort to maintain it is twenty minutes per sprint, when new edge cases surface from production traffic and get added to the set.

The Engineering Cost

Getting this stack in place takes one backend engineer two to three days, assuming they have shipped LLM features in production before. OpenTelemetry setup is the longest part. Prometheus alerts and Langfuse integration each take a few hours once the spans are flowing.

The alternative is discovering these failure modes after they have been running silently for a week. That is a four-hour diagnosis, a customer apology, and a post-mortem. Every team we have worked with that shipped this observability stack before their first production incident has been glad they did. Every team that shipped it after their first production incident has said the same thing.

Key Takeaways

Standard APM covers transport-layer health. The three AI-specific failure modes — silent context truncation, rate limit cascades, and prompt regression — each require dedicated instrumentation.
OpenTelemetry spans on every LLM call are the foundation. Token count, exact model version, and latency per call give you the raw data for everything else.
Pin to exact model identifiers, not aliases. One config change. Eliminates a class of four-hour debugging sessions.
Evaluation pipelines should be in CI before your first production incident, not after. The cost is two to three engineer-days. The cost of not having it is a week of silent degradation followed by a customer complaint.

SifrVentures builds dedicated engineering teams for tech companies. Based in Berlin. Learn how we work | Read more on our blog

Engineering While the CTO Role Is Empty: How DACH Scale-Ups Are Staying on Roadmap

Hassan — Thu, 30 Apr 2026 05:47:53 +0000

The three to six months between "we need a CTO" and "we have one onboarded" is the most technically expensive period a scale-up can go through. Most companies find out the hard way.

Across Berlin's tech scene in Q1 and Q2 2026, a specific hiring pattern has become visible: companies with 50 to 150 employees, real revenue, and active engineering teams posting for a CTPO or CTO while simultaneously shipping product, preparing a funding round, or scaling into new markets. These are not early-stage companies still figuring out product. They are operational businesses with engineering orgs that need technical leadership now.

The search will take four to six months, minimum. The question is not whether the company can survive that window. It is what gets broken, deferred, or quietly abandoned inside it.

What Actually Breaks During a CTO Search

The most visible risk is velocity. When there is no technical anchor, architecture decisions get deferred. A senior engineer who would normally escalate a database schema question waits for someone with the authority to decide. The decision gets made in a sprint review instead of an architecture session. Six decisions like that, and the data model has cracks that will take a quarter to fix.

But velocity is the measurable symptom. The harder problems are invisible until they compound.

Technical debt accumulates faster under consensus. With no CTO, architectural decisions often default to team consensus or the most vocal engineer in the room. Consensus-driven architecture is not inherently bad, but it tends toward local optimizations. Each team member solves their piece well. Nobody owns the system boundaries. Six months later, a new CTO arrives to find four services that should have been one, two different authentication patterns, and three ORMs in the same codebase.

Onboarding new engineers stalls. A clear technical authority accelerates onboarding by providing definitive answers on stack decisions, code standards, and architectural direction. When that authority is absent, new engineers spend their first four weeks navigating informal consensus. Time-to-contribution stretches from three weeks to six. In a company that needs to scale its engineering capacity during a CTO search, this is a direct constraint on output.

Attrition risk increases among senior engineers. This is the one that surprises founders most. Senior engineers do not leave because the CTO role is empty. They leave because decision-making becomes slow, frustrating, and politically complicated in the vacuum. The engineer who was happy building in a clear system starts spending meeting cycles relitigating settled questions. After two months of that, they start taking recruiter calls.

According to the DORA 2024 Accelerate State of DevOps report, teams reporting low psychological safety and unclear technical ownership show measurably lower deployment frequency and higher change failure rates. The effect is not subtle.

What the Search Process Does to the Engineering Org

A CTO search is not a background process. It pulls on the same people who are supposed to be shipping product.

Someone has to define the role. Someone has to run first-round interviews. Someone has to evaluate technical assessments. In companies without an existing CTO, that work lands on the founder, a VP of Product, or the most senior engineer, none of whom were hired to run a C-suite search while also managing a sprint cycle.

The most effective DACH companies navigating this transition have separated two things that are often conflated: the CTO search process and the technical leadership function.

The search is a hiring project. The leadership function is an operational requirement that cannot pause for three to six months. Treating them as the same problem is why so many companies arrive at the end of a CTO search with a strong new hire and a codebase that needs six weeks of architectural triage before they can get to actual roadmap work.

What Interim Technical Leadership Actually Looks Like

The companies that come through a CTO gap cleanest tend to do three things.

First, they document the implicit architecture. Before the search begins, the founding team or most senior engineers produce an honest snapshot of the current system: service boundaries, data models, known debt, and open architectural questions. This is not a polished document for investors. It is a working reference that stabilizes team decisions during the vacuum and gives the incoming CTO a real starting point rather than months of archaeology.

Second, they assign decision authority explicitly. Not with a title, but with a scope. "On API contract changes, this person has final say until we hire a CTO" is more effective than leaving decisions to consensus. The assignment should cover: system architecture, infrastructure spending, hiring decisions, and external technical commitments. Each category needs one owner.

Third, they extend the engineering capacity before the search ends. The instinct is to wait for the CTO to arrive and then hire. But the incoming CTO needs a functioning team to lead. Embedding one or two engineers with context on the specific stack before the hire lands means the new CTO spends their first weeks orienting on strategy rather than emergency staffing.

We have seen this pattern play out across DACH teams across multiple industries. The companies that handled the gap well had done the groundwork on people and process before the new technical leader arrived. The ones that struggled had paused engineering capacity decisions while waiting for the CTO to own them.

The Compounding Argument for Not Waiting

There is a financial case here that does not require any projections.

A Series A engineering team in Berlin has an average loaded cost of EUR 85 to 120K per engineer per year. Each month of reduced velocity during a CTO gap costs somewhere between two and four weeks of that team's effective output, depending on how well the gap is managed. Across a team of eight engineers and a five-month search, that is a conservative EUR 80 to 160K in reduced throughput, before accounting for any attrition.

The gap itself is not avoidable. The compounding is.

The Question Worth Asking Before You Start the Search

Most technical founders spend their energy on finding the right CTO candidate. Fewer spend equivalent energy on answering a more immediate question: what has to stay stable in our engineering org for the next six months regardless of who is leading it?

The answer to that question determines whether the incoming CTO inherits a team in good condition or a team that has been slowly falling apart since the day the search started.

What architectural decisions in your current codebase would cause the most damage if made by committee for the next four months?

SifrVentures builds dedicated engineering teams for tech companies. Based in Berlin. Learn how we work | Read more on our blog

AI Automated 50% of Your Operations. Your Backend Team Is Busier Than Ever.

Hassan — Thu, 23 Apr 2026 05:26:32 +0000

The AI deployment paradox: every workflow you automate creates three new engineering surfaces.

The companies building AI into production — nursing documentation, accounts payable, energy ops, patient intake — are discovering something uncomfortable. The AI is working. Response rates are up, manual tasks are shrinking, the demo looks great. And the backend engineering queue is longer than it was before the model shipped.

This is not a bug. It's the physics of AI at scale.

What Actually Happens After AI Ships

When you automate a manual workflow with AI, you don't reduce complexity. You transform it. The human who used to do the task understood context implicitly, recovered from edge cases, and escalated when something felt wrong. Your AI doesn't. It generates output, and the engineering team owns everything that happens next.

Three surfaces appear immediately:

The data pipeline. Your model is only as good as what feeds it. Clinical notes need cleaning before transcription. Invoice data needs normalization before extraction. Meter readings need validation before pricing decisions. The data engineers who were on the roadmap but not urgent? Now they're urgent.

The monitoring layer. Humans notice drift. Models don't. A nurse documentation system that starts categorizing wound care as medication administration will keep going until someone builds the detection logic to catch it. For every inference endpoint you put in production, you need latency monitoring, accuracy regression tracking, and a human escalation path. None of that ships with the model.

The integration surface. Your AI touches existing systems. EHR APIs, ERP connectors, billing modules, IoT device streams. Each integration is a live dependency with its own versioning, rate limits, and failure modes. As you expand across facilities, clients, or markets, every new customer brings a new integration variation.

The companies in DACH seeing this most acutely are the ones who shipped AI fastest: healthcare documentation platforms integrating with 50+ EHR systems, energy management platforms wiring IoT meter networks into dynamic pricing, HR API companies adding AI layers on top of 200+ existing integrations. Their engineering teams didn't shrink. They grew, and still couldn't keep pace.

What 12-18 Months Post-Launch Looks Like

We work with companies that have lean engineering teams, usually 10-20 engineers, building technically complex AI products. The same situation surfaces consistently around 12-18 months post-Series A or B:

The product is working. Customer count is growing. And the engineering team, which was sized for product build-out, is now also responsible for production reliability, data quality, and integration maintenance. The CTO is hiring for three roles simultaneously. Berlin's senior backend pool takes 4-6 months per hire. The roadmap slips because the people who could build the new features are keeping the existing system alive.

At one company building an AI product in a regulated sector, we started with a single backend engineer embedded in their team. Within a few months, as the data pipeline complexity grew, two more engineers joined to own the integration layer and monitoring infrastructure. The original engineer never left the team. That's the trajectory.

The work is additive, not a temporary spike.

What the Engineering Work Actually Looks Like

For teams in this position, the backlog typically breaks into three tracks:

Track 1: Data reliability. Write the validation jobs, anomaly detectors, and reconciliation scripts that catch model input failures before they corrupt output. This is Python and SQL work. It's not glamorous, and it compounds.

Track 2: Integration maintenance. The HIS in hospital A updated their API. The ERP at customer B sends timestamps in a different timezone. The IoT hub at site C drops packets under load. Each customer is an integration, and each integration has an owner. For companies expanding across Germany and into Austria or Switzerland, this surface grows with every new contract signed.

Track 3: AI observability. Latency tracking per model version, accuracy regression tests, alerting on output distribution shifts. None of this is in the LLM provider's dashboard. Your team builds it. TypeScript or Python, depending on stack. Deploys to the same Kubernetes cluster as the rest of the application. Requires engineers who understand both the ML context and production systems.

None of these tracks are one-time projects. They're ongoing engineering capacity.

The Hiring Math Doesn't Work in Berlin

Berlin has over 300 funded tech startups in active growth mode, all hiring from the same senior backend pool. The DACH software engineering salary range for senior backend roles sits between EUR 80k-120k (Source: Glassdoor DACH, 2025). Time-to-hire for a verified senior engineer runs 4-6 months including sourcing, interviews, and notice periods.

If you need two backend engineers now, you're making a bet that your production system holds for six months while you hire. In a regulated industry, with contractual SLA obligations and integration dependencies, that bet is expensive.

The alternative most teams reach for is contractors. That solves the speed problem but creates a different one: contractors don't stay on your codebase. Context doesn't accumulate. The integration engineer who joined to wire in the third EHR system is gone before the fourth one arrives, and the next contractor starts from scratch.

What we've observed across engagements: engineers who stay on a codebase long enough to own a domain ship faster and break fewer things than engineers who rotate through. The institutional knowledge compounds, and the codebase reflects it. Contractors break that cycle by design.

Key Takeaways

AI automation expands backend engineering scope. It doesn't reduce it. Plan for data pipelines, monitoring, and integrations as ongoing headcount, not one-time projects.
The 4-6 month Berlin hiring timeline is a product risk, not just a cost. If your AI is in production with SLA commitments, the gap is measured in reliability incidents.
Contractors solve the speed problem but break the context accumulation that makes the second and third integrations faster than the first.
The engineering team you need 12 months post-launch is 2-3 people larger than the one you budgeted for at Series A. The companies that plan for this hire ahead. The ones that don't, hire in crisis mode.

SifrVentures builds dedicated engineering teams for tech companies. Based in Berlin. Learn how we work | Read more on our blog

The AI Capacity Trap: Why Lean Teams Need More Engineers After They Automate

Hassan — Thu, 16 Apr 2026 05:26:45 +0000

The companies that used AI to stay lean are now discovering they need backend engineers to keep the AI running.

The pitch was compelling: instead of hiring 15 operations people, build AI workflows that handle 70% of tickets automatically. Keep the team small. Move fast. Raise on the story.

It worked. A wave of DACH scale-ups raised Series A and B rounds in 2025-2026 with exactly this model. Some had 50 employees doing what two years ago required 100. Some built care coordination AI agents that reduced manual case routing by half. Some shipped AI-assisted customer resolution that meant one support engineer could handle four times the volume.

Then the AI layer needed to scale. And the team that built it on sprint weekends while maintaining the core product hit a wall they did not see coming.

Why AI Infrastructure Is Not a Side Project

There is a category error that compounds here. When a team ships an AI feature quickly, they demonstrate that it can be built. What they do not demonstrate is that it can be maintained, scaled, and made reliable at production volume.

The difference matters in ways that are invisible until you hit them.

A care coordination AI agent that routes 50 cases a day needs different infrastructure than one routing 5,000. The prompt engineering that worked in development drifts when the model provider pushes a new version. The evaluation pipeline that caught quality regressions in staging needs continuous care as edge cases accumulate in production. The latency that was acceptable at low volume becomes a user experience problem at high volume.

None of this is research. It is plumbing. Backend engineers who understand queue management, observability, retry logic, and model versioning in production systems.

The problem is that the team who built the AI feature was the same team maintaining the core product. They are good engineers. But they are running at capacity on two incompatible modes simultaneously: the stability instincts of core product ownership and the iteration instincts of AI product development. The DORA State of DevOps research quantifies this directly: teams that split attention across two distinct product tracks have roughly half the deployment frequency of teams with focused ownership.

At 50-150 employees, you cannot absorb that tax for long.

The Pattern Across DACH Scale-Ups in 2026

This is not a prediction. It is already visible across the current cohort of DACH mid-market companies.

A Berlin healthtech company raised €37M in January 2026 with an AI agent as the core differentiation. Three months later, their job board lists backend engineering roles specifically for the AI workflow layer — separate from the core platform roles they have always hired for. The AI agent is working. Now it needs its own engineering team.

A Berlin HR-API company closed a $25M Series A in February 2026 and immediately opened "Product Engineer - AI Apply" roles alongside their standard full-stack positions. Their core integration product runs on a proven team. The AI product line is a second surface that needs dedicated ownership.

A Berlin design SaaS company with 59 engineers and $27M ARR is hiring for AI backend capacity while simultaneously hiring for core platform reliability. Two different engineering profiles, two different skill sets, same team posting.

The pattern: AI product launches with the existing team stretched across it. Traction follows. The AI layer grows. The existing team cannot own both the core product and the AI infrastructure at the required depth. Hiring starts — but now for a different profile than before.

What the AI Backend Engineering Profile Actually Requires

The engineers who maintain production AI systems are not the same profile as the engineers who built your MVP.

A backend engineer on a traditional product track optimizes for stability: migration safety, contract versioning, rollback plans. A backend engineer on an AI infrastructure track optimizes for iteration speed and observability: A/B evaluation pipelines, prompt version management, model fallback logic, latency profiling across inference providers.

Concretely, the AI backend role requires:

Prompt version control in production. Not just .env file management, but tracked, reviewed, and staged prompt changes with rollback capability. A prompt change is a code change. It needs a deployment workflow.

Evaluation pipelines, not unit tests. Unit tests verify that functions return expected values. Evaluation pipelines verify that AI outputs meet quality thresholds across representative samples. Building and maintaining these pipelines is engineering work, not prompt engineering.

Model provider abstraction. Inference providers release API changes, deprecate models, and adjust rate limits. AI backend engineers build abstraction layers that decouple application logic from provider contracts. This is the same discipline as building an integration API layer — it just applies to model calls instead of third-party REST APIs.

Observability at the output layer. Standard APM tools measure latency and error rates. AI backend observability also measures output quality drift, prompt-to-response fidelity, and hallucination rates in production. Instrumenting this requires engineers who understand both the observability stack and the model behavior.

This is a hireable profile. It is not rare. But it is a distinct hiring brief from "senior backend engineer," and the sourcing process is different.

What We've Seen Work

At one client, the AI product workstream was assigned to the same backend engineers maintaining the core platform. Within eight weeks, two things had degraded: the AI features were shipping with hardcoded model configurations instead of versioned prompt management, and a core platform refactor was deferred twice because the engineers were context-switching.

The fix was structural, not motivational. A dedicated squad took ownership of the AI infrastructure track. They ran separate standups, used different tooling, and operated on an evaluation-driven definition of done instead of a test-coverage definition. Within two months, both tracks had clearer velocity and the core platform team stopped accumulating deferred technical debt.

The staffing model that made this work was not hiring three new senior engineers in Berlin over six months. It was embedding two engineers hired specifically for the client's Node.js and Python stack, with AI infrastructure experience, in under three weeks. They joined the client's Slack on day one, attended the engineering standup on day two, and had a pull request reviewed by the end of week one.

The ramp worked because the engineering brief was specific before the hire happened. Not "backend engineer with AI experience." The client's deployment model, inference provider, evaluation framework, and prompt management approach were documented and used as the hiring filter. Engineers who matched that brief needed no ramp time to understand the problem.

Key Takeaways

AI-lean teams that achieved scale through automation now face a different engineering problem: maintaining and scaling the AI layer itself requires dedicated backend capacity.
The engineers who built the AI feature on sprint weekends are the same engineers maintaining the core product. This split attention halves deployment frequency on both tracks, per DORA research.
AI backend engineering is a distinct profile: prompt version management, evaluation pipelines, model provider abstraction, and AI-specific observability. It is hireable but not the same brief as "senior full-stack."
The structural fix is a dedicated squad with separate ownership, not sprint allocation. Team topology determines track velocity more reliably than headcount.
Embedded engineers hired to a specific AI backend brief can integrate in two to three weeks. The ramp speed depends entirely on how specific the brief was before the hire.

SifrVentures builds dedicated engineering teams for tech companies. Based in Berlin. Learn how we work | Read more on our blog

Your AI Feature Track Is Stalling Your Core Product

Hassan — Thu, 09 Apr 2026 05:27:54 +0000

Why launching an AI workstream with your existing team creates two failure modes at once — and what to do instead.

You closed your Series A or B six months ago. The roadmap committed to investors includes an AI-powered product track: an AI agent, an ML recommendation layer, an LLM-backed workflow. Your engineering team is good. You shipped the core product with them. Now they're stretched across two futures simultaneously, and both are moving slower than they should.

This is the most common engineering bottleneck we see at DACH scale-ups right now. It has a name, a cause, and a structural fix.

Why the Same Team Cannot Own Both Tracks

The core product and the AI feature track have fundamentally different engineering rhythms.

Core product work runs on predictability. You have a schema, a deployment cadence, a test suite, SLAs that customers depend on. Engineers managing this track optimize for stability. Breaking changes are expensive. The cost of a wrong migration at 3am is high. Teams working here develop instincts around caution.

AI feature work runs on experimentation. Prompt engineering iterations happen daily. Model providers release new API versions every six weeks. Evaluation pipelines replace unit tests. A feature that "works" at demo quality needs three more weeks of evals before it works reliably in production. Engineers on this track need to move fast, break things in staging, and rebuild. The instincts are opposite.

When you assign the same engineers to both, neither track gets the right instincts. Core product engineers ship the AI feature defensively, adding complexity and slowing iteration. The AI track accrues caution debt. Meanwhile, the core product slips because the senior engineers are context-switching across two incompatible modes.

The DORA State of DevOps research consistently shows that context-switching is not a minor inefficiency. Teams that split attention across two distinct products have deployment frequency that is roughly half that of teams with focused ownership. At 50-200 employees, you cannot absorb that.

What We've Seen

At one client, the AI agent track was staffed by pulling three backend engineers off core product delivery. Within six weeks, two things happened: the AI features shipped with hard-coded model configs instead of proper prompt versioning (because the engineers' mental model was "function, not experiment"), and a core product module that needed a refactor got deferred twice. By month three, the CTO was managing two teams that each felt under-resourced despite having the same total headcount.

The fix was splitting ownership at the team level, not the sprint level. A separate squad took over the AI workstream, with different tooling, different evaluation criteria, and different standups. The core product team stopped context-switching. Within eight weeks, both tracks had clearer velocity.

This pattern holds across the DACH scale-ups we work with. Berlin HealthTech companies launching care coordination AI agents. HR-API companies building AI-powered application flows. Design SaaS companies adding generative image features. The story is the same: net-new AI product, existing team stretched, two tracks bleeding into each other.

The Structural Fix: Separate the Squad, Not the Sprint

The principle is team topology, not sprint planning. Two parallel tracks need two teams with coherent ownership.

The AI workstream squad typically needs:

A backend engineer comfortable with Python, async processing, and working directly with LLM APIs (OpenAI, Anthropic, Gemini). This person writes the prompt management layer, the evaluation harness, the retry logic, and the streaming response handlers.
A data or ML engineer who can build evaluation pipelines, manage dataset versioning (think DVC or Weights and Biases), and interpret evals beyond vibes. At mid-market scale, this person does not need to train models — they need to work with pre-trained models and measure output quality reliably.
Optionally, a second backend engineer if the AI product has significant integration surface (webhooks, API consumers, OAuth flows connecting to third-party SaaS).

The core product team stays intact. They set the contracts the AI squad integrates against: API schemas, event topics, database access patterns. The AI squad treats the core product as a dependency, not a shared codebase.

This separation has a counterintuitive benefit: it forces interface clarity. When the AI squad cannot just reach into shared code, both teams end up with cleaner boundaries. The core API gets documented. Events get proper schemas. The architectural debt that "we'll clean up later" gets flushed out by necessity.

On tooling: the AI squad should own its own deployment path. A separate service, deployed independently, with its own CI pipeline and its own evaluation gate before promotion to production. Use LangSmith, Langfuse, or a homegrown eval harness — the specific choice matters less than having one. If your AI feature has no evaluation pipeline, it is not production-ready regardless of how good it looked in the demo.

For infrastructure, Kubernetes namespaces work well for isolation without separate clusters. Your platform team (or whoever owns your Terraform and Helm charts) adds the AI service namespace to existing infrastructure — typically a half-day of work, not a new greenfield setup.

Key Takeaways

Splitting AI and core product engineering at the sprint level does not solve the underlying context-switch problem. The fix is team ownership, not task allocation.
An AI workstream squad at this stage needs a backend engineer with LLM API experience and a data engineer who can build eval pipelines — not necessarily ML specialists.
Interface contracts forced by team separation improve your core architecture as a side effect. The pressure to define clean APIs and event schemas has long-term value beyond the AI track.
The cost of building this second squad in-house — recruiting, interviewing, onboarding — is 4-6 months on Berlin timelines. Embedding a dedicated squad hired for your stack cuts that to 3-4 weeks.

SifrVentures builds dedicated engineering teams for tech companies. Based in Berlin. Learn how we work | Read more on our blog

Launching a Second Product? Your Engineering Team Can't Build Both.

Hassan — Thu, 02 Apr 2026 05:20:10 +0000

Why shared engineering resources guarantee that your new product track ships late — and what a purpose-built team changes.

You've validated the first product. You have paying customers, a functioning team, and a roadmap your engineers know by heart. Now there's a second product. A new SaaS track. An AI suite. A platform for a vertical you weren't in before. Leadership is aligned, the market timing is right, and you need to ship.

The question is: who builds it?

The Borrowed Engineer Problem

The first answer is always the same. You pull one or two engineers from the core team. Temporarily. Just to get the foundation down, scope the architecture, unblock the first sprint. They know the codebase, they know how you work, and they're available right now.

Temporary rarely ends. Three months later, those engineers are context-switching between two codebases, two roadmaps, and two sets of stakeholder expectations. The core product slows down because they're unavailable for the work only they understand. The new product slows down because they're still on-call for the old one. You've created two half-staffed teams where you needed one focused team.

This isn't a management failure. It's a structural one. Borrowed engineers carry the cognitive cost of the thing they came from. They can't fully own the new product because they haven't left the old one.

Open Headcount Takes Longer Than Your Window

The alternative is to hire. Post the roles, run the pipeline, make the offers. For a three-person engineering team covering frontend, backend, and infrastructure, you're looking at nine to eighteen months of elapsed hiring time if everything goes well. One slow candidate, one declined offer, one extended notice period, and you're past the window you thought you had.

The German market compounds this. Senior engineers in Berlin and Munich face outreach from three or four employers simultaneously. A 2024 analysis of DACH tech hiring found median time-to-hire for senior software roles at 4.2 months, not counting ramp time to first meaningful contribution. By the time your new hires are shipping independently, six months have passed and the competitive dynamics have shifted.

The second product doesn't have six months. It has the urgency that justified building it in the first place.

Independence Is What Makes Small Teams Fast

The reason small teams can outship large ones is focus. A team of four engineers working on one product, one codebase, one set of user problems can move at a pace that a fifty-person team never can. They're not waiting for reviews from people who don't know the context. They're not blocked by decisions made for the other product. They own the outcome completely.

That independence disappears the moment the team is shared. A team that splits attention between two products is optimized for neither. The review cycles lengthen. The context-switching tax compounds. The product that feels secondary to the team becomes secondary in practice, regardless of what the roadmap says.

The second product needs its own team from day one. Not eventually. From the first sprint.

What a Purpose-Built Team Looks Like in Practice

We've built this exact structure for a client. The engagement started with one engineer, specifically hired for that client's stack and that product's requirements. Not pulled from a bench, not rotated from another client. Hired to be part of their team. That engineer embedded into their engineering org, learned the codebase, and started shipping in the first two weeks.

As the product scope expanded, the team expanded with it. Each engineer brought in was hired for the specific gap: a frontend specialist when the UI complexity increased, a data engineer when the pipeline work became the bottleneck. The team that started small is now a complete cross-functional team, fully integrated into the client's engineering org. The second product track they were built for is now the primary delivery engine.

This is the build-to-staff model. The developers are hired for you, not assigned to you. They join your team, use your tools, follow your process, and report into your engineering organization. The difference from contracting is ownership. The difference from hiring is speed: two to four weeks from scoping to first commit, not six months.

The Timing Question

If you're planning a second product track and asking where the engineering capacity comes from, the answer matters more than most structural decisions you'll make this quarter. Borrowed engineers slow both products. Open headcount misses the window. Purpose-built teams can start in weeks.

If your second product has a real timeline and you want to talk through the engineering structure, we're straightforward to reach. A thirty-minute conversation is enough to scope whether this model fits your situation.

SifrVentures builds dedicated engineering teams for tech companies. Based in Berlin. Learn how we work | Read more on our blog