DEV Community: Zainab saif

Your AI Agent Works. Now Try Breaking It.

Zainab saif — Mon, 07 Sep 2026 07:56:54 +0000

A practical engineering guide to testing AI agents before real users, real data, and real permissions do it for you.

The agent passed the demo.

It understood the prompt, chose the expected tool, returned the right answer, and everything looked fine.

Then someone gave it an incomplete request.

Another user asked for something outside its permissions.

An API timed out halfway through a workflow.

The retrieval system returned stale information.

And suddenly the same agent that looked reliable in testing started making decisions nobody had tested for.

That is the problem with testing AI agents like traditional software.

A normal application usually follows a path that engineers define.

An AI agent can decide what to do next.

That changes what we need to test.

Don't test an AI agent only to prove that it works. Test it to discover how it fails.

The Demo Trap

A typical agent demo looks something like this:

User request
     ↓
   Agent
     ↓
Choose tool
     ↓
Call API
     ↓
Return result

Everything is controlled.

The input is known. The data is available. The API works. The expected result is clear.

Production is different.

Unexpected input
       ↓
     Agent
       ↓
  ┌────┴─────┐
  ↓          ↓
Wrong      Correct
decision   decision
  ↓          ↓
Tool       Tool
failure?   success
  ↓
Unexpected outcome

The difficult part isn't getting the agent through the happy path.

The difficult part is discovering what happens when something goes wrong at every step.

For an agent, the failure might not even be an obvious crash.

It could:

choose the wrong tool
use incomplete information
interpret an ambiguous request incorrectly
retry an operation that shouldn't be retried
take an action without enough authorization
produce a confident answer when it should ask a question

So instead of asking:

“Does the agent work?”

start asking:

“What happens when I deliberately give it a situation it wasn't expecting?”

That's where agent testing gets interesting.

1. Break the Input

Start with the easiest thing to attack: the user's request.

Developers naturally test clean inputs:

Find my latest invoice.

But users don't always communicate like test cases.

They might say:

Find the invoice.

Or:

Can you handle the old invoices?

Or:

Find my latest invoice and remove the old ones.

Or they might provide conflicting instructions:

Don't change anything.
Actually, delete the old invoices.

These requests aren't equivalent.

Your agent needs to distinguish between:

clear instructions
ambiguous instructions
incomplete instructions
conflicting instructions
unauthorized instructions
potentially malicious instructions

A simple input test matrix

Input	What you're testing	Expected behavior
Clear request	Normal execution	Complete the task
Empty request	Missing intent	Ask for clarification
Ambiguous request	Unclear scope	Ask a question
Conflicting request	Instruction conflict	Resolve safely
Unsupported request	Capability boundary	Explain limitation
Malicious instruction	Instruction safety	Refuse or safely redirect

Consider:

“Delete the old invoices.”

What does old mean?

Older than 30 days?

A year?

Invoices already paid?

Invoices belonging to a particular customer?

If the agent decides the meaning itself and immediately calls a deletion tool, the problem isn't necessarily the model.

The problem is the system's definition of acceptable uncertainty.

A useful rule is:

If the cost of guessing is high, ambiguity should become a question—not an action.

2. Break the Tools

An agent can make the right decision and still fail because the tool it depends on fails.

Imagine your agent has access to:

search_customer()
get_invoice()
create_invoice()
delete_invoice()
send_email()

Your first test might be:

Agent → Tool → Successful response

That's not enough.

Test what happens when the tool:

times out
returns an empty response
returns malformed data
rejects authentication
rejects authorization
returns a server error
receives invalid parameters
becomes temporarily unavailable

For example:

Agent
  ↓
get_invoice()
  ↓
API timeout
  ↓
What happens now?

A weak implementation might retry blindly.

A better implementation might:

recognize the timeout
retry only when the operation is safe to retry
stop after a defined limit
explain that the requested information is temporarily unavailable
avoid inventing a result

The important distinction is between tool failure and model failure.

If the API didn't return the invoice, the agent shouldn't manufacture one.

Test the boundary between reasoning and execution

For every tool, ask:

“What should the agent do when this tool doesn't behave as expected?”

Write that behavior down before production.

For example:

Tool failure
     ↓
Retry?
 ┌───┴────┐
Yes       No
 ↓         ↓
Safe?    Explain
 ↓
Retry
 ↓
Still failing?
 ↓
Stop + report

This turns an unpredictable failure into an engineered behavior.

3. Break the Context

An agent can have the right model and the right tools and still make the wrong decision because its context is wrong.

This becomes especially important when an agent uses retrieval, databases, memory, documents, or external APIs.

Try testing these situations:

Stale data

The agent retrieves information that is technically valid but no longer current.

Conflicting data

Two sources contain different values.

Database → Customer status: Active
Document → Customer status: Suspended

What does the agent do?

Missing information

The answer simply isn't available in the retrieved context.

Irrelevant retrieval

The system returns documents that contain similar words but don't answer the actual question.

Context overload

Too much information is provided, and the important detail gets buried.

These cases expose an important question:

Does your agent know when its context isn't enough to act confidently?

That's different from asking whether your retrieval system can return documents.

A retrieval system can return something and still give the agent insufficient evidence.

Your tests should therefore measure more than retrieval success.

Test whether the agent:

recognizes missing information
distinguishes relevant from irrelevant context
handles conflicting sources
avoids treating stale information as authoritative
asks for clarification when necessary

A good agent isn't the one that always produces an answer.

Sometimes the correct result is:

“I don't have enough information to safely do that.”

4. Break the Model

Now attack the reasoning layer.

This doesn't mean trying to prove whether the model is “smart.”

Instead, test the decisions the model makes inside your workflow.

For example:

User request
     ↓
Agent reasoning
     ↓
Choose tool
     ↓
Choose parameters
     ↓
Execute

There are several places where things can go wrong.

Wrong tool selection

The user asks for customer information, but the agent calls an unrelated search tool.

Incorrect parameters

The correct tool is selected, but the agent sends the wrong customer ID or date range.

Hallucinated information

The agent fills a missing value instead of acknowledging that it doesn't have one.

Inconsistent decisions

The same situation produces different actions under slightly different wording.

Multi-step failure

The first step succeeds, but the agent makes a bad decision based on the result.

For example:

Find customer
     ↓
Customer found
     ↓
Check account status
     ↓
Account suspended
     ↓
Agent continues anyway
     ↓
Send confirmation

The failure isn't necessarily the final response.

It happened earlier in the decision chain.

That's why agent testing should capture what the agent decided to do, not just what it eventually said.

5. Test the Permission Boundary

This is where an agent moves from “interesting software” to something that can create real operational risk.

Not every action should have the same level of freedom.

Consider three categories:

READ
 ↓
Low-risk information retrieval

WRITE
 ↓
Create or modify something
 ↓
Validation required

DELETE / FINANCIAL / EXTERNAL ACTION
 ↓
Potentially irreversible
 ↓
Human approval

An agent might safely retrieve an invoice automatically.

That doesn't mean it should automatically delete one.

Likewise, sending an email, changing account information, issuing a refund, or modifying production data may require a stronger control.

Ask three questions for every tool

1. What can this tool read?

2. What can this tool change?

3. What can this tool do that cannot easily be undone?

Then test each boundary.

For example:

User:
"Delete all invoices from last year."

        ↓

Agent identifies delete action

        ↓

Is authorization sufficient?
        ↓
      NO
        ↓
Ask for confirmation / human approval
        ↓
Only then execute

The key design principle is simple:

The more consequential the action, the less you should rely on the model's judgment alone.

This doesn't mean removing autonomy from every agent.

It means matching autonomy to risk.

6. Make Failure Observable

Finding a failure is only useful if your team can understand why it happened.

Imagine a user reports:

“The agent sent the wrong email.”

Can you answer:

What did the user ask?
What context did the agent receive?
What tools did it call?
What parameters did it send?
What did those tools return?
Which decision led to the email?
How long did each step take?
Did the agent retry anything?
Was approval required?
What version of the prompt or workflow was running?

If the answer is simply:

“The model generated the wrong response.”

you probably don't have enough observability.

At minimum, agent workflows should make important execution details inspectable.

That can include:

Request
  ↓
Retrieved context
  ↓
Model decision
  ↓
Tool call
  ↓
Tool response
  ↓
Next decision
  ↓
Final output

Depending on the system, useful telemetry can include:

execution logs
traces
tool-call history
retrieval results
latency
token usage
errors
evaluation results
approval events

The exact implementation will vary.

The principle doesn't:

If you can't reconstruct the failure, you can't reliably fix it.

7. Turn Failures Into Tests

This is the part that can change how your team approaches agent reliability.

Suppose a production incident happens.

The easy response is:

Fix bug → deploy → move on

Instead:

Production failure
       ↓
Capture the scenario
       ↓
Understand why it failed
       ↓
Create a regression test
       ↓
Fix the system
       ↓
Run the test again

Imagine an agent called a delete tool without sufficient approval.

Turn that incident into a permanent test:

Test: Unauthorized deletion

Input

Delete all invoices.

Expected behavior

The agent should not execute the deletion without the required authorization or approval.

Failure condition

delete_invoice()

is called before the required approval exists.

Now the next release has a concrete test for something that previously happened only in production.

Over time, your test suite becomes a record of the ways your agent has already failed.

That's valuable because agent behavior can change when you modify:

prompts
models
tools
retrieval logic
system instructions
permissions
workflows

A change that fixes one scenario can unintentionally affect another.

Regression tests give you a way to catch that.

Build a Break-Your-Agent Test Suite

You don't need hundreds of tests on day one.

Start by attacking the major failure surfaces.

Input

Can the agent handle an empty request?
What happens with an ambiguous request?
What happens with conflicting instructions?
What happens with unsupported requests?
What happens with adversarial input?

Context

What happens when information is missing?
What happens when sources disagree?
What happens when retrieved information is stale?
What happens when irrelevant documents are returned?
What happens when the context becomes too large?

Tools

What happens when an API times out?
What happens when parameters are invalid?
What happens when authorization fails?
What happens when the response is malformed?
What happens when the tool becomes unavailable?

Model decisions

Can it choose the wrong tool?
Can it invent missing information?
Can it make inconsistent decisions?
Can it continue after a failed step?
Can it misunderstand the result of a previous tool call?

Safety

Can it read data it shouldn't?
Can it modify data without validation?
Can it delete data without approval?
Can it perform financial actions without the required controls?
Can it trigger external actions without authorization?

Operations

Can you trace an individual execution?
Can you identify failed tool calls?
Can you inspect retrieved context?
Can you reproduce important failures?
Are production failures converted into regression tests?

You don't need to predict every possible failure.

You need a process that gets better at discovering them.

Before → During → After Production

Agent testing shouldn't stop when the application ships.

Think about it as three stages.

BEFORE PRODUCTION
       ↓
Break inputs
Break tools
Break context
Test permissions
Test failure handling
       ↓
DURING PRODUCTION
       ↓
Observe executions
Trace failures
Monitor behavior
Collect evaluation data
       ↓
AFTER A FAILURE
       ↓
Capture scenario
Understand failure
Create regression test
Fix
Retest

This creates a feedback loop.

The production environment shouldn't simply be where your users discover your missing test cases.

It should also help your engineering team build better ones.

The Engineering Principle

AI agents introduce an uncomfortable reality for developers:

You can't enumerate every path an agent might take.

But you can design the system so that unexpected behavior is:

detectable
observable
constrained
recoverable
testable

That's a much more useful goal than pretending the agent will always behave perfectly.

A production-ready agent isn't one that never fails.

It's one where the important failure modes have been anticipated, risky actions are bounded, failures are visible, and new failures become tests instead of recurring surprises.

So before you give your agent access to real users, real data, or real permissions, try something uncomfortable.

Try to break it.

Give it bad input.

Take away its tools.

Return incomplete context.

Make the API fail.

Create conflicting information.

Ask it to perform an action it shouldn't be allowed to perform.

Then watch what it does.

Because the most important test isn't:

“Can my AI agent complete the task?”

It's:

“What does my AI agent do when the task doesn't go according to plan?”

And every time you find an answer, turn it into a test.

Don't test whether it works. Test how it fails.

A question for developers

If you were trying to break your own AI agent tomorrow, what would you test first?

Building Real Time AI Systems: What Changes When Computer Vision Meets Production Software

Zainab saif — Thu, 13 Aug 2026 14:31:38 +0000

A computer vision prototype is straightforward to build: point a model at a video feed, run inference on a frame, return a set of detections. It demos well. Running the same system continuously, at scale, with real users depending on its output, surfaces a different category of problem — one that has comparatively little to do with the model itself.

This article examines that second phase: what changes, architecturally and operationally, when a computer vision model moves from a research artifact to one component inside a production system. It draws on the operational realities of running real-time inference pipelines, the parts of the work that rarely make it into a model card or a conference talk: backpressure, partial failure handling, output-quality drift, model rollout strategy, and the persistent gap between "the model works" and "the system is production-ready.

Where the architecture actually changes

A prototype usually looks like this:

camera / video file → model → result

That's fine for a notebook. It falls apart the moment the input is continuous and the output needs to go somewhere useful. A production pipeline looks closer to what's shown below: video enters through an ingestion layer, moves through preprocessing and inference, gets validated before it reaches application logic, and every stage feeds telemetry back into monitoring.

The important shift is conceptual: the model becomes a service inside a larger system, not the system itself. Once you accept that framing, a lot of the "AI engineering" problem turns into fairly conventional distributed systems work — queues, retries, backpressure, observability , applied to a workload that happens to include a neural network.

Designing the real time pipeline

Ingestion is usually the first place teams underestimate the work. Video sources are messy: variable frame rates, dropped connections, inconsistent resolutions, occasional corrupt frames. Before any frame reaches the model, the pipeline needs to normalize all of that — decode reliably, handle reconnections, and discard frames that fail basic sanity checks (wrong dimensions, all-black frames, decode errors).

Not every frame needs to reach the model. For many use cases — occupancy counting, motion-triggered detection, or analytics on digital signage and out-of-home displays — sampling at a lower rate than the source video, or skipping frames when the scene hasn't meaningfully changed, reduces compute cost without a meaningful loss in accuracy. This decision is worth making deliberately, since it affects nearly everything downstream: queue sizing, GPU provisioning, and per-stream cost. It's a pattern that shows up consistently in production computer-vision analytics work, including systems built by teams such as Macromodule's AI/ML engineering group, where the ingestion layer is treated as a design decision in its own right rather than a default.

A complete, runnable frame-sampling gate:

**import time
from dataclasses import dataclass

CHANGE_THRESHOLD = 5.0  # tune per use case; depends on frame representation

@dataclass
class LastFrame:
    timestamp: float  # milliseconds, from time.monotonic() * 1000
    data: list        # simplified representation; use a real diff metric in practice

def frame_difference(current, previous) -> float:
    """Mean absolute difference between two same-length frame summaries.
    In practice, replace this with a proper metric: pixel-level diff,
    histogram distance, or a lightweight motion-detection heuristic —
    not the full model.
    """
    return sum(abs(a - b) for a, b in zip(current, previous)) / len(current)

def should_process_frame(frame, last_processed_frame: LastFrame, min_interval_ms=100) -> bool:
    now = time.monotonic() * 1000
    if now - last_processed_frame.timestamp < min_interval_ms:
        return False
    if frame_difference(frame, last_processed_frame.data) < CHANGE_THRESHOLD:
        return False
    return True

plaintext

The logic itself is simple; the decision of what threshold and what interval to use belongs to product requirements, not to the model, and is worth pinning down explicitly rather than left as a framework default. Note that frame_difference here is a placeholder — a real implementation should use a proper motion or histogram-based metric rather than comparing raw pixel arrays, which is both slow and noisy.

Latency is more than inference time

Teams often benchmark a model in isolation — "inference takes 15ms" — and treat that as the system's latency budget. It isn't. End-to-end latency is the sum of several stages, most of which have nothing to do with the model:

A model that runs in 15ms can easily sit inside a pipeline that takes 400ms end to end, because preprocessing is unbatched, the database write is synchronous, or the result has to round-trip through an API gateway. This is the single most common gap between "the model is fast" and "the product feels slow," and it's worth instrumenting each stage separately — with timestamps recorded at each boundary — rather than reporting one aggregate number that hides where the time actually goes.

Throughput, concurrency, and backpressure

A single camera stream is a manageable engineering problem. Ten concurrent streams, or a hundred, introduce a different class of problem: what happens when frames arrive faster than the pipeline can process them?

Left unhandled, this shows up as an ever-growing queue, increasing memory use, and eventually stale results — detections computed on frames that are seconds old and no longer represent the current scene. For most real-time use cases, an old detection is often worse than no detection.

The standard approaches apply here, and they're worth taking seriously rather than treating as an afterthought:

Bounded queues with drop policies. Cap queue depth and drop the oldest frames when full, rather than letting memory grow unbounded.
Worker pools sized to actual throughput, not to peak concurrency — oversized pools just contend for the same GPU.
Backpressure signals fed back to the ingestion layer, so upstream producers slow down instead of silently overwhelming the pipeline.

import asyncio

class BoundedFrameQueue:
    """A queue that drops the oldest frame instead of blocking or growing
    unbounded when the pipeline falls behind."""

    def __init__(self, max_size: int = 50):
        self.queue: asyncio.Queue = asyncio.Queue(maxsize=max_size)

    async def put(self, frame) -> None:
        if self.queue.full():
            # Drop the oldest frame rather than blocking ingestion.
            # Safe here because there's no await between the check
            # and the drop, so no other task can interleave.
            _ = self.queue.get_nowait()
        await self.queue.put(frame)

    async def get(self):
        return await self.queue.get()

It's a small pattern, but the alternative — an unbounded queue — is one of the more common root causes of a real-time system degrading gradually into an outage rather than failing loudly and immediately.

What happens when AI or dependent services fail

A model can return an empty result, a malformed response, or simply time out. An external API it depends on can go down. None of this is exotic; it's the normal operating condition of any networked service, and computer vision pipelines are no exception.

Worth handling explicitly:

inference timeouts
malformed or unexpected model output
downstream API failures
transient network errors
degraded input (corrupted frame, unsupported format)

import asyncio

class InferenceError(Exception):
    """Raised by the model client on a known, non-retryable failure."""

async def run_inference_with_retry(
    model,
    frame,
    max_retries: int = 2,
    timeout_s: float = 1.0,
) -> dict:
    """Runs inference with a timeout and bounded exponential backoff.
    Returns a degraded/error result instead of raising, so callers
    always get a well-formed response to work with.
    """
    for attempt in range(max_retries + 1):
        try:
            return await asyncio.wait_for(model.infer(frame), timeout=timeout_s)
        except asyncio.TimeoutError:
            if attempt == max_retries:
                return {"status": "degraded", "detections": []}
            await asyncio.sleep(0.1 * (2 ** attempt))
        except InferenceError:
            return {"status": "error", "detections": []}

The specific numbers matter less than the principle: a real-time system should have an explicit, tested answer for "the model didn't respond in time," rather than letting that case propagate as an unhandled exception three layers up the stack. This is also where a circuit breaker earns its keep — if a downstream service is consistently timing out, retrying every request just adds load to an already struggling dependency. Failing fast for a cool-down period, then probing occasionally to see if it's recovered, is usually the better trade-off.

Validating output before it reaches the application

It's easy to treat model output as trustworthy simply because it came from the model. In production, output needs the same skepticism as any other untrusted input:

confidence thresholds appropriate to the use case, not a default value copied from a tutorial
schema validation on the response shape
sanity checks on bounding boxes (in-frame, non-degenerate dimensions)
deduplication of overlapping detections
checks for classes the application doesn't expect
temporal consistency checks — a detection that flickers in and out frame to frame is often noise, not a real event

def validate_detection(det: dict, frame_width: int, frame_height: int, min_confidence: float = 0.5) -> bool:
    """Returns True only if the detection passes basic sanity checks.
    Expects det = {"confidence": float, "bbox": [x1, y1, x2, y2]}.
    """
    if det["confidence"] < min_confidence:
        return False

    x1, y1, x2, y2 = det["bbox"]
    if x2 <= x1 or y2 <= y1:
        return False
    if x1 < 0 or y1 < 0 or x2 > frame_width or y2 > frame_height:
        return False

    return True

This step is easy to omit during development, since it doesn't affect whether a demo runs successfully. It's typically the first layer that matters once real, unfiltered production input starts flowing through the system.

Testing and rolling out model changes safely

Testing a computer vision system is not the same problem as testing a typical backend service, because the "correctness" of a model's output is probabilistic rather than deterministic. A few practices that hold up in production:

Golden datasets. Maintain a fixed, versioned set of representative frames — including edge cases like poor lighting, occlusion, and unusual angles — and run every model candidate against it before deployment. This catches regressions that a single accuracy number can hide.
Shadow deployment. Run a new model version alongside the current one on live traffic, log both sets of predictions, but only serve the current model's output to the application. Compare divergence before cutting over.
Canary rollout. Route a small percentage of streams to the new model version, watch the output-quality metrics described below, and expand gradually rather than switching all traffic at once.
Version everything together. Model weights, preprocessing code, and postprocessing thresholds should be versioned as a unit. A model upgraded without its matching preprocessing changes is a common, hard-to-diagnose source of silent accuracy loss.

None of this requires elaborate infrastructure to start — even a simple script that reruns the golden dataset and diffs the output against the previous version, run as a pre-deployment check, catches a meaningful share of regressions before they reach production traffic.

Monitoring a computer vision system

Standard infrastructure monitoring — CPU, memory, HTTP error rates — tells you whether the servers are healthy. It tells you almost nothing about whether the system is doing its job correctly. A computer vision pipeline needs a second layer of monitoring focused on the workload itself:

Category	Example metrics
Pipeline health	inference latency, end-to-end latency, queue depth, dropped frames
Throughput	frames processed per second, per-stream processing rate
Reliability	API/model error rate, timeout rate, retry rate
Output quality	detection rate over time, confidence distribution, anomalous class frequency
Resources	GPU/CPU utilization, memory, cost per stream

The output-quality category is the one teams most often skip, and it's the one that catches real problems: a camera that's drifted out of position, lighting conditions the model wasn't trained on, or a silent model regression after a deployment. None of those trigger an infrastructure alert — the servers are healthy, the API returns 200s, and the system is quietly producing wrong answers. A practical baseline is to alert on sudden shifts in detection rate or confidence distribution relative to a rolling historical average, not just on hard thresholds, since "normal" varies by time of day, camera, and scene.

Data retention, privacy, and access control

Video pipelines carry more regulatory and security weight than most backend services, and it's worth treating this as a first-class design concern rather than an afterthought bolted on before a compliance review:

Retention policy. Decide, before launch, how long raw frames, derived detections, and any identifying data are kept, and enforce it programmatically rather than manually. Raw video is usually the most sensitive and least necessary to retain long-term — in many pipelines, only the derived detections need to persist past a short debugging window.
Access control on the inference API, not just the surrounding application. An inference endpoint that accepts arbitrary uploaded frames without authentication is a real exposure, especially if it's compute-expensive and internet-reachable.
Data minimization. Where the use case allows it (occupancy counts, aggregate analytics), storing counts or bounding-box metadata rather than raw imagery reduces both storage cost and privacy exposure.
Jurisdiction-specific rules. Video analytics involving people frequently intersects with biometric and surveillance regulations that vary meaningfully by region — this is worth a specific legal review rather than a generic privacy policy, particularly for anything resembling facial recognition or persistent identity tracking.

Scaling from one stream to a hundred

The jump from one camera to ten is mostly a capacity question. The jump from ten to a hundred usually forces architectural changes: workload distribution across GPU workers, queue partitioning per stream or per region, and storage/bandwidth costs that scale linearly with stream count in a way that's easy to underestimate early on.

It's worth resisting the temptation to name specific infrastructure (a particular message broker, orchestration platform, or cloud service) unless it's actually in use — the underlying decisions (how work is distributed, how failures are isolated, how state is partitioned) matter more than the specific tools, and the right tools vary a lot by scale and existing infrastructure.

Cost tends to grow in three places that are easy to overlook during initial design: GPU idle time from poorly batched or unevenly distributed inference requests, egress and storage cost from retaining more raw video than the use case actually requires, and the operational cost of running enough redundant capacity to tolerate a single worker or region failure without dropping streams.

A practical readiness checklist

Before calling a real-time computer vision system production-ready, it's worth having explicit, testable answers — not just intentions — for each of these:

What's the actual end-to-end latency budget, measured stage by stage?
What happens when the model times out or returns malformed output?
What's the behavior under sustained overload — degrade gracefully, or fall over?
Is output quality monitored separately from infrastructure health?
Is there a tested rollout process for new model versions, including a rollback path?
What's the data retention policy, and is it enforced automatically?
What's the plan for scaling stream count, and where does the current architecture stop working?
Are security and access controls applied to both the video input and the inference API, not just the surrounding application?
What's the actual cost per stream at current and projected scale?

If any of these don't have a concrete answer, that's the gap to close before launch, not after.

Closing thought

None of this is specific to a single industry or use case, and that's largely the point. Whether the pipeline handles occupancy counting, defect detection, or analytics on out-of-home advertising displays — the class of problem behind platforms like Oohlytics, Macromodule's computer-vision analytics product for billboard and signage measurement — the underlying engineering challenge holds steady: a model that performs well in isolation is not the same thing as a system that performs reliably under continuous, real-world load. The model is frequently the more tractable part of the problem. The surrounding pipeline — ingestion, validation, monitoring, rollout, and data governance — is where the engineering effort actually goes, and where the difference between a demo and a product gets decided.