Vibe coding observability: what to check before production

On a Monday morning a team merges an order export written almost entirely by a coding assistant. Tests are green, the review finds nothing odd, the endpoint returns 200. Three weeks later a customer reports missing rows in their file. That is the real question behind vibe coding observability: not « is the AI any good at coding? » but « what proves, in production, that this code does what I think it does? »

I start from that operating problem rather than from a feature list. There are three decisions to make: where to put spans, what to look at in Dynatrace SaaS, and what to demand before a merge. For the vendor’s own take, read its post on runtime; the structure and examples below are mine.

Method: I ran the Python code in the screenshots with opentelemetry-sdk 1.45.2 and show the real output. I did not run the DQL queries or send any data to a Dynatrace environment, so they are marked « not verified ». Everything applies to Dynatrace SaaS; Managed is not covered.

Why vibe coding observability is not a separate discipline

An assistant writes plausible code. It compiles, it passes the tests it wrote itself, it handles the happy path. What it lacks is knowledge of your production: real volumes, dirty data, slow dependencies. That is no moral failing, just a blind spot: nobody gave it that context.

I quote no figure on the quality of generated code, because I have not read a study I could check. The argument holds without one. The more code you accept without writing it line by line, the more the proof must come from execution rather than reading. It is plain SRE hygiene: a measured SLI beats declared confidence.

Do not mix up two jobs. Monitoring the assistant itself (cost, usage, sessions) is covered in monitoring a coding agent. This article is about the code the assistant delivers once it runs in production. For getting started with an AI-assisted IDE, the ThinkRail piece is a different subject.

A textbook case: 200 OK and a quarter of the rows gone

I rebuilt the scenario in twenty lines. The function does what assistants often do: loop over orders, run one query per order to find the customer, and wrap everything in a try/except that swallows any error so the export « never crashes ».

Two classic flaws hide in there. First, one query per order: 1 + 40 queries for 40 rows. Second, the exception is ignored, so the response stays green. Here is the actual run.

• The response claims success, yet 10 orders out of 40 are missing: one customer is absent from the table.
• One HTTP request produced 41 spans: the signature of an N+1.
• The root span stays UNSET, so an SLI computed on it alone would see nothing.
• The 10 child spans carry ERROR and the message: that is where the truth settled.

First decision: instrument the low layer, not the handler

The lesson is architectural. The span recorded the error because the exception crossed the with block in query_one before the calling code silenced it. Had I wrapped only the handler, I would have got a green span.

A practical rule: put spans as close as possible to outgoing calls (database, HTTP, message queue), where generated code cannot hide them by catching everything. OpenTelemetry instrumentation libraries often do this for you; I did not check which ones cover your driver. To know what a span holds, read the anatomy of an OpenTelemetry span.

Second rule: ban except Exception: pass in review unless it logs or counts. That is not purism. It is the line that manufactures invisible incidents.

Sending the telemetry to Dynatrace SaaS

According to the OTLP page of the documentation, the SaaS endpoint is https://{environment-id}.live.dynatrace.com/api/v2/otlp, with /v1/traces for traces. Authentication uses an Authorization: Bearer header and a token with the openpipeline:traces:ingest scope. Transport is HTTP with binary Protobuf: neither gRPC nor JSON. So set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf, or go through a Collector that converts.

If you copy the ID from the browser address bar, drop the .apps from the host name. Keep the token in a vault, never in the assistant’s prompt or in the repository. For a complete Collector setup, see the OpenTelemetry Collector to Dynatrace.

I did not test this ingestion end to end. Nor do I know how your contract bills this data: not verified, check your licence.

Second decision: three questions for your traces

Once spans arrive, do not search for « everything ». Ask three questions, in this order.

1. How many spans per trace? A simple request that counts dozens points to an N+1 or a runaway loop.
2. Which child spans fail while the root is green? That is the pattern of our export.
3. Which operations changed duration after the last deploy? Regenerated code can swap an index lookup for a full scan without any test noticing.

For the first, a DQL query to adapt (not verified on a real environment):

fetch spans, from: now()-2h
| summarize spans = count(), by: {trace.id}
| sort spans desc
| limit 10

For the second, exception details live in span events: the query taken from the documentation is described in OpenTelemetry exceptions in traces. Then compare the result with the root’s error rate: the gap measures what your code hides.

The vendor post also mentions tooling that brings runtime context into the IDE through MCP, and inspecting a running application without redeploying. I have not tried either here: read the Dynatrace SaaS MCP server setup before promising anything to your team.

Third decision: a merge gate, not a permanent audit

A check that costs nothing gets done. Here is a minimal gate to adapt:

• Every function calling a database or remote API is covered by a span, named after the operation rather than the file.
• No broad except without a counter or a log carrying the trace ID.
• A short load test on realistic data, with a ceiling on spans per request.
• An SLI defined before the merge (share of rows exported, p95 latency), however crude.

The last point pays best. In our export, a simple « rows expected versus rows written » counter would have shown 75% success on day one.

What can go wrong

• An assistant can also delete or weaken your spans while « tidying » the code: lock them with a test.
• Instrumenting everything produces a lot of data; sample if needed, knowing it erases some rare cases.
• Exception messages can contain personal data: review them before they leave for a SaaS. See also vulnerable AI services for the security angle.
• My example is deliberately caricatured: real code rarely has every flaw at once, but it often has two.

Good practice

• Instrument outgoing calls before you write the first feature prompt.
• Always compare the root’s status with its children’s.
• Measure a business outcome, not only the HTTP code.
• Keep the human review: observability detects, it does not replace reading.

Frequently asked questions

Does vibe coding observability need a specific tool?

No: it uses the same OpenTelemetry spans, metrics and logs as any code. What changes is the discipline: you assume the accepted code was not read line by line.

Should Dynatrace tell generated code from hand-written code?

I found no native field for that. If you want to track it, add a resource attribute or a deployment tag you control: it is a team convention, not a verified feature.

Does this work on Dynatrace Managed?

I only read the SaaS documentation (OTLP endpoint, token scopes). For Managed, check the matching page before reusing these settings.

Conclusion

Generated code is judged by how it runs: instrument the low layer, compare root and children, demand an SLI before the merge. Start with one service, measure spans per request, and see what execution teaches you that the review did not show.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Brands mentioned belong to their owners. Tools change fast: check commands and options against the official documentation.

---

Originally published on span.observer. French version: https://span.observer/fr/vibe-coding-observabilite-dynatrace/.