DEV Community: Raluca Crisan

What Is My Agent Actually Doing?

Raluca Crisan — Fri, 14 Aug 2026 14:40:40 +0000

Visualization tends to get treated as the afterthought of the observability stack. I'd half agree with that, except that the visualization is also what people seemed to like most about my last post.

So here's a condensed version of that blogpost, focused on the pictures only. The question I actually want answered mid-run is a boring one: what is this thing doing right now, and can I trust the bit it just finished? There are a few different layers that claim to answer it. One layer is formed of the pipeline source, the inputs, the outputs. LLM-call observability gives you the prompts, the tool calls, the spans, the token counts - all very useful, but still leaving a gap. Code execution observability is the third thing: mapping the code as it runs and tracing artifacts and their lineage deterministically, without manual instrumentation (a bit like extreme auto-logging). What comes out is intermediate data states, models, agent states, each traced back to the function that produced/transformed it; this would amount to what the executed code actually did, which intermediate step fed which result, and potentially (paired with a few other elements) where an error entered the workflow.
Below is the visualization of a code execution graph for a agent: the agent is moving back and forth through the graph, resolving nodes. At the start, the job has issues throughout every section:

Then, as the agent (in this case codex) works through the graph nested function by nested function, some of them start clearing:

You can open a node if you want more information:

The visualization can be vastly improved, but hopefully it paints a picture. It can help with debugging and understanding what happened afterwards, but also, for any regulated sector, it is pretty useful as a deterministic account of what was executed by your agent.

Finally, Observability Can Do More Than Watch: How Observability Can Help Agents Be Better

Raluca Crisan — Wed, 12 Aug 2026 13:08:32 +0000

Summary

What is it:
Imagine you have an agent that has a lot of different jobs to do (primarily in code) and the jobs are usually sequential and dependent on each other; it potentially has loads of subagents that it spawns itself, and it has to do an end-to-end job (like market research - > coverage -> distribution -> reporting, etc.). Instead of this agent usually just relying on semantic memory, outputs, and code to pass the job from one phase to the next, it uses a code execution graph to see where in the job it actually has an issue, and repair from there/resume from a trusted boundary of the graph.

Has it been done:
Not really, some substitutes for the graph creation were attempted. Or maybe it's been done, just not published!

Why not:
Because it’s hard to make this graph, then you are still left with questions around how does your agent operate on the graph and how does it define/assess a trusted boundary.

Why does this matter:
Because these agents go wrong a lot and you can’t expect them to cope with all the context they produce. But if you do get a long horizon agent right - autonomy is right around the corner, plus this gives you governance as well.

Long Version

If you’ve read any of my blog posts before then you’ll know this series is solely focused on code execution graphs (Etiq’s in particular) and what they are good for. In this part, I’ll handle a very interesting use case: long horizon agents, or agents that have to do more and as a result struggle more - with context, with keeping track of memory, shared tasks, etc. As a disclaimer: for the experiment side, I haven’t gotten as far as a full long horizon agent partly because on less complex agents the dynamic between the different issues at play is easier to see, but initial findings on a simpler agent are definitely pointing in the right direction (also full disclaimer: this blogpost is not attempting to pretend that it comes from a long horizon expert, my interest is from the point of view of impact of observability on these and other complex agents).

The main idea is that using an execution graph that builds across all the code/pipelines coded and run by a ‘long horizon’ (or similarly complex) agent will help the agent keep track of its steps better and orchestrate handoff points and collaboration tasks more easily.

These agents often suffer from errors that compound - they might make an error early on in a step and then not realise till they’re close to the end of the run by which point it might be too late and a lot of work would be wasted. Moreover, the agent might not then even learn from what happened either, as the context has ballooned too much for it to tell the root cause of the issue, and thus truly waste a run. With the execution graph, as the agents execute their code via the runtime capture, the graphs of different pipelines run by different spawns can get collated together and build a large map of what happened, where some regions/boundaries can be labelled as trusted or not trusted. Then the agent can use this map to orient itself during the many steps it needs to undergo - with some more specific operations underneath. If some boundaries can be defined as trusted that makes it easier for the agent to retrace its steps, or resume from a known trusted boundary.

The other key element this graph could potentially tackle is the context issues these agents end up having (and I’m still exploring how exactly that would happen) - here the assumption is that if only the LLM were given an optimal amount of context it would answer perfectly every time, thus the key to a great long horizon or any kind of complex agent is parcelling out the tasks in a way which encourages this optimal allocation. There are elements of this approach in this initial experiment, namely: splitting the execution graph into smaller regions and having the agent review or repair each of the regions in an independent session with in-depth context only about the relevant region and high level knowledge about the rest of the task - so the agent is given the optimal/relevant context only to help it achieve a good outcome. From a human perspective, it is helpful to have a trace of what the agent has done in code (especially if the trace is deterministic), - and go beyond the usual OpenTelemetry observability and semantic memory, as it can provide a layer of governance.

I have yet to master the art of writing captivating lit reviews (and chatGPT hasn’t mastered it yet either), but really there are some indications that structured, execution-aware state for long-horizon agents is not the craziest idea.

The MAGE paper is probably the best-known paper in this general area, and here’s a short AI generated description: MAGE treats memory as an evolving execution state rather than a collection of semantically similar fragments. It organises raw action- observation traces and compressed subgoal summaries into a hierarchical tree, allowing the agent to validate progress, return to an earlier boundary after an error and continue along a repaired branch. On MemoryArena, it improved success rates by 7.8–20.4 percentage points while using 55.1% fewer tokens than retaining the full history. [1]

The main difference with an execution graph is that MAGE is based on textual traces and LLM-generated judgements rather than an execution graph, but the operations on the graph can be very similar [1]. As we’ll see during the experiment, choosing too strictly the area on which the LLM applies the judgement is a double-edged sword… on the one hand it might be a good idea to limit it based on a real execution graph, on the other hand LLMs are just very good on their own… but this is part of the crux of this problem.

SWE-Search is also a good model for using trajectories and different agent paths and good inspiration for the kind of different operations that can potentially be applied on a graph for a coding agent… but of course, it is very different from an execution graph [2].

Zhaoling Chen et al. - LocAgent: Graph-Guided LLM Agents for Code Localization shows that graph structure can improve software agents’ navigation. Its graph is predominantly derived through static analysis, not dynamic [3]. A survey paper Yiqi Wang et al.: From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agents describes the landscape a bit, pretty aligned I would say with the notion that the tooling landscape for observability is a bit fragmented and not quite fit for purpose at the moment and that a new generation of this tooling would be welcome [4].

Some newer, less cited papers are close to the topic - let’s call it emerging literature. Yohei Nakajima, The Log Is the Agent: Event-Sourced Reactive Graphs for Auditable, Forkable Agentic Systems has a different type of graph: agent-level events vs. the execution graph [5]. Josh Rosen and Seth Rosen, From Agent Loops to Deterministic Graphs: Execution Lineage for Reproducible AI-Native Work models AI-assisted work as a directed acyclic graph of artifact-producing computations with explicit dependencies, stable intermediate boundaries and identity-based replay. This is very closely tied to the idea of using graphs/dags for these architectures, but the paper assumes that the computation graph is specified correctly, rather than showcasing an actual retrieval mechanic at play; the level of the graph is also probably slightly different [6].

A somewhat more distant paper: Lakshya Agrawal et al. - GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning is a kind of agent prompt optimiser based on recorded trajectories - including reasoning, tool calls, outputs and evaluator feedback. In theory it could be combined with an execution graph [7].

Agent architecture

The initial agent architecture involves a series of codex jobs and it is NOT yet a long horizon agent. This is a marketing agent and it includes jobs like

1) ‘segment’ - which takes some initial instruction on product and audience and returns some segments

2) ‘market_demand’ - the actual Python pipeline that will collect and synthesise market-demand evidence for one segment

3) ‘coverage’ - the Python pipeline that maps trusted demand against product capabilities and determines coverage gaps,

4) ‘synthesizer’ - gathering the final result if enough coverage for the segment exists.

Each job has a fresh session and there are a few sub-jobs that are execution/graph dependent such as review, repair, etc. (for more details see the appendix).

The ideal version is a fuller sequence, building up to an ‘end-to-end marketing agent’ to include distribution (actual campaigns) and measurement steps, with codex as the orchestrator and worker agents that are OSS agents with pydantic and aider harnesses. And the end-to-end process should be fully autonomous including when it comes to distribution. But this experiment is starting light.

The execution graph

As a quick reminder: the Etiq scanner provides runtime observability for agent-, copilot- and human-created code, supporting Python, Spark and SQL across data pipelines, ML/AI workflows, agent builds and larger codebases. Starting from a Python entry file, it scans the code as it runs and builds a deterministic causal-lineage graph from actual execution artifacts rather than LLM-generated claims. The scanner captures datasets, dataframes, models, agent states and other intermediate outputs, tracing each object back to the functions that produced or transformed it. This reveals what the executed code actually did, which intermediate step contributed to a result and where an error may have entered the workflow. The graph is generated without manual or agent instrumentation.

The initial job just gets passed the high level prompt - as it is a planning job it does not run through the scanner. Once codex codes up the plan, every code that is run is run through the scanner, which captures the nodes - the interim and input/output data states, model and LLM calls, and the relationships between them. In essence the scanner pretty much provides the execution graph out of the box and all that is left to do is link the various graphs of the various executed pipelines together which is not very hard to do as they share input/output nodes.

Now comes arguably the most complicated part of this fairly simple architecture and the one that still requires experimentation.

The idea here is that the high level graph gets divided into sections based on the nodes captured (a node would include both an interim data state - structured or unstructured and the producer function), and as the reviewer codex goes through the nodes, it notes some as trusted, some as suspect, some as wrong. When the repair codex starts its journey, it then focuses on the issue node and any root or neighbouring context to fix the focused issue. The idea further down the line with a true long horizon job is that this will prevent the agent from making an early issue and then running a long expensive process based on a wrong output it got early in the process. Conversely the details of production of output that has been declared trusted can be cleaned out of the context.

While nice in theory, the implementation of this is not very straightforward yet (to me). There are three key considerations:

1) The selection itself: How is the boundary between different parts of the graph defined?

2) What gets passed on to the agent for its next job? (or in our ideal version to the various agents collaborating on the overall shared tasks and in the truly ideal version what the agent passes itself for its next task)

3) What happens next - what is the operation codex is asked to do? And the follow-up key consideration: in this set-up codex is meant to review a previous job (that another version of codex coded) to define a section of the pipeline and its output, as a ‘trusted boundary’.

1) The selection itself:

At the moment the set-up is a rather arbitrary approach asking codex itself when it codes the pipeline to assign some high level sections; a simple search matches the sections back to the corresponding nodes, which are then used moving forwards. There is a limit on the number of nodes and relationships per section. However there are many other ways to do this, and it requires further experimentation, e.g. selecting them based on the context they would likely represent (which at an optimal context window would then probably optimise performance?).

The scanner can distinguish between nested functions and helper functions at different levels of depth and render different levels of granularity. This gives a bit more flexibility to the selection - if an issue is spotted in a high level region, our repairer agent can dig a bit deeper into the nodes in the region, without having been flooded with that information initially. (while this experiment uses one approach to the detailed investigation, we need to tweak and test different approaches).

2) The package codex is passed on:

Below is a version included in this experiment, but there’s still a few different combinations to try.

Package field	Exact content
`run_id`	Current pipeline execution ID
`section`	Assigned units, context units, bounds and excluded overlap
`assigned_units`	Unit ID, function name, stack prefix and boundary health
`visible_evidence_by_unit`	Visible nodes, relationships, expanded helpers and still-collapsed helpers
`job_request`	Product, audience and original job configuration
`segments`	Generated segment definitions
`pipeline_input`	Exact runtime input given to the pipeline
`semantic_result`	Complete parsed output produced by the pipeline
`pipeline_stdout`	Complete pipeline stdout
`pipeline_stderr`	Complete pipeline stderr
`captured_nodes`	Selected node records
`captured_relationships`	Selected edge records
`artifact_inspections_by_unit`	Results of any requested artifact inspection
`allowed_evidence_refs`	Exact references the review is permitted to cite

Codex definitely does not prefer to look at the graph vs. words and code, that much is clear. However, having trusted boundaries has to help it logically as the details of its job start muddying its context, and second, actually looking at the data or output, not just at the code, has to improve codex’s repair/review capabilities, and the nodes in the graph focus it on it. While the code itself might not overflow codex’s context, depending on the task and general theme of the agent, the data and output definitely can overflow it… but it is also what matters for a well performing, accurate review. Furthermore, as the agent continues its path and begins to chain together more and more outputs, the context will for sure start overflowing.

3) The exact operations codex makes:

Operation	What codex is asked to return	What the controller sends next
Review graph evidence	Decision, reasoning, criteria and cited references for every assigned unit	Nothing further if the review is valid and complete
Expand a helper	Exact helper stack in `expand_helper_prefixes`	Rebuilt package containing that helper’s nodes and relationships
Inspect an artifact	Node, range, columns and optional search query in `inspect_artifacts`	Same package plus the requested artifact slice
Cite graph evidence	Exact node and relationship references in `evidence_refs`	Citations are validated against the supplied package
Identify suspect nodes	Exact node references in `suspect_node_refs`	Controller uses them when selecting and explaining a repair target
Judge a review unit	`trusted`, `failed`, `suspect`, `superseded`, or `not_pipeline_step`	Controller either accepts the result, requests a corrected receipt, or starts repair
Repair selected source	Replacement source and change summary	Controller inserts it, reruns the pipeline and captures a new graph

The key of course is that codex itself at the moment defines whether a boundary is trusted or not - given that each operation is a fresh invocation, this might be a reasonable approach, but there are potential issues with the accuracy of labeling the trusted boundary and it’s unclear the pressures this might put later on in the agent’s task path.

There are many operations that can be added to the current architecture, e.g. run arbitrary graph queries, ask for an arbitrary node by ID if it is hidden, traverse an arbitrary number of edges interactively, etc., but in this modest implementation, codex currently cannot do them.

Just to summarise the architecture: codex codes away... as the pipeline executes it then executes via the scanner, and the graph starts building up. A (for now) deterministic wrapper selects the regions of the graph to look at on an individual basis; in its next job - i.e. the review job, codex looks at a region (in depth if needed), assigns it as trusted or not, and then passes its findings on to a repair codex if needed, alternatively, it passes on the output as trusted to the next codex job.

Experiment set-up/implementation

We’ve tried to make the experiment we ran as fair as possible, and the experiment itself gave us a lot of ideas for how to change the graph selection for codex. But at the moment the experiment is measuring whether the detailed graph information is actually useful for the agent, rather than measuring the impact of correct selection and splitting the space into regions to be reviewed independently. (we will change this in further experiments).

At the moment we have 4 arms:

1) Semantic_only - doesn’t receive nodes inside a selection, only receives the given selection

2) History_full - doesn’t receive nodes inside a selection, but receives the entire history

3) Graph_full - receives the full graph/nodes

4) Graph_selected - receives the relevant nodes inside the graph selection but can also dig deeper if it chooses

For every pipeline execution, the controller:

Captures an execution graph.
Matches declared review-boundary function names against observed function stacks.
Creates review units.
Orders those units using captured dataflow relationships.
Packs the ordered units into bounded review sections.
Sends each section to a fresh review agent.

Even semantic_only uses the execution graph internally to determine which functions are assigned for review. The difference is that nodes and relationships are not shown to that arm’s codex reviewer.

The same is the case for the full history which is asked to give reviews for each region as well - regions as defined by the graph rather than just the initial pipeline declaration which also corrupts the results by leaking the graph into this agent. So I know these are improvements that need to be made to make it a fairer comparison, but I’d expect the impact to favour the execution graph arm.

`semantic_only`	`history_full`	`graph_full`	`graph_selected`
Full pipeline source. Full pipeline input. Full semantic output. Full stdout and stderr. Assigned unit names and function-stack prefixes. Section metadata. No nodes. No relationships. No artifact inspection. No accumulated history. Its assigned units were still created using the graph, but their boundary_health is deliberately omitted.	The review agent receives everything in semantic_only, plus accumulated non-scanner artifacts such as: Previous pipeline source versions. Previous semantic results. Previous inputs and logs. Applied-repair records. Repair targets and diffs where allowed. Parent-run relationships. Authoring retry records.	The review agent receives: Full pipeline source. Full input, output and logs. Every node in the current graph snapshot. Every captured relationship. Artifact previews and small artifacts. Permission to inspect any visible captured artifact. No accumulated history.	The review agent receives: No full pipeline source during review. Full input, output and logs. Assigned review-unit descriptions. Only selected nodes and relationships. Collapsed-helper descriptions. Bounded artifact previews or small complete artifacts. No accumulated history.*

Regarding the nested functions, below is an example (for the graph-selected arm):

extract_evidence_records included

get_context_value collapsed

source-processing loop collapsed

source_is_secondary... collapsed

Codex can request one direct collapsed child at a time. It cannot request an arbitrary unrelated prefix.

The experiment is supposed to measure how far the job advances correctly (according to a blind codex judge - given the same criteria to follow for each arm), and how much context is consumed.

Detailed findings

Arm	Total input tokens	Relative to semantic	Final # of boundaries flagged as not trusted*	Effective repairs
Semantic only	568,715	1.00×	4	0/3
History full	1,824,533	3.21×	4	1/3
Graph full	1,331,686	2.34×	3	0/3
Graph selected	1,000,797	1.76×	2	1/3

* a blind judge labels trusted vs. not trusted in the end for each arm based on the same criteria, but this does include some potential issues for error in measurement

From the point of view of graph performance, it is overall not bad - graph-selected is still a bit of a context guzzler but it delivers on the actual improvements. I feel, however, that the real improvement will come later in the journey of a full blown long horizon agent once history has really accumulated. We also have used very straightforward set-ups and limited codex’s operations on the graph quite heavily.

Various disclaimers: certainly this initial experiment is just a start and it is far from proving that graph selection caused the better result.

Reasons include:

This is one controlled pipeline run, not a repeated sample.
Codex decisions are nondeterministic.
Different arms produced different repaired pipelines, etc.

What we can say is that in this one paltry run, the graph-selected branch ended with the narrowest final issue set and used less context than full graph or accumulated history. This is promising evidence for graph-selected context, but as we stabilize and improve both the graph layer and the experiment set-up, we will starting running multiple experiments.

Visualiser & observability

This execution graph is born from (and will probably die with) observability. The hope is that giving the agent itself the observability of its own past code as it goes further and further away into the future of the task will help improve it. While that’s work in progress :), we still have observability benefitting us humans. It’s always good to know what your agent is up to.

So this is an initial attempt at visualising this agent as it moves back and forth through resolving the nodes… it can be vastly improved but hopefully it paints a picture:

You can tell that the job started with issues throughout every section:

Then as codex goes through the graph, nested function by nested function, it starts solving some of those issues:

This blogpost has references 😳

[1] Chen, Y., Lai, H., Feng, Y., Han, C., Zhang, Q., Lu, B., Li, M., Wang, X., Wang, Z., Xu, S., Li, Z., Jin, Z., Wu, H., Li, C., and Chen, Q. (2026). Beyond Semantic Organization: Memory as Execution State Management for Long-Horizon Agents. arXiv:2606.06090.

[2] Antoniades, A., Örwall, A., Zhang, K., Xie, Y., Goyal, A., and Wang, W. (2025). SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement. ICLR 2025; arXiv:2410.20285.

[3] Chen, Z., Tang, R., Deng, G., Wu, F., Wu, J., Jiang, Z., Prasanna, V., Cohan, A., and Wang, X. (2025). LocAgent: Graph-Guided LLM Agents for Code Localization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pp. 8697–8727.

[4] Wang, Y., Zhang, J., Wu, Z., Cai, T., Liu, Z., Sun, Q., Sun, Z., Dong, M., Zheng, M., Yin, X., and Zhu, Y. (2026). From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agents. arXiv:2606.04990.

[5] Nakajima, Y. (2026). The Log Is the Agent: Event-Sourced Reactive Graphs for Auditable, Forkable Agentic Systems. arXiv:2605.21997.

[6] Rosen, J., and Rosen, S. (2026). From Agent Loops to Deterministic Graphs: Execution Lineage for Reproducible AI-Native Work. arXiv:2605.06365.

[7] Agrawal, L. A., Tan, S., Soylu, D., Ziems, N., Khare, R., Opsahl-Ong, K., Singhvi, A., Shandilya, H., Ryan, M. J., Jiang, M., Potts, C., Sen, K., Dimakis, A. G., Stoica, I., Klein, D., Zaharia, M., and Khattab, O. (2026). GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. ICLR 2026 (Oral); arXiv:2507.19457.

Turning Observability into a Tunable Search Space

Raluca Crisan — Sun, 10 May 2026 13:24:39 +0000

In the Mlops world, people have long used DAGs/graphs or at least the consensus has been that best practice was to use them. With AI and agents, the types of graphs used for orchestration or instrumentation are more varied, but they share the same core idea: capturing and tracking the artifacts produced by a pipeline, along with their parent and child relationships. The reason for this is intuitive: a common pattern across data, ML, and agent pipelines is a sequence of steps that can be represented, stored, and discovered through a graph structure. Tracking, observing, and optimizing this sequence broadly supports monitoring, reproducibility, orchestration, backfills, retraining, and related workflows.
Etiq is a tool that creates a lineage and captures artifacts (a bit like a DAG or similar graph) on executed code. The question I’m addressing is what impact would such a tool have on different types of coding agents and coding agents architectures given how it seems tailor made for these types of agents.

Experiment Description

This first quick project looks at trying to incorporate AutoML type tuning into a coding agent whose main task is to solve various data science related challenges. Everything but the Etiq tool is more or less vibe-coded and the agent repo is not designed to be used but just to illustrate a point and for some quick hypothesis testing.

The main flow of the agent is:

Gets given a data science type task
A codegenerator codes and runs a script to answer the task till if finds a configuration that runs
Etiq tracks the artifacts and lineage (graph) of this baseline script, including the intermediate data objects, the model object, and the flow between them
A small, controlled search space from the script’s configuration is derived and passed to SMAC (a known AutoML tuner) for optimization.
Each SMAC trial reruns the same script with a different configuration, captures the resulting metric, and stores the full attempt record

The current system turns the DAG into tunable space via a number of rules: only include executed nodes, block certain node classes, inspect only literal call arguments when deciding the type of tuning, drop low-impact knobs, and optionally add safe remove-node controls. But really this is a rather arbitrary part of this process.

Also, in this implementation, the lineage edges are not used to create dependency constraints between knobs. The current implementation is therefore better described as executed-node-to-source-control tuning than true DAG-topology optimization.

Metrics and benchmark

The hardest part here was finding what to compare against. The real question is does this set-up help with something - performance, time to best result or cost in terms of LLM API calls? But what is a fair comparison point?

The starting point is always an executable script by an LLM tasked to address the given data science problem, but after this initial step, initially 4 different approaches were explored to see if we can isolate the impact of using an Etiq/DAG. The tasks themselves were adapted from MLE-Bench - only 5 of them and for structured data only. Well performing solutions to the tasks are short scripts (of no more than a few hundred lines of codes each).

While it quickly became apparent that this type of comparison above is fraught, some lessons have been learned (by me).

Because the No-DAG + SMAC also needed a tunable space, a kind of ad-hoc space was procured through some AST parsing + rules combo. The implementation and the idea was quite half baked and although it ran on a few of the tasks, it was problematic. What SMAC truly optimizes in this instance is the model. When the whole-pipeline DAG pretense was dropped from the approach, and the SMAC only optimized the model, it all made a lot more sense. In both cases the DAG+SMAC approach outperformed the No-DAG + SMAC one, in the second instance because it optimized for the data prep as well as the model (and, as we all know, data matters!). The difference was not too large which is a trend and also made sense on the small tasks/pipelines the comparison was ran on.

The harder but more interesting lesson (which made me think a bit more about the logic behind what I’m trying to do) was that the free LLM search usually outperforms everything else (or one cannot tell the difference). Again here when the LLM-search was constrained by some ad-hoc made up search space using parsing and arbitrary rules (to make the comparison seem more ‘fair’), the no-DAG LLM-search also failed or underperformed slightly the DAG version. But when the search was completely free, the LLM only (no DAG) did outperform.

Why having a DAG can help, and in which instances

When thinking about it a bit harder, this finding kind of made sense.
There are a few potential benefits of the DAG + LLM search approach vs. free LLM search approach:

Lower cost via localization (in theory DAG acts as a king of context compression, and I would emphasize ‘in theory’ here)
Better search/higher overall performance

Cost is a trickier story, but generally the second benefit only really show up for context windows that are large enough. For smaller pipelines/scripts, like the ones produced to answer the benchmark used here, it really doesn’t matter, if anything it makes things worse. At the very beginning, when the pipeline is still small, the model benefits from seeing the whole design end to end. A DAG can potentially start to become useful once the pipeline has stabilized into recognizable stages, is large enough and/or most changes become local. At that point, the DAG could help because it externalizes structure that the no-DAG agent would otherwise have to rediscover from code again and again. Additionally, if the pipelines/codebase truly is too large for one context/attention window, then the DAG is an appropriate search optimization approach.

Before concluding, it is worth making a quick detour to see if this DAG-based idea appears in other ‘nearby’ areas.

First, semantic search seems to me the closest comparison to a DAG-based approach because both aim to avoid resending the full script on every iteration. However, one localizes context by similarity, while a DAG localizes context by explicit dependency structure. E.g. a DAG can show which pipeline stage feeds another, what is upstream or downstream, and which artifacts connect different components. In data-science and ML-type pipelines, the important code may matter because of execution order, dataflow, or artifact dependencies, not because it looks textually similar to the request.

Second, looking at it from the coding agent angle, most coding agents do not natively extract a pipeline DAG and use it to guide local rewrites. Aider is the closest mainstream example, but it uses a repository graph rather than a true pipeline or dataflow DAG. And tools like Cline, Roo Code, and Sourcegraph Cody mainly rely on semantic search, AST/file analysis, repository maps.
DAG-like approaches may appear more often in context engines and MCP tools than in mainstream coding agents, but they are primarily based on static analysis, not runtime observation. Static-analysis tools usually parse files into AST-like structures and combine them into a repository-level index or graph, which can indeed be very useful for general coding. But for data-science pipelines runtime DAGs are often more relevant. because failures and performance issues depend on the specific data and configuration used.
I believe the reason we don’t really see these DAGs in practice are two-fold. One, they are extremely hard to produce reliably and then integrate in a useful manner, and two, and more importantly, their benefits only show up on either large codebases, where the main approach is so far semantic based, or specific types of long-horizon agents, which don’t really show up so often in practice. In the next blogpost, I will try to explore how setting up these observability DAGs as part of a long horizon architecture itself improves the performance (or doesn’t).

3 Levels of Observability for Coding Agents

Raluca Crisan — Sun, 10 May 2026 11:45:43 +0000

In this blogpost I am exploring how a framework that translates code into a graph fits within the observability stack. Intuitively, something that helps decompose a pipeline code into its corresponding elements should help - it should help coding agent with testing & verification, optimized debugging, auditability. It should help support a host of coding agent architectures, especially with longer-term horizons. But this blogpost is trying to explore less of what it can do, and more of where this framework fits in.

First, as a quick reminder: the framework I’m exploring - Etiq, maps your code and traces artifacts and their lineage deterministically and without manual instrumentation (a bit like extreme auto-logging) and it works for data and AI pipelines. It does so on a mix of static analysis and run-time execution.

Second, the observability stack for agent derived code is a bit hard to pin down but it can roughly fit three buckets:

Agent ‘orchestration’ - state/memory store
Telemetry
Anything that helps you assess what actually happens in the code

The diagram below represents a high level view of a coding agent structure - or at least the main idea. In a coding agent, an orchestrator manages the task, asks the LLM what to do next, invokes tools, runs code in an isolated environment, formats and checks the results before returning outputs. High level it uses something that can be described as a plan - act - verify loop, with complexity increasing depending on the agent.

Translated into our three buckets, we have the below:

Light green = state / memory store & separately artifact store
Light blue = OpenTelemetry
Pink = QA & test/ grading record

The first bucket - light green on the diagram - helps provide the agent context. That context is essential for spotting potential issues, because it shows the shape of the run and what was intended, e.g. why was a patch made, did the agent originally intend to modify one file before branching into a different fix, etc. This bucket provides what the system believed it was doing and the end artifact store: the end outputs produced by an end-to-end run.

The second bucket, the light blue one, is the runtime execution capture via OpenTelemetry. This layer captures traces, metrics, and logs, which in a coding-agent system can include model and tool-call spans, subprocess execution, HTTP and database activity, timings, statuses, exit codes, service-to-service requests, and logs and metrics surrounding the run.

Runtime telemetry provides evidence that does not depend on whether the agent was honest, accurate, or even aware of what happened. The process either ran or it did not; the HTTP request either happened or it did not. OpenTelemetry shows what the platform observed rather than what the agent claimed. It can answer questions such as whether the model call happened, whether the patch step executed, whether the script ran, if/where latency occurred, and which retry loop consumed most of the time.

The third bucket - the pink one - looks in more detail at what happens with the code that was produced by the agent in this run. It can look at code logic, unit tests, static analysis and capturing vulnerabilities. And with the Etiq framework it can have in depth observability on the executed code beyond OpenTelemetry. Let’s say this is an agent that creates workflows based on various data feeds. At some point it calls an LLM, but prior to this call, it does 10 steps that are just about data processing, once the LLM returns an answer this gets joined up with another data source and the pipeline keeps going. The green bucket would provide us with the agent’s intention in writing this code and hopefully a coherent plan, the blue telemetry bucket would capture the API calls to the LLM and to get the initial data and would associate the full code with them. But regarding the 10 interim steps there is no way to log them in an observability framework outside instructing the agent itself to capture the artifacts and associate them with the appropriate function. Semantic search does not have a direct link to the produced interim artifacts. And this is where a framework like Etiq comes in - that is able to log granular steps of interim artifact/functions pairs and lineage.

In the case of a very simple example code generation agent with the following structure:

The orchestration would capture details on each of the agent’s nodes, below just for example purposes:

The OpenTelemetry logging would capture information as per below:

And Etiq would log the detail of what actually ran during the code execution for the given run:

The information produced via the Etiq framework serves a few different purposes:

It captures interim artifacts/function pairs thus allowing verification, test harnesses and checks on them - this enables the kind of granular testing data and AI pipelines need
It optimizes debugging as it can point exactly to the function that is producing the wrong interim step
It provides a level of audibility that open telemetry and agent orchestration or end artifact capture cannot do as it traces the lineage of data through the pipeline

Fundamentally it is great that we are able to to observe what the system is trying to do and what it stores at the end as code or output artifacts, it is equally important that we can capture the API calls and tool calls to the data sources, LLMs, the sandbox in which the code runs, etc. But there is currently a gap when it comes to observing the executed code the system produces. And the solution to this gap is an observability framework beyond what we currently have in the space, namely a framework that can trace the interim artifacts produced by the code and their producer functions and map their relationships, so they can be tested, debugged and audited.

The observability gap for data science and analytics agents

Raluca Crisan — Sun, 10 May 2026 11:05:31 +0000

Databricks and similar enterprise data platforms have spent a great deal of effort and time to full-proof their product suite with relevant observability and tracing. Not surprisingly this is needed as part of enterprise support especially in regulated sectors. But for the specific case of sophisticated data science and analytics agents there is a gap in the observability suite not just for Databricks but across all big and small analytics and data science agent providers.

In the case of Databricks, even with notebooks as a primary user interface, given the offerings across data lineage, data management and MLflow, the level of control and tracing is no doubt high. However both large vendors like Databricks and Snowflake and smaller analytics and data science agents suppliers share an observability gap. The gap is inherent to coding agent architectures and does not apply equally to all agents. A text-to-SQL assistant can be wrong in an ‘obvious’ way: the result makes no sense. A multi-step python or spark pipeline produced by an agent is different. Even when made by a human, it’s hard to unpick pipeline logic given endless combinations of joins, data issues, data characteristics. This problem doesn’t go away when an agent is involved. E.g. Genie can plan a solution,run code, use cell outputs to improve results, and fix errors automatically. The question is what beyond the initial reasoning and the final artifact can be inspected in this instance and what can be reliably/not-probabilistically logged.

To achieve their objectives, these more sophisticated data science and analytics agents need to create relatively complex multi-step pipelines. Past the initial data retrieval and the final storage step, the pipelines themselves are just arbitrary code. Observability for this type of scripts when they are man-made span a whole area of companies in the MLOps space including Databricks’ own Mlflow. But it is unclear what observability is out there when this code is produced by agents - short of asking the agent itself to instrument the code (probabilistically), thus somewhat defeating the purpose of observability in the first place.

Now that we’ve narrowed the gap in observability from the bigger data platform context to a specific area: the ‘executed pipeline code’ element part of these more sophisticated analytics and data science agents workflow, my first question was to see if Mlflow or a different ‘off-the-shelf’ tool in the ecosystem can fill this gap directly. For why OpenTelemetry is not enough here please see the previous blogpost.

Unsurprisingly, Mlflow is heading in the direction of more granular instrumentation with the least amount of effort - on anyone’s part, human or agent. For classic ML, a single mlflow.autolog() call can automatically capture params, metrics, models, datasets, and artifacts around supported training APIs, while for GenAI and agent workflows, one-line tracing primitives like @mlflow.trace, mlflow.trace(...), and mlflow.start_span() add function- and block-level visibility, including parent-child relationships, inputs, outputs, exceptions, and execution time.

My initial experiments with trying to instrument agent-created code with Mlflow deterministically have allowed me to track the models as experiments which was a good step in the right direction 👍, but of course I cannot track data transformations - with Mlflow or with anything else that I’m familiar with.
Trying to track with autolog was the better option for me - rather than the tracing function, because I’m not really tracking the agent, I’m trying to track what’s happening in the code produced by the agent when it runs. Below some example basic tracking:

The gap is of course tracking what actually happens inside the pipeline outside the model itself, all the data operations for which no observability is present. While the code is of course the best evidence in other use cases, for pipeline types structures where the outcomes are heavily influenced by the particulars of the data, the code is not enough - observability on code and runtime execution both is needed and for these data science and analytics agents, the code they produce (outside the model itself) is currently a black box - an example table of interim artifacts below (made using Etiq), which at the moment tooling like Mlflow does not capture for agent written code.

In this space we were brainwashed to believe that observability matters at all cost; however I feel for this instance given the perception of coding agents in the market, an argument might have to be made for why it really matters.
First, it’s about auditability. Truly not everyone cares about this and not everyone should. But in regulated sectors like finance or healthcare this matters. For model validation in e.g. finance, the type of data lineage documentation required involves more than what gets stored in Unity catalogue, Delta lakes or Mlflow model tracking - all useful components. This type of use case needs to reflect the transformations that happen in the code itself once executed and teams currently do this manually. At the moment, the use of semiautonomous coding agents for these use cases is minimal but this is not where the enterprise stack is going.

Second, observability for these more sophisticated agents moves into other related risks, such as reproducibility, error propagation across longer pipelines, and general control issues for agent generated code.
Without observability, it is harder to track ‘semantic mistakes’ the agent might make, such as not using the correct metric definition, or applying the analysis or model to the wrong population. A bad transformation early in the pipeline affects everything downstream. I’m not sure what exactly is the level of observability needed to help us mitigate the potential issues, but without any we certainly would struggle.

Reproducibility is another area that does require some level of observability: if transformation execution is not observable, the final notebook may not be a faithful record of the run that produced the result. Similarly, we would struggle to compare agent runs over time (or rather without observability we would struggle more).

The key argument for in-depth-observability on agent generated code is enterprise level control especially for regulated sectors. Usage of these sophisticated data science and analytics agents in regulated sectors might be small to begin with relative to the size of the overall data platform offering. However as Databricks and large enterprise data platforms are feeling the pressure from coding agents and foundational models, there just aren’t that many avenues left to go into. If Databricks’ long-term position is around providing the governed system in which semiautonomous enterprise agents can actually run, then any observability gap will prove problematic.