For my own reference, I was trying to answer a simple sounding question about an n8n workflow which branches did this execution exercise?
It seems like a lookup given an execution, find the nodes, and for their outputs, observe which items were present.
That gives you the exercised branches. This might be possible for some nodes.
Then you get to filters.
Then you get to loops.
Then you realize no, the trace doesn't record everything. It records specific things, elides others, and sometimes gives you enough information to reconstruct what it didn't record.
That distinction matters if you're trying to build anything on top of this.
The useful question to ask isn't simply what does the trace say? but also what can I prove from it, and what am I assuming?
And here's what I found.
The obvious setup branch coverage sounds like a lookup
Consider this node
┌── true ──> Send email
Trigger → IF ────┤
└── false ─> Create task
After executing, I'd like to know whether both branches were exercised.
A coverage report might then look like this
IF
├── true ✓ exercised
└── false ✗ not exercised
That's useful when testing a workflow, since the UI can tell me that both branches exist, but only an actual execution can tell me whether either of them was exercised.
The same applies to a Switch
Trigger
v
Switch
/ | \
A B C
If a test run only uses A, I'd like the coverage tool to be able to report that B and C weren't exercised.
That sounds simple enough.
The execution trace already has the node outputs, so why not simply read which outputs had items?
That's actually the case for branching nodes.
The problems begin when you start trying to apply the same logic to every node.
So let's talk about what the trace does record.
What the trace does record
One of the neat things about the execution trace is how node outputs are structured.
They're arrays of items, with each item having a JSON field, and multiple outputs per node
json
{"Switch": [[{ "json": { "id": 101 } }],[],[{ "json": { "id": 103 } }]}
Let's say these are the outputs of a switch.
Output 0 had an item, output 1 had nothing, and output 2 had an item.
That gives us three states
output had items the branch was taken
output was empty the branch was skipped
the node wasn't present at all the branch didn't exist
The third state arises from the simplest of causes if the node wasn't executed, it won't appear in the execution trace.
So, for branching nodes, it's actually possible to reconstruct a basic coverage report directly from the trace.
If IF was executed, and its true output has items, then the true branch was exercised.
If the false output is empty, then that branch wasn't taken.
And if the downstream node doesn't appear in the trace at all, it wasn't executed in this run.
That's actually incredibly useful, since it means that for IF like nodes, coverage is directly observable.
The graph can tell you that a branch exists, but only the execution can tell you whether it was taken.
For most other nodes, those are orthogonal pieces of information.
That's why I'm tempted to think of this as the simplest possible coverage report.
But there's a trap waiting for us here, and it's in the third state absence from the trace.
An empty output means something specific, but an absent node means something else entirely, and the trace won't tell you which.
That's important enough that we'll come back to it.
Let's look at nodes that actually process items and see what the trace can and can't say about them.
What the trace doesn't record
Let's take a Filter:
Input
v
Filter
v
Output
Let's say that 100 items were input into the filter, and the condition was met for 70 of them.
The trace will then have the 70 items in the output.
But there is no field in the trace that would have the number 100 in it, so the number of input items is not directly observable.
That means the trace can't tell you how many items were dropped by the filter.
It can help you reconstruct that number, assuming you know the number of input items:
100 - 70 = 30
If the upstream node emitted 100 items, and the filter only emitted 70, it's reasonable to assume that 30 were dropped.
But that's an inference that's not directly supported by the trace.
This makes all the difference when it comes to tools that report coverage.
I think it's much more useful to see something like this
Input: 100 (inferred from upstream)
Output: 70
Dropped: 30 (inferred)
rather than
Filter dropped 30 items
The reason for this is simple the second bullet implies that n8n itself knows that 30 items were dropped, when in reality, that number was reconstructed from other values.
This is especially useful when it comes to loops, since the obvious source of truth about items in them stops being reliable.
The loop trap
The standard pattern for looping in n8n looks like this
┌─────────────────────┐
│ │
│ ┌─────────────┐ │
Input ──────────────┼──>│ Loop Over │ │
│ │ Items │ │
│ └──────┬──────┘ │
│ │ │
│ v │
│ Loop body │
│ │ │
│ └──────────┘
│ back to
│ input
└─────────────────────┘
│
Done ───────────────> next node
By default, Loop Over Items nodes feed batches of items back into themselves until the loop is completed, with the Done output being used to signal that the loop has completed.
This has an impact on the analysis of loop cardinality, since the number of loop iterations isn't directly recorded.
The most obvious approach to loop iterations is to infer them from input items
Loop iterations = number of items input to loop
If true, then with a batch size of 1, a single item would cause the loop to execute once.
However, the input to this loop comes from two places: the initial input to the loop node and the loop body itself.
A naive approach to counting loop iterations would then mistakenly count the number of items input to the loop as the number of loop iterations.
For example, if you used this pattern to implement a map
Loop iterations = number of items input to loop node
then a cardinality analysis of the loop body would erroneously report that the loop executed more times than it actually did.
The reason for this has to do with how feedback connections work in n8n.
If you're trying to count loop iterations, the number you're looking for is not the number of items input to the loop node, but the number of items dispatched from its loop slot
Loop iterations = number of items dispatched from loop slot
Using the cardinality of the loop body as a proxy for loop iterations is not reliable, since the loop body may have other inputs besides the loop node itself.
This is why I think of it as the cardinality of the loop dispatch itself.
Conceptually, it's similar to this
input
v
┌───────────────┐
│ Loop Over │
│ Items │
└───────┬───────┘
v
loop body
└───────────────┐
v
loop input again
Done ───────────────> after completion
The input to the loop body isn't a new original input item, but a re used item from the loop output.
This means that the same cardinality tracking rules that applied to the filter don't automatically apply to loops.
You have to examine the context of the node and, specifically, the slot you're looking at, since each slot represents a different kind of connection.
That brings us to the next subject what you can't observe.
What you can't observe
The example that I think has the most interesting edge case is a missing Loop Over Items node.
If it's not in the execution trace, does that mean that it was reached with zero items, or that it was never reached at all?
There is no flag in the trace that would distinguish between the two cases, which means that a coverage tool can't know for sure which one is the case.
This means that a branch coverage tool can't reliably distinguish between
- the loop was reached, but no items were present when it was executed
and
- the loop was never reached.
There's no separate "never executed" marker in the trace.
So the best coverage report for such a case is this
Loop: UNOBSERVABLE
this might be frustrating for the user, but it's the correct report given the information available in the trace.
The loop could have been executed zero times: either because it had no items to process, or because it was never reached.
This is an example of why I think such a distinction is useful for coverage tools, especially the ones that have to run on top of the same execution traces that n8n uses.
When you start writing tools that need to reason about arbitrary node executions, you find that binary outcome was it executed isn't always useful.
This is especially the case when the tool can't look at the node's code, only at the execution trace.
That's where the unobservable outcome is useful.
I think this can be especially valuable in testing: a test report that knows when it can't observe certain outcomes is much more useful than one that tries to pretend that it knows everything.
Two CLI traps
There are also two gotchas when it comes to using the CLI to drive execution in scripts.
--rawOutput isn't as raw as the documentation suggests
The --rawOutput option is supposed to suppress any extra text and only print the JSON, but that's not always the case.
In particular, n8n may print some diagnostic information at the start of the execution.
So a script like this
n8n execute --rawOutput ... | jq
is not guaranteed to parse the JSON correctly, since some diagnostic text may appear above it.
This was the case in the 2.33.4 release, which printed a runner identifier line like this
Runner ID: abc123
So a naïve Python script that tried to do this:
python
import json
json.loads(stdout) would fail.
It's better to process the stream more carefully, for example, by finding the actual JSON in the stream.
That said, this approach may not be necessary if you're writing the tool yourself.
The point is that the CLI's behavior may change, and if you're writing automation around it, you should be prepared to handle those changes.
A good rule of thumb is not to make assumptions about the machine parseable output format.
If you're writing a script that's supposed to run on n8n and parse execution data, design it in such a way that it would continue working if the logging format changes.
It won't be easy, but it's worth it if you're trying to write long lasting automation.
execute --file can return success exit code when the file wasn't executed
The second gotcha is much more serious.
If you're using n8n execute --file to run a workflow as part of your testing infrastructure, it's reasonable to assume that a nonzero exit code means that the execution has failed.
However, if you try to run a file that n8n refuses to execute for any reason (for example, because it's not a valid JSON), it'll exit with code zero.
This means that this simple check
n8n execute --file workflow.json
if [ $? -ne 0 ]; then
echo "execution failed"fi
is not sufficient to determine whether the file was executed.
The tool has exited successfully, but the execution wasn't performed.
Your testing infrastructure now has to do more sophisticated analysis, for example, by looking at the standard output and checking for specific markers.
It's an easy mistake to make, since exit codes are usually the primary signal that a process has completed successfully.
But in this case, they're not sufficient to determine whether the tool actually did what you wanted it to do.
That's why provenance is so important in testing.
Why every coverage report needs provenance
Once you start building workflow coverage from execution data, the most useful feature isn't a clever branch counter
When the report says
IF → false branch
NOT COVERED
I want to know what that means.
When it says
Filter
Dropped: 30
I want to know whether that's directly observed or reconstructed.
And when it says
Loop
UNOBSERVABLE
I want to see something like this
Node absent from execution data.
The trace does not distinguish between
reached with zero items
never reached
That way, I can actually reason about the report.
That's the value of the provenance driven approach, and it's the reason why these distinctions are important to me when writing such tools.
n8n's execution trace is not a perfect source of truth about the execution, and I don't think it was ever meant to be one.
However, it does have enough structure to make some things directly observable, enough information to allow for reconstructing others, and some gaps that can't be reconstructed.
For example, for branching nodes, the presence of output arrays allows us to directly observe which branches were taken.
For filters, the number of dropped items can be reconstructed from other information, but the fact that it's reconstruction needs to be noted.
For loops, the cardinality can be observed in the dispatch output, not the input, due to the way feedback connections work.
For an absent loop node, the trace may not be able to distinguish between zero items and no execution at all, and the coverage tool should report that as UNOBSERVABLE.
These are the insights that let me write a program that analyzes n8n execution traces. They're also the reason why the small CLI tool I'm working on reports this information alongside each coverage finding
The tool itself is not that interesting, but the ideas behind it are.
That's why I'm writing this post I think that execution analysis is an interesting space, and the way n8n exposes execution information makes some tasks much more interesting than they would be elsewhere.
Top comments (0)