Postmortem: Each SSE Tool Delta Was JSON-Decoded Alone

A merge gate dispatched agent tools from partial JSON. Each SSE delta was decoded as a complete object. Empty defaults then looked like a valid tool call.

This postmortem reconstructs that class of incident. It is a lab walkthrough, not a vendor claim. The durable fix is assembly plus schema checks.

What actually failed

The gate subscribed to a streamed chat completion. Tool-call arguments arrived as many tiny SSE deltas. The parser called json.loads on every streamed delta.

Most streamed chunks were still invalid JSON fragments. The catch block returned an empty dict. The dispatcher treated that dict as complete arguments.

The run_tests tool then executed with path set to None. The wrapper skipped the suite and exited zero. The merge bot published a green status.

The repository clone was intact. The model stream was intact. Only the argument parser failed closed into success.

Lab timeline

The following timeline is a reconstructed lab run. It is not a production outage report. Clocks are relative to job start.

1. T+00s. The worker cloned the target repo into /job/src.
2. T+08s. The agent opened a streamed completion with tools enabled.
3. T+11s. The first delta contained a tool name, run_tests.
4. T+12s. Later deltas contained fragments of {"path":"tests/"}.
5. T+12s. The parser decoded each fragment and stored {}.
6. T+13s. The finish event arrived with finish_reason=tool_calls.
7. T+14s. The dispatcher invoked run_tests with path=None.
8. T+14s. The wrapper logged a skip and returned ok.
9. T+15s. The merge gate recorded failures=0 and passed.

No test process started in that window. Pytest never wrote a report file. Green status meant the skip path ran.

Contributing factors

Several small choices stacked into a green lie.

• Per-chunk json.loads ran on SSE tool argument deltas.
• except json.JSONDecodeError: return {} hid every truncated buffer.
• The tool schema marked path as optional for convenience.
• The wrapper skipped work when path is None and still returned ok.
• Skip paths kept process exit code 0 for the merge bot.
• The gate never asserted that pytest actually started.
• The gate never required a terminal finish_reason before dispatch.
• Logs printed tool names but omitted raw argument buffers.

None of these look severe alone. Together they certify a no-op run. Optional kwargs are the wrong default for streamed model output.

Why empty dicts are toxic here

Empty dicts look like valid kwargs in Python. func(**{}) is a fully legal call. Optional parameters then take their defaults without protest.

Defaults exist for humans typing CLI flags. They are not a contract for truncated model streams. A missing path is an error in this gate.

Exit code 0 is also toxic in this path. A skipped suite is not a passed suite. The gate must distinguish skip, collect-empty, and real pass.

Raw SSE shape, labeled as lab data

The next lines are reconstructed teaching data. They are not a captured vendor trace. Wrappers add data: prefixes and choice indexes.

# reconstructed; not a live provider capture
data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"name":"run_tests","arguments":"{\""}}]}}]}

data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"path"}}]}}]}

data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\":\"tests/\"}"}}]}}]}

data: {"choices":[{"finish_reason":"tool_calls"}]}

Naive code JSON-decodes the arguments string on each event. The first event is '{"', which is not an object. The catch block stores {} and keeps going.

A later finish event does not repair that store. Dispatch then uses the last “successful” object. That object is still empty.

Artifact: prove the parser, not the model

The next files reconstruct the bug without any network. Save them under harness/ in a throwaway repo. Run the tests before changing production parsers.

harness/stream_chunks.py

"""Reconstructed argument deltas. Not a live API capture."""

CHUNKS = [
    '{"',
    'path',
    '":"',
    'tests/',
    '"}',
]

COMPLETE = '{"path":"tests/"}'

harness/parse_naive.py

import json
from typing import Any


def parse_delta(delta: str) -> dict[str, Any]:
    try:
        data = json.loads(delta)
    except json.JSONDecodeError:
        return {}
    if not isinstance(data, dict):
        return {}
    return data


def dispatch_from_stream(chunks: list[str]) -> dict[str, Any]:
    last: dict[str, Any] = {}
    for chunk in chunks:
        parsed = parse_delta(chunk)
        if parsed:
            last = parsed
    return last

harness/parse_assemble.py

import hashlib
import json
from typing import Any

REQUIRED = ("path",)


def assemble_arguments(chunks: list[str]) -> str:
    return "".join(chunks)


def buffer_id(raw: str) -> str:
    digest = hashlib.sha256(raw.encode("utf-8")).hexdigest()
    return digest[:12]


def parse_assembled(chunks: list[str]) -> dict[str, Any]:
    raw = assemble_arguments(chunks)
    if not raw.strip():
        raise ValueError("empty tool arguments")
    data = json.loads(raw)
    if not isinstance(data, dict):
        raise ValueError("tool arguments must be an object")
    missing = [key for key in REQUIRED if key not in data]
    if missing:
        raise ValueError(f"missing keys: {missing}")
    path = data["path"]
    if not isinstance(path, str) or not path.strip():
        raise ValueError("path must be a non-empty string")
    data["_buffer_id"] = buffer_id(raw)
    return data

harness/test_parse.py

import json

import pytest

from parse_assemble import parse_assembled
from parse_naive import dispatch_from_stream
from stream_chunks import CHUNKS, COMPLETE


def test_naive_parser_hides_truncation():
    result = dispatch_from_stream(CHUNKS)
    assert result == {}


def test_assembled_parser_recovers_path():
    result = parse_assembled(CHUNKS)
    assert result["path"] == "tests/"
    assert result["path"] == json.loads(COMPLETE)["path"]
    assert "_buffer_id" in result


def test_assembled_parser_rejects_empty():
    with pytest.raises(ValueError, match="empty"):
        parse_assembled([])


def test_assembled_parser_rejects_partial():
    with pytest.raises(json.JSONDecodeError):
        parse_assembled(CHUNKS[:2])

The naive test documents the production bug. It should stay red-to-green as a lock. Delete it only after the worker loses parse_delta.

Test plan

1. Create a clean venv in the throwaway repo.
2. Install pytest only, with no extra plugins.
3. Run python -m pytest harness/test_parse.py -q.
4. Confirm the naive test still observes {}.
5. Confirm the assembler recovers path tests/.
6. Confirm partial buffers raise JSONDecodeError.
7. Confirm empty buffers raise ValueError.
8. Wire parse_assembled into the worker only after that.

python -m venv .venv
. .venv/bin/activate
pip install pytest
python -m pytest harness/test_parse.py -q

Chunk table from the lab buffer

Chunk	json.loads	Naive store
'{"'	error	{}
'path'	error	{}
'":"'	error	{}
'tests/'	error	{}
'"}'	error	{}
joined	{"path":"tests/"}	not used

The joined row is the only legal decode. Every other row is a trap. Per-chunk success is the wrong success metric.

Durable fix

Do not decode deltas as JSON objects. Concatenate argument strings in index order. Decode once after the stream ends.

Then apply a schema with required fields. Treat parse errors as job failures. Treat skipped tools as job failures too.

Worker rules

1. Buffer tool_calls[i].function.arguments as raw text only.
2. Ignore JSON until finish_reason is a terminal value.
3. Reject the job if that buffer is empty or whitespace.
4. Decode the buffer with json.loads exactly once.
5. Validate required keys with no default values.
6. Record buffer_id in the immutable job log.
7. Reject path=None before any test wrapper runs.
8. Fail if pytest never starts or collects zero nodes.

Fail-closed wrapper

import subprocess
import sys
from pathlib import Path


def run_tests(path: str, job_root: Path) -> int:
    if not path or not str(path).strip():
        raise ValueError("run_tests requires path")
    target = (job_root / path).resolve()
    if not str(target).startswith(str(job_root.resolve())):
        raise ValueError("path escapes job_root")
    if not target.exists():
        raise FileNotFoundError(target)
    collect = subprocess.run(
        [sys.executable, "-m", "pytest", str(target), "--collect-only", "-q"],
        cwd=job_root,
        check=False,
    )
    if collect.returncode != 0:
        return collect.returncode
    return subprocess.call(
        [sys.executable, "-m", "pytest", str(target), "-q"],
        cwd=job_root,
    )

The wrapper takes path: str, not path: str | None = None. Missing input cannot compile into a skip. Path also stays inside job_root.

Collect guard for defense in depth

#!/usr/bin/env bash
set -euo pipefail
ROOT="${1:?job root required}"
TARGET="${2:?test path required}"
cd "$ROOT"
if ! python -m pytest "$TARGET" --collect-only -q | grep -Eq '[1-9][0-9]* tests? collected'; then
  echo "collect-guard: zero tests" >&2
  exit 2
fi
python -m pytest "$TARGET" -q

Use this after the parser already required path. Empty directories still fail. A skip wrapper can no longer return zero quietly.

Log fields that would have caught this

Keep a small structured record per tool call. Do not log full prompts if they contain secrets. Do log parse mechanics.

• tool_name and tool index from the stream.
• finish_reason copied verbatim from the final event.
• arg_bytes length of the assembled argument buffer.
• buffer_id short sha256 of that buffer.
• dispatch_ok boolean after schema validation.
• pytest_started boolean from the wrapper.
• collected integer from --collect-only.

A row with arg_bytes=0 and dispatch_ok=true is the incident. Alert on that pair. Do not alert on model prose.

Decision table

Signal	Naive gate	Durable gate
Empty delta	Store {}	Keep buffering
Fragment '{"'	Store {}	Keep buffering
Full object after join	Dispatch	Schema check, then dispatch
Finish with empty args	Dispatch skip	Reject job
path missing	path=None skip	Reject job
Zero tests collected	Pass	Reject job
Parse error	Pass via {}	Reject job

The durable column fails closed. Closed means the merge stays blocked. Blocked is cheaper than a green no-op.

Isolated process for the harness

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. A free server can host the parser tests off-laptop.

The same checks run on any SSH host. Teams already using that free server can run this harness there. No other product capability is claimed in this writeup.

Turn the network off for the parser tests. The chunks file is the only input. A live agent stream comes after the tests fail closed.

Limitations

The lab chunks are hand-written teaching data. They do not prove one vendor stream format. Real SSE wrappers add prefixes, indexes, and multiple tool slots.

Assembly does not fix a model that emits the wrong path. It only fixes premature decode. Schema checks do not replace a real test suite.

json.loads still accepts extra keys by default. A hostile buffer can include unused fields. Path values still need an allowlist against job_root.

Pytest collection can still find tests that never assert. This guard only blocks empty collection. Assertion quality remains a separate review step.

Reused workspaces can still hide older reports. Pair this parser fix with a fresh job root. Do not reuse /tmp across tenants.

Who should not use this approach

Do not keep per-chunk decode in any tool dispatcher. That pattern is the bug, not a style. Convenience defaults do not belong on streamed arguments.

Skip this exact chunk file if the agent never streams arguments. Some runtimes deliver one complete object. Assembly is then a no-op, but schema checks still apply.

Do not use a shared server for untrusted repos without job isolation. A free server does not imply a strong sandbox. Create a fresh workspace per job.

Do not treat this postmortem as evidence of a public outage. It is a reconstructed failure mode. Ship the tests before changing the worker.

Close

Decode streamed tool arguments once, after the buffer closes. Fail the job on empty JSON, missing keys, and zero collected tests. Green status must mean tests ran, not that a parser returned {}.