An agent trajectory dataset needs more than final answers

An agent trajectory dataset should preserve the behavior that produced an outcome, including the steps before the final response.

That means keeping the task, observations, tool calls, tool results, decisions, corrections, and authoritative outcome. Without provenance, privacy controls, and a frozen evaluation boundary, a large trajectory collection can still be a poor training dataset.

Choose the training unit

One example might be a full episode, a single decision point, or a correction pair. I keep the original task and enough state to explain each action.

Hidden evaluator feedback that the runtime will never receive should not leak into the training input.

Capture typed events

Observations, permitted reasoning, tool names, arguments, results, errors, and final outputs should use a versioned schema.

Raw source references need to remain available so the transformation can be audited. Flattening every event into one undifferentiated chat transcript throws away useful structure.

Filter for useful behavior

I remove duplicates, leaked tests, private values, malformed tool calls, and trajectories without an authoritative outcome.

Successful examples, correction data, and hard negatives should stay distinguishable. Every acceptance or rejection should retain its reason.

Split by source and task family

Random row splits can place nearly identical episodes in training and test data. I group by repository, task template, session, or source artifact before splitting and freeze the holdout before tuning the recipe.

Train the narrowest useful target

Trajectory data can support supervised fine-tuning, preference pairs, distillation, routing, or an evaluator. The objective should follow the observed failure.

If the model already over-edits, an objective that increases edit pressure is the wrong response.

Evaluate behavior, not imitation

The important measures are whether the specialist chooses the right tool, produces valid arguments, responds correctly to results, stops at the right point, and completes the task.

Unsupported actions and breadth regressions need their own gates.

PostTrainLLM includes trajectory conversion, correction-to-data, synthesis, filtering, deduplication, tool-calling evaluations, and factory-run evidence. The full workflow is at https://posttrainllm.com/agent-trajectory-dataset.