Exporting Conversation History and Normalising It

#ai #machinelearning #tutorial #llm

A chat export arrives as a JSON file that is nearly, but not quite, the message array every inference API wants. The gap is usually one of two things: content that is a list of parts rather than a string, or a conversation stored as a graph of nodes rather than a list. This walks through detecting which you have and converting either.

The two shapes you will find

Export schemas are internal formats. They are rarely documented, they change between platform releases, and no two products agree. But almost all of them are one of two structures underneath, and it takes thirty seconds to tell which by opening the file.

Shape A, the flat array. Each conversation object has a title, some timestamps, and a messages array already in order. Each entry has a role and content. This is the easy case and it is what you get from products that never supported editing a message.

Shape B, the node map. The conversation is an object keyed by node id — often called mapping — where each node holds a message, a parent id and a children array, plus a pointer to the current leaf. This shape exists because the product let users edit a prompt or regenerate an answer, which forks the conversation. What you see in the UI is one path from the root to one leaf; the file contains every branch that was ever created.

The failure mode here is specific and common: a transform that iterates the node map in key order produces a message list containing every abandoned branch, interleaved, in insertion order. It looks plausible, it parses, and it is not a conversation. You have to walk the parent chain from a leaf.

The target format

The destination is the message array that chat completion APIs take: a list of objects with a role and a content, roles drawn from system, user, assistant and tool. We will emit one JSON object per line — JSONL — because that is what evaluation harnesses, fine-tuning upload endpoints and batch APIs all consume, and because it streams.

{"id":"c_01","title":"Refund policy","created":1754870400,"messages":[
  {"role":"system","content":"You are a support assistant."},
  {"role":"user","content":"How long do refunds take?"},
  {"role":"assistant","content":"Refunds post within five business days."}
]}

Content in the target is a plain string. If your source has multi-modal parts — images, files, audio — flattening to text loses them. Decide that deliberately: for replaying conversations as evaluation cases the text is usually all you want, and for archival it is not.

The transform

This is one file, no dependencies, Node 18 or later. It detects the shape, linearises shape B by walking parent pointers from the leaf, flattens the three content encodings that occur in practice, and drops conversations too short to be useful.

// normalise-export.mjs
import { readFileSync, writeFileSync } from "node:fs";

const ROLES = new Set(["system", "user", "assistant", "tool"]);

// Content shows up as a string, an array of parts, or an object with .parts.
function flattenContent(content) {
  if (typeof content === "string") return content;
  if (Array.isArray(content)) {
    return content
      .map((p) => (typeof p === "string" ? p : (p.text ?? p.value ?? "")))
      .filter(Boolean)
      .join("");
  }
  if (content && Array.isArray(content.parts)) {
    return content.parts.filter((p) => typeof p === "string").join("");
  }
  return "";
}

function linearise(convo) {
  // Shape A: already an ordered list.
  if (Array.isArray(convo.messages)) return convo.messages;

  // Shape B: node map. Walk parents from the current leaf, then reverse.
  const nodes = convo.mapping;
  if (!nodes) return [];
  const leaf =
    convo.current_node ??
    Object.keys(nodes).find((id) => (nodes[id].children ?? []).length === 0);

  const chain = [];
  const seen = new Set();
  for (let id = leaf; id && !seen.has(id); id = nodes[id]?.parent) {
    seen.add(id);
    const msg = nodes[id]?.message;
    if (msg) chain.push(msg);
  }
  return chain.reverse();
}

function toChatMessage(raw) {
  const role = raw.role ?? raw.author?.role;
  if (!ROLES.has(role)) return null;      // drops "system"-ish internal roles
  const content = flattenContent(raw.content).trim();
  if (!content) return null;               // drops empty placeholder nodes
  return { role, content };
}

const parsed = JSON.parse(readFileSync(process.argv[2], "utf8"));
const conversations = Array.isArray(parsed) ? parsed : [parsed];

const lines = [];
let skipped = 0;
for (const convo of conversations) {
  const messages = linearise(convo).map(toChatMessage).filter(Boolean);
  if (messages.length < 2) { skipped++; continue; }
  lines.push(JSON.stringify({
    id: convo.id ?? convo.conversation_id ?? null,
    title: convo.title ?? "",
    created: convo.create_time ?? convo.created_at ?? null,
    messages,
  }));
}

writeFileSync("messages.jsonl", lines.join("\n") + "\n");
console.log(lines.length + " written, " + skipped + " skipped");

Running it

Request the export from the platform’s settings and wait for the archive. Unpack it and find the conversations file — it is usually the largest JSON in the bundle.
Open the first 200 lines and answer one question: is there a messages array, or a node map? Everything else in the file is decoration.
Run node normalise-export.mjs conversations.json. It writes messages.jsonl and prints how many conversations it kept and how many it skipped.
Check the skip count. A high skip rate almost always means shape B was not detected — the node map is under a key other than mapping — rather than that your conversations are short. Adjust the one line and rerun.
Spot-check three conversations by eye against the product UI. In particular check one where you know you regenerated an answer: if the transcript contains both the discarded and the kept response, the parent walk is not being used.
Load the JSONL into whatever consumes it. As evaluation cases, take all messages up to the last user turn as input and the final assistant turn as the reference.

The edges that bite

Tool turns need more than a role. Where an assistant message carried tool calls, a faithful replay needs the call arguments and a matching identifier on the tool result. Exports frequently drop one or both, which turns a tool exchange into an assistant message with empty content — dropped by the filter above, silently. If tool use matters, count how many messages the filter discards for empty content before you trust the output.
Branches are data too. The parent walk keeps one path and throws the rest away. If you are mining the export for preference pairs, the discarded siblings of a regenerated answer are exactly what you want, and this transform deletes them. Collect children instead of following one leaf.
System prompts are usually absent. The example output above shows one for shape; most exports do not contain it, because it was never a message the user sent. Replaying without it reproduces the words and not the behaviour.
Timestamps may be seconds or milliseconds, and occasionally strings. Normalise on the way out rather than discovering it during a sort.
Role vocabularies do not match. Products invent internal roles for their own bookkeeping — a hidden setup turn, a tool-orchestration pseudo-role, a placeholder for a message still streaming. The allowlist above silently discards anything outside the four standard roles, which is the behaviour you want, but print the distinct role values you encountered on the first run so that you are discarding them knowingly rather than by accident.
Very large exports do not fit in memory. JSON.parse on a multi-gigabyte archive fails outright rather than slowly. Above roughly a gigabyte, split the file per conversation first or use a streaming parser; the rest of the transform is unchanged because it already works one conversation at a time.
Personal data comes with the file. An export of real conversations is a production dataset with everything users ever typed in it, including things they typed by mistake. Treat it with the controls your production database has, and redact before it becomes a fixture.

The transform gets you a replayable conversation. It does not get you a complete one, and the difference is worth knowing before you plan a migration around it — what exports leave out covers which fields are missing and what each one costs you.

DEV Community