Turning Reddit threads into RAG-ready Markdown

AI assistance disclosure: this tutorial was drafted with AI assistance.

A collection of thread text becomes more useful for retrieval when each chunk retains its source, timestamp, and position in the conversation. A flat string loses the distinction between a question and its replies. This tutorial starts with an existing JSON export that you are entitled to process. It does not collect data from Reddit or require a particular paid service.

Keep provenance alongside the text

Use a small normalized record: id, parentId, threadId, title, text, permalink, and createdAt. Omit usernames unless the application has a specific need for them. Store a collection timestamp separately from the original publication timestamp. These are different facts and matter when a source changes.

The example fixture is synthetic. It demonstrates formatting, not an actual Reddit discussion:

[
  {"id":"post-example","parentId":null,"threadId":"post-example","title":"How should a team review alerts?","text":"We want fewer duplicate notifications.","permalink":"https://example.com/thread","createdAt":"2026-10-22T09:00:00Z"},
  {"id":"reply-example","parentId":"post-example","threadId":"post-example","title":"","text":"Keep a stable event ID at the destination.","permalink":"https://example.com/thread#reply","createdAt":"2026-10-22T09:05:00Z"}
]

Make bounded, self-contained chunks

This standard-library Python example repeats the title and source metadata on each chunk. The limit is in characters, not model tokens. A production pipeline should use the tokenizer and limits of its chosen embedding model.

import hashlib
import json
import re

FENCE = chr(96) * 3

def plain(value):
    # Exported content is untrusted. Prevent it from closing our fence.
    return str(value or "").replace(FENCE, "~~~").replace("\r\n", "\n")

def split_text(text, limit=1200):
    if limit < 1:
        raise ValueError("limit must be positive")
    text = plain(text).strip()
    while text:
        cut = min(limit, len(text))
        if cut < len(text):
            boundary = text.rfind(" ", 0, cut)
            if boundary > limit // 2:
                cut = boundary
        yield text[:cut].strip()
        text = text[cut:].lstrip()

def chunks(records):
    titles = {r["id"]: r.get("title", "") for r in records}
    for record in records:
        body = record.get("text", "")
        title = titles.get(record.get("threadId"), "") or record.get("title", "")
        for index, part in enumerate(split_text(body)):
            # Content hash changes when the normalized chunk changes.
            identity = f'{record["id"]}:{index}'
            digest = hashlib.sha256(part.encode("utf-8")).hexdigest()
            metadata = {k: record.get(k) for k in
                        ("id", "threadId", "parentId", "permalink", "createdAt")}
            metadata.update(chunkId=identity, contentHash=digest, part=index)
            document = (
                f'# {plain(title)}\n\n'
                f'{FENCE}json\n{json.dumps(metadata, ensure_ascii=False)}\n{FENCE}\n\n'
                f'## Quoted source content\n\n{FENCE}text\n{part}\n{FENCE}\n'
            )
            yield {"id": identity, "metadata": metadata, "markdown": document}

if __name__ == "__main__":
    with open("threads.json", encoding="utf-8") as stream:
        records = json.load(stream)
    with open("chunks.jsonl", "w", encoding="utf-8") as stream:
        for item in chunks(records):
            stream.write(json.dumps(item, ensure_ascii=False) + "\n")

The fenced source section is a presentation convention, not a security boundary. When these documents reach an LLM, treat their contents as untrusted data. A source might contain instructions that conflict with your application. Keep system instructions outside the retrieved text and do not give retrieved content authority over tool use.

Handle updates and deletions deliberately

Use chunkId as an upsert key and contentHash to avoid unnecessary embedding work. If a shorter revision creates fewer chunks, remove stale trailing chunks for that record. A missing item in a partial export is not proof of deletion; reconcile only against an authoritative status or a defined retention policy. Keep an inventory of emitted chunk IDs for each source record.

Store parentId as metadata so retrieval can fetch nearby context. A reply such as “that worked” is not meaningful without its parent. Avoid concatenating an entire long thread into one document merely to preserve order. Retrieve a bounded parent or sibling window after finding an initial match.

Evaluate retrieval before adding more data

Write a few questions with known supporting passages. Check whether retrieval returns the right passage and whether the answer links to its source. Include a question the export cannot answer, so abstention is tested too. More chunks do not automatically produce better answers.

This script prepares Markdown and JSONL only. It does not create embeddings, validate all possible export schemas, or provide deletion synchronization. Those should be explicit later steps, selected for the actual application and its data permissions.