I Fixed LLM Markdown Errors with Jinja2 and AST Parsing

Stop Fighting Prompts: How I Reduced Formatting Errors to 0.1%

LLMs are great at generating content, but terrible at keeping it clean. In the ai-developer-knowledge-hub project, we faced a recurring nightmare: the technical documents generated by the LLM were riddled with formatting issues. Specifically, code blocks often lacked closing markers or had unclosed strings, crashing our frontend rendering engine.

We tried the obvious route: optimizing the Prompt. We begged the model to "output correct markdown syntax." The result? A 15% error rate. That's unacceptable for an automated publishing pipeline.

The core challenge is bridging the gap between a probabilistic system (the LLM) and a deterministic requirement (valid Markdown). Direct Regex cleaning was too fragile, and letting the LLM self-correct led to infinite loops.

The Root Cause

1. Symptom: Missing closing backticks or quotes in code blocks break the Markdown structure.
2. Mechanism: Relying on Prompts is a "soft constraint." The model follows syntax rules probabilistically, not deterministically.
3. Gap: We lacked a structured intermediate layer. We were treating raw streaming text as the final product, letting errors slip through.
4. The Breaking Point: A missing } in a JSON config block once threw a TemplateSyntaxError in Jinja2, blocking the entire publishing pipeline.

The Solution: AST Parsing & Jinja2 Hard Rendering

The breakthrough was decoupling content generation from style rendering. Instead of trusting the raw text, we pipe it through a validation layer using AST (Abstract Syntax Tree) parsing.

If the AST check fails, we sanitize. If it passes, we extract structured blocks and feed them into a Jinja2 template. This ensures the output structure is 100% locked down by the template engine, not guessed by the LLM.

Here is the implementation:

# Before: Relying on Prompt engineering (fragile)
prompt = "Please output markdown code blocks with correct syntax."
raw_text = llm.generate(prompt)

# After: Pipeline processing with forced validation
def render_pipeline(llm_output: str) -> str:
    # 1. AST Syntax Check (catches missing closing quotes/markers)
    try:
        markdown_parser.parse(llm_output)
    except SyntaxError:
        return fallback_sanitize(llm_output)

    # 2. Structured extraction and cleaning
    content_blocks = extract_code_blocks(llm_output)

    # 3. Jinja2 hard constraint rendering
    template = jinja_env.get_template("article_layout.md")
    return template.render(blocks=content_blocks)

Production-Grade Fallback & Retry

Parsing can fail, and LLMs can hang. We needed a strategy that prioritizes content delivery over perfection. We implemented an exponential backoff retry mechanism with a "text-only" fallback.

If rendering fails after retries, we don't crash; we strip the formatting and serve the raw text. Content is king, but we also log 10% of these failures for debugging without exploding our storage costs.

# Before: Simple retry, no circuit breaker
for _ in range(3):
    result = generate_and_check()

# After: Exponential backoff + Hard fallback + Sampling logs
MAX_RETRIES = 2
TIMEOUT = 5.0  # seconds
LOG_SAMPLE_RATE = 0.1  # 10% error sampling rate

for attempt in range(MAX_RETRIES):
    try:
        return strict_render(llm_output, timeout=TIMEOUT)
    except ASTParseError as e:
        if attempt == MAX_RETRIES - 1:
            # Last retry failed: downgrade to plain text, keep content, drop format
            if random.random() < LOG_SAMPLE_RATE:
                logger.error(f"Render failed: {e}")
            return text_only_fallback(llm_output)
        time.sleep(2 ** attempt) # Exponential backoff

Key Takeaways

• Hard Constraints > Soft Constraints: Engineering determinism beats prompt engineering. Jinja2 guarantees structure, bringing error rates from 15% down to 0.1%.
• Fail Gracefully: When AST parsing fails, never let the pipeline crash. Output a sanitized text version and log it for later review.
• Timeouts are Non-Negotiable: LLM outputs or parsing can block indefinitely. A 5-second timeout fuse prevents the whole document pipeline from stalling.

By moving the formatting responsibility from the LLM to a deterministic rendering pipeline, we solved the reliability issue once and for all.