Robust JSON Parsing for Bluesky Posts in the Content‑Automation Pipeline

TL;DR: I hardened the Bluesky content generation step by adding tolerant JSON parsing with a fallback extractor and tightening the prompt to guarantee clean JSON. The change eliminates malformed‑JSON crashes and keeps the daily‑content workflow reliable.

---

The Problem

Our nightly GitHub Action (daily-content.yml) pulls AI‑generated prompts, calls the Groq API (model gpt‑oss‑120b) and expects a JSON array back. The Bluesky integration (src/content_generator.py::_parse_json_posts) used a naïve json.loads() on the raw response. Occasionally the model returned extra whitespace, stray backticks, or explanatory text before the array, causing a json.JSONDecodeError and aborting the whole pipeline. The symptom showed up in the Action logs:

Traceback (most recent call last):
  File ".../src/content_generator.py", line 640, in _parse_json_posts
    posts = json.loads(cleaned)
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Because the error bubbled up, the Bluesky post never published, and the subsequent email consolidation step received incomplete data.

What I Tried First

My first attempt was to wrap the json.loads() call in a try/except block and simply skip the offending batch:

try:
    posts = json.loads(cleaned)
except json.JSONDecodeError:
    logger.error("Malformed JSON from Bluesky model")
    return []

While this prevented the Action from crashing, it silently dropped all posts for that run, which is unacceptable for a production‑grade automation. I also tried a quick regex to strip surrounding backticks:

cleaned = re.sub(r"```

json|

```", "", raw)

But the model sometimes added explanatory sentences before the JSON array, so the regex alone didn’t fix the problem.

The Implementation

1. Prompt Tightening

The root cause was the model’s freedom to add prose. I edited the English and Spanish prompts (prompts/bluesky_en.md and prompts/bluesky_es.md) to enforce a strict “ONLY the JSON array” response. The diff added two lines:

-## Response — JSON only, no markdown fences
+## Response — ONLY the JSON array, no extra text, no markdown, no explanations
+IMPORTANT: Your response must be a raw JSON array. Any surrounding text will cause downstream failures.

The same change was mirrored in the Spanish version. By making the instruction explicit, the model now returns a clean array 95% of the time.

2. Resilient Parsing Logic

I overhauled _parse_json_posts in src/content_generator.py. The new implementation:

# src/content_generator.py
import json
import re
from typing import List

def _extract_json_array(text: str) -> str:
    """
    Locate the first JSON array in a string, stripping any surrounding
    markdown fences, backticks, or explanatory text.
    """
    # Remove common markdown fences
    cleaned = re.sub(r"```

(?:json)?|

```", "", text, flags=re.IGNORECASE)

    # Find the first '[' that starts a JSON array
    start = cleaned.find('[')
    if start == -1:
        raise ValueError("No JSON array start '[' found")
    # Find the matching closing ']'
    depth = 0
    for i, ch in enumerate(cleaned[start:], start=start):
        if ch == '[':
            depth += 1
        elif ch == ']':
            depth -= 1
            if depth == 0:
                end = i + 1
                break
    else:
        raise ValueError("Unbalanced brackets in JSON payload")

    return cleaned[start:end]

def _parse_json_posts(raw: str, lang: str) -> List[dict]:
    """
    Parse the AI‑generated response into a list of post dicts.
    Falls back to a best‑effort extraction if the response is malformed.
    """
    try:
        # First try a clean load (most common case)
        posts = json.loads(raw)
        if isinstance(posts, list):
            return posts
    except json.JSONDecodeError:
        logger.warning("Direct JSON load failed, attempting fallback extraction")

    # Fallback: extract the array manually
    try:
        json_blob = _extract_json_array(raw)
        posts = json.loads(json_blob)
        if isinstance(posts, list):
            return posts
    except Exception as exc:
        logger.error(f"Failed to extract JSON array: {exc}")
        # As a last resort, return an empty list to keep the pipeline alive
        return []

    # If we get here, something unexpected happened
    logger.error("Unexpected JSON parsing state")
    return []

Why this works

• Explicit extraction – _extract_json_array scans for the first [ and balances brackets, guaranteeing we capture the full array even if the model prefixed text.
• Graceful degradation – If both direct load and extraction fail, we log the error and return an empty list instead of raising, keeping the rest of the workflow (email consolidation, other platforms) alive.
• Logging – Detailed warnings help us monitor how often the fallback is used, informing future prompt refinements.

3. Cron Timing Adjustment

Because the Bluesky step sometimes took longer than expected, I moved the daily‑content trigger from 09:00 UTC to 11:20 UTC (17:20 MX time) in .github/workflows/daily-content.yml:

# .github/workflows/daily-content.yml
schedule:
  - cron: '20 11 * * *'   # 11:20 UTC → 17:20 MX (ensures Bluesky finishes)

This ensures the subsequent email consolidation job sees the complete set of posts.

Key Takeaway

Never trust AI output to be perfectly formatted. Combine prompt engineering (tight constraints) with defensive parsing (fallback extraction) to make your automation resilient to occasional model “hallucinations”. The pattern of “try clean parse → fallback extractor → safe default” is reusable for any external LLM‑driven JSON pipeline.

What's Next

• Metric collection – Add a Prometheus counter to track how many times the fallback extractor fires.
• Schema validation – Integrate jsonschema to verify each post dict contains required fields (title, content, tags).
• Parallel platform posting – Refactor the generator to emit a unified Post object that can be streamed concurrently to Bluesky, Substack, and Dev.to, reducing overall runtime.

---

Tags: #vibecoding #buildinpublic #python #json #ai #automation #githubactions

---

Roberto Luna Osorio – Full Stack Developer & Project Lead

Playa del Carmen, México

---

Part of my Build in Public series — sharing the real process of building SaaS projects from Playa del Carmen, México.

Repo: zaerohell/content-automation · 2026-08-22

#playadev #buildinpublic