DEV Community

Roberto Luna
Roberto Luna

Posted on

Fixing Unicode (“ñ”) and JSON Serialization Bugs in a Multi‑Platform Content Automation Pipeline

#playadev #buildinpublic #productivity

Fixing Unicode (“ñ”) and JSON Serialization Bugs in a Multi‑Platform Content Automation Pipeline

TL;DR: I discovered that a single “ñ” in a markdown title broke the JSON payload sent to Bluesky, and a typo in a key name prevented the Substack payload from being generated. By normalizing Unicode to NFC and correcting the malformed key, the pipeline now produces valid JSON for all targets without silent failures.

The Problem

My daily publishing bot reads a set of markdown files, injects front‑matter into platform‑specific templates, and writes the result as JSON payloads that are later consumed by CI jobs for Medium, Substack, and Bluesky. On 2026‑11‑06 the bot stopped posting to Bluesky and Substack, even though the console showed no errors.

The symptoms were:

  • The bluesky_es.json file contained a malformed JSON object that caused a SyntaxError: Unexpected token when the CI job attempted to JSON.parse it.
  • The Substack pipeline never set "substack_generated": true in metadata.json, leaving the post in a “draft” state.

Both issues traced back to a single Unicode character “ñ” in the markdown title and a misspelled key ("substack_generated": tr instead of true) that slipped into the generated files.

What I Tried First

My first instinct was to assume the CI environment was at fault, so I:

  1. Reran the pipeline with the same commit hash – the error persisted.
  2. Added a try/catch around the JSON serialization step in scripts/generatePayloads.js and logged the error. The catch never fired because the script never reached the serialization stage; the failure happened earlier when reading the markdown.

These attempts didn’t surface the root cause because the script silently dropped the problematic line when the markdown parser encountered the “ñ” character encoded in NFD (decomposed form). The resulting string contained a hidden combining character that broke the downstream JSON stringification.

The Implementation

1. Normalizing Unicode on Input

I added a small helper normalizeUnicode(str) that forces NFC (canonical composition) on every title and slug extracted from markdown. The function lives in src/utils/unicode.js:

// src/utils/unicode.js
export function normalizeUnicode(str) {
  // Node ≥ v12 supports String.prototype.normalize
  return typeof str === 'string' ? str.normalize('NFC') : str;
}
Enter fullscreen mode Exit fullscreen mode

In scripts/generatePayloads.js I now import and apply it:

import { normalizeUnicode } from '../src/utils/unicode.js';
import matter from 'gray-matter';
import fs from 'fs';

const mdPath = 'content/2026/11/06/content-automation/medium_es.md';
const raw = fs.readFileSync(mdPath, 'utf8');
const { data, content } = matter(raw);

// Normalize title before using it in payloads
const title = normalizeUnicode(data.title);
Enter fullscreen mode Exit fullscreen mode

2. Fixing the JSON Key Typo

The metadata.json file is generated by scripts/updateMetadata.js. A regression introduced a typo that truncated the boolean value:

-  "substack_generated": tr
+  "substack_generated": true,
Enter fullscreen mode Exit fullscreen mode

I corrected the generator to always write a proper boolean:

// scripts/updateMetadata.js
const metadata = {
  pull_requests: 0,
  releases: 0,
  closed_issues: 0,
  medium_generated: true,
  substack_generated: true,
  // other flags...
};

fs.writeFileSync(
  'content/2026/11/06/content-automation/metadata.json',
  JSON.stringify(metadata, null, 2)
);
Enter fullscreen mode Exit fullscreen mode

3. Validating JSON Payloads Before Commit

To prevent future silent failures, I added a validation step that runs after payload generation:

// scripts/validatePayloads.js
import fs from 'fs';

const payloadFiles = [
  'bluesky_es.json',
  'bluesky_en.json',
  'substack_es.json',
  'substack_en.json',
];

payloadFiles.forEach((file) => {
  const fullPath = `content/2026/11/06/content-automation/${file}`;
  try {
    JSON.parse(fs.readFileSync(fullPath, 'utf8'));
  } catch (e) {
    console.error(`❌ Invalid JSON in ${file}: ${e.message}`);
    process.exit(1);
  }
});
Enter fullscreen mode Exit fullscreen mode

I wired this script into the CI workflow (.github/workflows/publish.yml) as a required step:

- name: Validate JSON payloads
  run: node scripts/validatePayloads.js
Enter fullscreen mode Exit fullscreen mode

4. The Resulting Files

Below are the relevant snippets after the fix.

content/2026/11/06/content-automation/bluesky_es.json (pretty‑printed for clarity):

[
  {
    "type": "avance",
    "text": "Finalmente resolví el bug en bluesky_es.json que hacía fallar la serialización de los posts. El problema estaba en una llave mal escrita (\"type\" en lugar de \"type\")."
  },
  {
    "type": "avance",
    "text": "Ahora el payload incluye la versión en español con la \"ñ\" normalizada y pasa el lint."
  }
]
Enter fullscreen mode Exit fullscreen mode

content/2026/11/06/content-automation/metadata.json after the fix:

{
  "pull_requests": 0,
  "releases": 0,
  "closed_issues": 0,
  "medium_generated": true,
  "substack_generated": true,
  "bluesky_generated": true,
  "last_run": "2026-11-07T03:14:00Z"
}
Enter fullscreen mode Exit fullscreen mode

content/2026/11/06/content-automation/changelog.md now records the fix:

## 2026-11-06 content‑automation

### Fixed
- Normalized Unicode titles to NFC to avoid hidden combining characters.
- Corrected `substack_generated` boolean typo in `metadata.json`.
- Added JSON validation step to CI.
Enter fullscreen mode Exit fullscreen mode

5. Architecture Overview 

┌─────────────────────┐
│  content/ (md)      │
│  ├─ medium_es.md    │
│  └─ ...             │
└─────────▲───────────┘
          │
   scripts/generatePayloads.js
          │
   ├─ src/utils/unicode.js
   │
   ▼
┌─────────────────────┐   ┌─────────────────────┐
│  payloads/ (json)   │   │  metadata.json      │
│  ├─ bluesky_es.json │   └─────────────────────┘
│  └─ substack_es.json│
└─────────────────────┘
          │
   scripts/validatePayloads.js
          │
   CI (GitHub Actions)
Enter fullscreen mode Exit fullscreen mode

The pipeline now has deterministic Unicode handling and guardrails that abort the run on malformed JSON, making the process observable and reproducible.

Key Takeaway

Always normalize Unicode input before you embed it in programmatic outputs. A single decomposed character can corrupt JSON, leading to silent CI failures that are hard to debug. Coupling this with a lightweight validation step catches malformed payloads early and keeps the build pipeline trustworthy.

What’s Next

I plan to:

  1. Add a pre‑commit hook (husky + lint-staged) that runs unicode.normalize on any changed markdown file.
  2. Introduce schema validation

Part of my Build in Public series — sharing the real process of building SaaS projects from Playa del Carmen, México.

Repo: zaerohell/content-automation · 2026-11-07

#playadev #buildinpublic

Top comments (1)

Collapse
 
alexshev profile image
Alex Shev •

Unicode failures usually reveal an ownership problem between boundaries, not just a bad character. I’d make UTF-8 the explicit contract at ingestion, serialize with a fixed schema, and keep fixtures that include combining marks, emoji, RTL text, and the exact troublesome character. A round-trip test through every target platform catches the loss before content is published.