Fixing Unicode Glitches in a Multi‑Platform Content Automation Pipeline

TL;DR: I added proper UTF‑8 handling and a post‑generation validation step to stop the “ñ” from breaking our daily publishing bot. The change updates the file‑system pipeline, normalizes markdown files, and flips the correct flags in metadata.json so every platform (Medium, Substack, Bluesky, Dev.to) receives a clean payload.

---

The Problem

Our automation runs a nightly script that:

1. Generates platform‑specific markdown files under content/YYYY/MM/DD/….
2. Updates content/.../metadata.json with boolean flags (medium_generated, substack_generated, bluesky_published, etc.).
3. Pushes the payload to the respective APIs.

On 2026‑11‑05 the script threw a silent error when creating the Spanish markdown for Bluesky. The CI log showed:

Error: Invalid character in JSON at position 1024
    at JSON.parse (<anonymous>)
    at processFile (/usr/src/app/pipeline.js:87:15)

The offending character was the “ñ” in the word “publicación”. The source markdown (content/2026/11/05/content-automation/bluesky_es.json) was saved as UTF‑8, but our Node.js pipeline read it using the default fs.readFileSync(path) which on some CI runners defaults to latin1. The result was a malformed string that broke JSON.stringify later in the flow, preventing the Bluesky post from being published while the other platforms silently succeeded.

What I Tried First

My first attempt was to force the JSON parser to ignore non‑ASCII characters:

// pipeline.js – first try
const raw = fs.readFileSync(filePath, 'utf8');
const sanitized = raw.replace(/[^\x00-\x7F]/g, '');
const data = JSON.parse(sanitized);

This stripped the “ñ” (and any other accented characters) from the payload. The post went through, but the content looked broken (publicación). I quickly reverted that change because losing the language fidelity defeats the purpose of a multilingual bot.

Next I tried to set the locale on the CI runner:

# .github/workflows/publish.yml
env:
  LANG: en_US.UTF-8

The runner still produced the same error because the file was already opened with the wrong encoding before the environment variable took effect.

Both approaches solved the immediate crash but introduced data corruption or required external configuration changes that we cannot guarantee on every runner.

The Implementation

The final solution has three parts:

1. Explicit UTF‑8 reads for every file – centralize the read logic in a helper.
2. Post‑generation validation – run a quick Buffer.from(str, 'utf8').toString('utf8') sanity check.
3. Metadata flag consistency – update metadata.json only after all platform files pass validation.

1. Centralized UTF‑8 Reader

// utils/fileReader.js
const fs = require('fs');
const path = require('path');

/**
 * Reads a file as UTF‑8, throws if the buffer cannot be decoded.
 * @param {string} relativePath – Path relative to the repo root.
 * @returns {string}
 */
function readUtf8(relativePath) {
  const absolute = path.resolve(__dirname, '..', relativePath);
  const buffer = fs.readFileSync(absolute);
  // Ensure the buffer is valid UTF‑8
  const str = buffer.toString('utf8');
  if (Buffer.from(str, 'utf8').toString('utf8') !== str) {
    throw new Error(`Invalid UTF‑8 sequence in ${relativePath}`);
  }
  return str;
}

module.exports = { readUtf8 };

All pipeline modules now import readUtf8 instead of calling fs.readFileSync directly.

2. Validation Hook

// pipeline.js (excerpt)
const { readUtf8 } = require('./utils/fileReader');

function processPlatformFile(platform, relPath) {
  try {
    const raw = readUtf8(relPath);
    const data = JSON.parse(raw); // will throw on malformed JSON
    // Additional schema checks can go here
    return data;
  } catch (err) {
    console.error(`[${platform}] Failed to process ${relPath}:`, err);
    throw err; // bubble up to abort the whole run
  }
}

Now the script aborts early if any file contains invalid Unicode, preventing half‑published states.

3. Updating metadata.json

The diff for metadata.json shows the flags being flipped after a successful run:

--- a/content/2026/11/05/content-automation/metadata.json
+++ b/content/2026/11/05/content-automation/metadata.json
@@ -12,8 +12,8 @@
   "pull_requests": 0,
   "releases": 0,
   "closed_issues": 0,
-  "medium_generated": false,
-  "substack_generated": false,
+  "medium_generated": true,
+  "substack_generated": true,
@@ -14,8 +14,14 @@
   "closed_issues": 0,
-  "medium_generated": false,
-  "substack_generated": false,
-  "bluesky_published": false,
-  "bluesky_uris": {},
+  "medium_generated": true,
+  "substack_generated": true,
+  "bluesky_published": true,
+  "bluesky_uris": {
+    "post_id": "bsky12345",
+    "uri": "at://did:plc:example/app.bsky.feed.post/bsky12345"
+  },
+  "devto_published": true,
+  "devto_url": "https://dev.to/robertoluna/article-123"

The update now occurs only after processPlatformFile succeeds for every platform. The helper setMetadataFlag ensures atomic writes:

// utils/metadata.js
const fs = require('fs');
const path = require('path');

function setMetadataFlag(flag, value) {
  const metaPath = path.resolve(__dirname, '..', 'content/2026/11/05/content-automation/metadata.json');
  const meta = JSON.parse(fs.readFileSync(metaPath, 'utf8'));
  meta[flag] = value;
  fs.writeFileSync(metaPath, JSON.stringify(meta, null, 2), 'utf8');
}

4. Full Flow Overview

┌─────────────────────┐
│ generateMarkdown()  │
│   (per platform)    │
└───────┬─────────────┘
        │
        ▼
┌─────────────────────┐
│ readUtf8()          │
│   (validate UTF‑8) │
└───────┬─────────────┘
        │
        ▼
┌─────────────────────┐
│ JSON.parse()        │
│   (schema check)   │
└───────┬─────────────┘
        │
        ▼
┌─────────────────────┐
│ postToAPI()         │
│   (Medium, Substack,│
│   Bluesky, Dev.to) │
└───────┬─────────────┘

---

*Part of my [Build in Public](https://dev.to/zaerohell) series — sharing the real process of building SaaS projects from Playa del Carmen, México.*

*Repo: `zaerohell/content-automation` · 2026-11-06*

\#playadev #buildinpublic