Adding a Structured Metadata Manifest to Content‑Automation for Reliable CI Publishing
TL;DR: I introduced a metadata.json file per content batch so the CI pipeline can discover, validate, and route posts without relying on fragile folder naming. This change makes the automation deterministic and ready for multilingual scaling.
The Problem
Our content-automation repo drives a GitHub Actions workflow that reads markdown files, builds front‑matter, and pushes the payload to Dev.to, Bluesky, and a static site generator. The workflow used a directory‑scan approach:
# pseudo‑code used in the old pipeline
for file in content/**/post.md; do
# infer language from folder name, e.g. content/en/...
# infer date from folder hierarchy, e.g. content/2026/11/15/
done
When we added a new language (fr) and started grouping posts by topic instead of date, the scanner started missing files and, worse, mis‑routing posts to the wrong platform. The error we saw in the Action logs was:
Error: Unable to determine language for ./content/2026/11/15/post.md
The root cause was that the pipeline assumed a fixed folder structure (content/<year>/<month>/<day>/<lang>/post.md). Any deviation broke the discovery step, leading to silent drops and duplicated posts.
What I Tried First
My first instinct was to normalize the folder names on the fly:
// attempt 1 – normalizePath()
function normalizePath(p) {
const parts = p.split('/');
// try to guess language by matching known codes
const lang = parts.find(part => ['en','es','de'].includes(part));
// fallback to default 'en'
return { lang: lang || 'en', date: parts.slice(1,4).join('-') };
}
I added this helper to the loader script and ran the workflow locally. It worked for existing content, but when a post was placed directly under content/2026/11/15/ without a language subfolder, the function defaulted to 'en'. That caused incorrect language tags on the published article, which we caught only after the post went live.
The approach also introduced a maintenance burden: every time we added a new language or changed the folder hierarchy we had to update the heuristic list. It was clear we needed a more explicit contract between the content and the automation.
The Implementation
1. Define a JSON manifest per batch
I created a new file content/2026/11/15/content-automation/metadata.json. The diff from the commit looks like this:
+{
+ "repo": "content-automation",
+ "date": "2026-11-15",
+ "languages": [
+ "es",
+ "en"
+ ],
+ "topics": [
+ "Productivity"
+ ],
+ "commits": 4,
+ "pull_requests": null
+}
The manifest captures everything the pipeline needs:
| Field | Why it matters |
|---|---|
repo |
Allows the same script to be reused across multiple repos. |
date |
Single source of truth for the batch date (used for front‑matter). |
languages |
Explicit list; no guessing. |
topics |
Drives tag generation on each platform. |
commits / pull_requests
|
Optional metrics for analytics dashboards. |
2. Update the loader (src/loader.js)
I added a small module that reads the manifest, validates it, and returns a typed object. The file now lives at src/loader.js:
// src/loader.js
const fs = require('fs');
const path = require('path');
function loadMetadata(batchPath) {
const metaPath = path.join(batchPath, 'metadata.json');
if (!fs.existsSync(metaPath)) {
throw new Error(`Metadata file missing at ${metaPath}`);
}
const raw = fs.readFileSync(metaPath, 'utf-8');
let meta;
try {
meta = JSON.parse(raw);
} catch (e) {
throw new Error(`Invalid JSON in ${metaPath}: ${e.message}`);
}
// Minimal validation – keep it lightweight for CI
const required = ['repo', 'date', 'languages', 'topics'];
for (const key of required) {
if (!meta[key]) {
throw new Error(`Metadata validation error: missing "${key}"`);
}
}
return meta;
}
// Export for the workflow script
module.exports = { loadMetadata };
The CI script (.github/workflows/publish.yml) now calls this loader:
# .github/workflows/publish.yml (excerpt)
- name: Load batch metadata
id: meta
run: |
node -e "const {loadMetadata}=require('./src/loader');
const meta=loadMetadata('content/2026/11/15/content-automation');
console.log(JSON.stringify(meta));
" > /tmp/meta.json
- name: Set output
run: echo "metadata=$(cat /tmp/meta.json)" >> $GITHUB_OUTPUT
3. Wire the manifest into the publishing logic
In src/publish.js I replaced the heuristic language detection with the explicit list from the manifest:
// src/publish.js (excerpt)
const { loadMetadata } = require('./loader');
const meta = loadMetadata(process.env.BATCH_PATH);
meta.languages.forEach(lang => {
const mdPath = path.join(process.env.BATCH_PATH, `${lang}.md`);
if (!fs.existsSync(mdPath)) {
console.warn(`⚠️ No markdown for language "${lang}" – skipping`);
return;
}
const content = fs.readFileSync(mdPath, 'utf-8');
const frontMatter = `---
title: "My Post"
date: ${meta.date}
tags: ${meta.topics.join(', ')}
lang: ${lang}
---\n`;
// Build payload for each platform
const payload = frontMatter + content;
// sendPayload(payload, platform);
});
4. Add a tiny schema test (future‑proofing)
While not required for the first release, I committed a
Part of my Build in Public series — sharing the real process of building SaaS projects from Playa del Carmen, México.
Repo: zaerohell/content-automation · 2026-11-16
#playadev #buildinpublic
Top comments (0)