DEV Community

Roberto Luna
Roberto Luna

Posted on

Adding a Structured Metadata Manifest to Content‑Automation for Reliable CI Publishing

Adding a Structured Metadata Manifest to Content‑Automation for Reliable CI Publishing

TL;DR: I introduced a metadata.json file per content batch so the CI pipeline can discover, validate, and route posts without relying on fragile folder naming. This change makes the automation deterministic and ready for multilingual scaling.


The Problem

Our content-automation repo drives a GitHub Actions workflow that reads markdown files, builds front‑matter, and pushes the payload to Dev.to, Bluesky, and a static site generator. The workflow used a directory‑scan approach:

# pseudo‑code used in the old pipeline
for file in content/**/post.md; do
  # infer language from folder name, e.g. content/en/...
  # infer date from folder hierarchy, e.g. content/2026/11/15/
done
Enter fullscreen mode Exit fullscreen mode

When we added a new language (fr) and started grouping posts by topic instead of date, the scanner started missing files and, worse, mis‑routing posts to the wrong platform. The error we saw in the Action logs was:

Error: Unable to determine language for ./content/2026/11/15/post.md
Enter fullscreen mode Exit fullscreen mode

The root cause was that the pipeline assumed a fixed folder structure (content/<year>/<month>/<day>/<lang>/post.md). Any deviation broke the discovery step, leading to silent drops and duplicated posts.


What I Tried First

My first instinct was to normalize the folder names on the fly:

// attempt 1 – normalizePath()
function normalizePath(p) {
  const parts = p.split('/');
  // try to guess language by matching known codes
  const lang = parts.find(part => ['en','es','de'].includes(part));
  // fallback to default 'en'
  return { lang: lang || 'en', date: parts.slice(1,4).join('-') };
}
Enter fullscreen mode Exit fullscreen mode

I added this helper to the loader script and ran the workflow locally. It worked for existing content, but when a post was placed directly under content/2026/11/15/ without a language subfolder, the function defaulted to 'en'. That caused incorrect language tags on the published article, which we caught only after the post went live.

The approach also introduced a maintenance burden: every time we added a new language or changed the folder hierarchy we had to update the heuristic list. It was clear we needed a more explicit contract between the content and the automation.


The Implementation

1. Define a JSON manifest per batch

I created a new file content/2026/11/15/content-automation/metadata.json. The diff from the commit looks like this:

+{
+  "repo": "content-automation",
+  "date": "2026-11-15",
+  "languages": [
+    "es",
+    "en"
+  ],
+  "topics": [
+    "Productivity"
+  ],
+  "commits": 4,
+  "pull_requests": null
+}
Enter fullscreen mode Exit fullscreen mode

The manifest captures everything the pipeline needs:

Field Why it matters
repo Allows the same script to be reused across multiple repos.
date Single source of truth for the batch date (used for front‑matter).
languages Explicit list; no guessing.
topics Drives tag generation on each platform.
commits / pull_requests Optional metrics for analytics dashboards.

2. Update the loader (src/loader.js)

I added a small module that reads the manifest, validates it, and returns a typed object. The file now lives at src/loader.js:

// src/loader.js
const fs = require('fs');
const path = require('path');

function loadMetadata(batchPath) {
  const metaPath = path.join(batchPath, 'metadata.json');
  if (!fs.existsSync(metaPath)) {
    throw new Error(`Metadata file missing at ${metaPath}`);
  }

  const raw = fs.readFileSync(metaPath, 'utf-8');
  let meta;
  try {
    meta = JSON.parse(raw);
  } catch (e) {
    throw new Error(`Invalid JSON in ${metaPath}: ${e.message}`);
  }

  // Minimal validation – keep it lightweight for CI
  const required = ['repo', 'date', 'languages', 'topics'];
  for (const key of required) {
    if (!meta[key]) {
      throw new Error(`Metadata validation error: missing "${key}"`);
    }
  }

  return meta;
}

// Export for the workflow script
module.exports = { loadMetadata };
Enter fullscreen mode Exit fullscreen mode

The CI script (.github/workflows/publish.yml) now calls this loader:

# .github/workflows/publish.yml (excerpt)
- name: Load batch metadata
  id: meta
  run: |
    node -e "const {loadMetadata}=require('./src/loader');
    const meta=loadMetadata('content/2026/11/15/content-automation');
    console.log(JSON.stringify(meta));
    " > /tmp/meta.json
- name: Set output
  run: echo "metadata=$(cat /tmp/meta.json)" >> $GITHUB_OUTPUT
Enter fullscreen mode Exit fullscreen mode

3. Wire the manifest into the publishing logic

In src/publish.js I replaced the heuristic language detection with the explicit list from the manifest:

// src/publish.js (excerpt)
const { loadMetadata } = require('./loader');
const meta = loadMetadata(process.env.BATCH_PATH);

meta.languages.forEach(lang => {
  const mdPath = path.join(process.env.BATCH_PATH, `${lang}.md`);
  if (!fs.existsSync(mdPath)) {
    console.warn(`⚠️ No markdown for language "${lang}" – skipping`);
    return;
  }

  const content = fs.readFileSync(mdPath, 'utf-8');
  const frontMatter = `---
title: "My Post"
date: ${meta.date}
tags: ${meta.topics.join(', ')}
lang: ${lang}
---\n`;

  // Build payload for each platform
  const payload = frontMatter + content;
  // sendPayload(payload, platform);
});
Enter fullscreen mode Exit fullscreen mode

4. Add a tiny schema test (future‑proofing)

While not required for the first release, I committed a


Part of my Build in Public series — sharing the real process of building SaaS projects from Playa del Carmen, México.

Repo: zaerohell/content-automation · 2026-11-16

#playadev #buildinpublic

Top comments (0)