At the end of the previous part, we were sitting on a perfectly working, fully tested pipeline. Our converter successfully chewed through Doxygen XML and generated custom HTML pages for our API documentation. The team was already getting ready to pop the champagne and push everything to production when the Infrastructure Lead walked over. He asked one simple, yet absolutely devastating question: "Why are you still messing around with HTML generation when off-the-shelf tools like Hugo, Docusaurus, or VitePress exist?"
The Pure HTML Trap
It took us exactly a minute to realize the sheer scale of our mistake. Generating pure HTML sounds great right up until you try to build a modern developer portal around it. It turned out we were about to reinvent the wheel for practically every standard web feature. Global search would require writing our own indexer. For navigation, we'd have to manually maintain a consistent sidebar across hundreds of generated pages, and synchronizing our styles with the corporate CSS threatened to turn into an endless chore. Instead of a documentation system, we had somehow started writing our own web framework.
The SSG Awakening
Static Site Generators (SSGs) like Hugo or Docusaurus solve all these problems out of the box. They come with built-in search, routing, and plugin systems. The only catch was that they strictly eat Markdown as input. We had to make a sharp pivot and re-engineer our pipeline from Doxygen XML ➡️ HTML to Doxygen XML ➡️ Markdown.
Rewriting the output layer was surprisingly painless. Even during the HTML generation phase, we had strictly separated the parsing and rendering logic, introducing an intermediate data layer we called the Intermediate Representation (or simply IR). This guaranteed the parser's stability while we ripped out the HTML generator and bolted on Markdown instead. And the practice of TDD definitely played its part—thanks to our tests, the Markdown generator was written quickly and confidently. As we worked on the new exporter, the puzzle pieces suddenly clicked together.
The Engine Singularity
Our architecture now consisted of two isolated phases. First, the Parser reads the source code and builds an abstract tree (that same IR), and then the Renderer takes the finished tree and translates it into Markdown. That’s when it hit us: the renderer absolutely does not care where this data came from.
We recalled the situation from the second article, when the AI tried to throw out Doxygen and started parsing the Python code directly via tree-sitter. Back then, we panicked about losing control and rolled back to parsing XML (even though the AI offered a working solution). Now, with a strict TDD contract and the intermediate IR layer in place, there was nothing to fear. We could safely ditch Doxygen and pull data directly from the source code.
Using an IR also came with the bonus of caching and faster generation times. If we fully build the tree in memory on the first run, there is no reason to rebuild it entirely on the next pass. We can simply use incremental parsing and update only the chunks that actually changed. And if we can read Python directly, nothing stops us from reading our main C++ SDK and our other languages. Or maybe even—dare I say it—languages that aren't ours.
This was the exact moment our highly specialized script mutated into a full-fledged Universal Documentation Engine (UDE). We did have to lengthen the name a bit due to an existing trademark for a German debugger called Universal Debug Engine. We tacked on the characteristics Fast and Layered, and thus Flude was born. Now we can plug any parser into the frontend, and the pipeline's backend will assemble beautiful Markdown for our SSG (complete with unified search and a proper design).
From a Local Task to a Massive Project
A tiny local utility for a couple of Python modules suddenly gave us the chance to design a scalable architecture from scratch. Building a product like this meant we could no longer rely on manual builds or running scripts locally on our laptops. We needed a fully automated CI/CD process.
I didn't have to figure this out by myself—my AI teammate took on the lion's share of the setup work. In the next episode, I'll explain why we decided to build our pipelines specifically on GitHub Actions.
Originally published on our blog: https://blog.flude.guide/blog/the-markdown-singularity
Also read us:
Top comments (0)