Twenty Years of Pandoc: The Architecture Behind a Universal Document Converter

On August 3, 2006, philosophy professor John MacFarlane published the first version of Pandoc. It was roughly 3,000 lines of Haskell, relied only on the Glasgow Haskell Compiler’s standard library, and converted among a small set of text formats: Markdown, reStructuredText, HTML, and LaTeX, with RTF and S5 as additional outputs.

Twenty years later, Pandoc supports 51 input formats and 76 output formats. That is 3,876 possible input-output pairs before accounting for extensions and configuration. It sits underneath academic publishing systems, notebooks, static-site workflows, documentation pipelines, ebooks, slide decks, and countless one-off shell commands. More than 600 people have contributed, and the core packages contain about 85,000 lines of Haskell excluding tests.

Those numbers are impressive, but they are not the main story. Pandoc lasted because its first important design decision made future growth cheaper. Instead of writing a separate converter for every pair of formats, it parses a document into a shared representation, transforms that representation, and renders it into the target format.

That sounds obvious now. In 2006, for Markdown tooling, it was not.

The Experiment That Found a Real Problem

Pandoc did not begin with a market analysis or a grand plan for universal publishing. MacFarlane wanted to learn Haskell. He saw that the language was well suited to parsers and compilers, especially through the Parsec parser-combinator library, and chose a Markdown parser as a practical exercise.

Most early Markdown implementations worked by applying a sequence of regular-expression substitutions directly to the source. That approach can be compact, but markup is structured and context-sensitive. A heading marker inside a code block is not a heading. Emphasis delimiters interact with nesting. Lists, links, raw HTML, and escaping create cases that are difficult to model as a chain of text replacements.

Pandoc instead treated a document like a compiler treats a program:

source document -> reader -> document tree -> writer -> output document

The document tree is an abstract syntax tree, or AST. It represents meaning rather than source punctuation: this node is a paragraph, that one is emphasized text, another is a link, a table, a citation, or a code block. Once a reader has produced that structure, a writer does not need to understand the original syntax. It only needs to render the tree in its own format.

This changes the scaling problem. Pairwise converters grow roughly with every combination of source and target. With a shared AST, adding one reader makes every existing writer available to that input; adding one writer gives every existing reader a new destination. In simplified terms, N readers plus M writers enable N × M conversions.

MacFarlane soon added reStructuredText because he used it for lecture notes, LaTeX because he wanted PDFs, and a Markdown writer because he wanted to migrate those notes. The learning exercise had turned into a tool that removed friction from his own work. That tight feedback loop—build something, use it, discover the next constraint—would guide the project for years.

Distribution Mattered as Much as Code

The first public release received almost no promotion. MacFarlane emailed two friends. There was no GitHub, Hackage was not yet available, and he did not use social media.

The first big distribution event came from someone else. A Turkish developer, Recai Oktaş, contacted him about packaging Pandoc for Debian. Working through Debian’s standards improved the software and put it in front of users who would never have discovered a personal website. In 2007, Hackage and cabal-install made Haskell packages easier to publish, install, and compose. Pandoc could now take dependencies on reusable libraries instead of keeping everything inside one repository.

The pattern is worth noticing: useful software does not spread through implementation quality alone. Package managers, operating-system repositories, stable releases, documentation, and boring installation paths are product features. Pandoc’s architecture made it capable; distribution made it available.

Pandoc 1: From Converter to Publishing System

Pandoc 1.0 arrived in September 2008. It added outputs for MediaWiki, GNU Texinfo, OpenDocument, and ODT, along with fenced code blocks and syntax highlighting. Some of those features required Haskell libraries that did not exist, so MacFarlane wrote them. zip-archive enabled container formats such as ODT. highlighting-kate turned KDE syntax definitions into Haskell highlighters and gave Pandoc broad language coverage.

This period established a second theme in Pandoc’s history: supporting document formats often means building the missing infrastructure around them. A DOCX file is not merely formatted text; it is a ZIP container containing XML, relationships, media, styles, and specialized equation markup. High-fidelity conversion forces the converter to understand these surrounding systems.

The project moved from Google Code to GitHub in 2010. A template system replaced hard-coded wrappers, letting users control complete output documents. EPUB, Org mode, Textile, MathML, and other formats followed. Pandoc 1.9 added DOCX output in 2012, including Word’s Office Math Markup Language for equations.

By 2013, Pandoc was becoming a platform rather than a fixed conversion matrix. Four changes were especially important:

• Markdown extensions made dialect choices explicit instead of pretending there was one universal Markdown.
• YAML metadata blocks carried structured document-level information into templates.
• Custom Lua writers let users produce niche output formats without modifying Pandoc itself.
• JSON filters let any program transform the AST between parsing and rendering.

The filter model is a direct consequence of the shared representation:

input -> reader -> AST -> filter -> AST -> writer -> output

A filter can number sections, rewrite links, execute selected code blocks, generate diagrams, remove content, or enforce a house style without caring whether the source was Markdown and the output is HTML, DOCX, or LaTeX. It operates on document meaning, not on fragile surface syntax.

Markdown’s Ambiguity and the CommonMark Detour

While maintaining Pandoc’s Markdown parser, MacFarlane encountered a deeper problem: Markdown’s original description left many edge cases undefined. Implementations disagreed about nested emphasis, list indentation, code blocks, escaping, and how constructs interact.

Beginning in 2012, he joined a group with representatives from GitHub, Reddit, Stack Overflow, and others to create an unambiguous specification. When the group lost momentum, he wrote a formal specification and reference parsers. The effort became CommonMark after John Gruber objected to the original name, “Standard Markdown.”

CommonMark gave implementers a testable core instead of an informal description. Most modern Markdown processors now share its basic rules, even when they add different extensions.

There is an amusing wrinkle: Pandoc still keeps its older, extension-rich Markdown parser alongside CommonMark-based formats. Users can choose markdown, commonmark, gfm, or commonmark_x depending on the behavior they need. A project can help standardize its ecosystem without immediately replacing every piece of its own history.

Pandoc 2: Effects, Filters, and Rich Documents

Pandoc 2.0, released in 2017, addressed a tension in the original model. Pure readers and writers are easy to reason about, but some formats require input and output. A reStructuredText document can include another file. A writer may need to fetch an image and inspect its dimensions. DOCX and EPUB contain embedded resources.

The solution was not to let I/O spread invisibly through the codebase. Readers and writers were generalized over a PandocMonad interface. One implementation permits controlled I/O; another remains pure. The abstraction later enabled Pandoc’s --sandbox mode, which guarantees that readers and writers cannot touch the filesystem or network. A design introduced to handle richer conversions also became a security boundary.

Pandoc 2.0 also embedded Lua filters. JSON filters are language-neutral, but they serialize a whole AST and launch another process. Lua filters operate directly on the tree inside Pandoc, require no separate runtime, and are generally faster. This made custom document transformations practical for ordinary users, not just Haskell developers.

The format list continued growing: GitHub-flavored Markdown, JATS, PowerPoint, Vimwiki, TikiWiki, Creole, and more. Jupyter Notebook support arrived in 2019. Defaults files made repeatable command configurations easier to store and review.

Some work was less visible but more disruptive. Pandoc’s original table model could not express row spans or column spans. Fixing that meant redesigning the AST type and updating every reader and writer that handled tables. This is where Haskell’s algebraic data types and compiler checks paid off: when a central data structure changes, the compiler identifies code that has not been adapted.

Citation processing followed a similar path. Pandoc had long supported bibliographies through an external filter, but the implementation had become slow and difficult to maintain. MacFarlane wrote a new CSL processor from the specification and test suite. It became built-in with Pandoc 2.11, improving performance and standards fidelity. Correct sorting required yet another supporting library implementing the Unicode Collation Algorithm.

By the end of the Pandoc 2 era, users could also write custom readers in Lua, run Pandoc as an HTTP service, convert bibliography databases, and use sandboxed parsing and rendering.

Pandoc 3: Splitting the Monolith Without Breaking the Model

By 2023, Pandoc had accumulated a web server, a Lua runtime, dozens of format implementations, citation machinery, templates, highlighting, and resource handling. Not every user wanted the full binary.

Pandoc 3.0 split the system into four packages:

• pandoc provides the core Haskell library.
• pandoc-lua-engine contains Lua integration.
• pandoc-server exposes conversion through HTTP.
• pandoc-cli builds the command-line application and can omit optional components.

The architecture remained recognizable. Readers still produce the shared AST; writers still consume it. The package boundaries reduced the cost of carrying features a deployment did not need.

New formats kept testing the limits of the model. Supporting Typst input required implementing much of an interpreter because Typst is not just markup; it is a programmable typesetting language. Djot, MacFarlane’s attempt to address long-standing Markdown design problems, became both an input and output format. Later releases added terminal-oriented ANSI output, PowerPoint and Excel readers, AsciiDoc input, XML serialization of the AST, and more.

Pandoc 3.9, released in February 2026, added WebAssembly support, making a full-featured converter available in the browser. The deployment environment changed radically from a Debian package and a shell command, but the reader–AST–writer core survived.

Why Haskell Worked for a Twenty-Year Codebase

Pandoc was written in Haskell because its creator wanted to learn Haskell, not because of a formal technology selection. In retrospect, the language matched the problem unusually well.

Algebraic data types provide a precise vocabulary for structured documents. Pattern matching makes transformations over that structure direct. A strong type system helps maintainers change central types safely across a large codebase. Purity makes side effects visible and controllable, which improves testing and supports guarantees such as sandbox mode. Parser combinators make complex grammars composable without hiding their structure in regular-expression pipelines.

None of this means Haskell made maintenance automatic. MacFarlane still spends time reviewing patches, answering questions, fixing tiny incompatibilities, repairing release infrastructure, signing builds, and improving documentation. Language features reduce certain categories of risk; they do not replace stewardship.

Pandoc also demonstrates a social trade-off. A less common language may reduce the number of casual contributors, but it can attract contributors who care deeply about parsers, types, compilers, and formal document structure. For a project with limited management capacity, contributor fit can matter more than raw volume.

What Pandoc Teaches About Durable Tools

Pandoc’s history offers several practical lessons for software intended to last.

Choose a stable internal model. The shared AST allowed formats, filters, and interfaces to grow around one center. The model evolved—figures and rich tables required changes—but it was strong enough to absorb those changes.

Separate meaning from representation. A document heading is not the # character that happens to introduce it in one syntax. Systems become more reusable when they model what data means, then isolate encoding and rendering at the edges.

Create extension points before you can predict every use. JSON filters, Lua filters, templates, defaults files, and custom readers and writers let users solve specialized problems without waiting for them to become core features.

Turn constraints into explicit interfaces. PandocMonad did more than permit I/O; it defined where effects were allowed. That made both full-featured conversion and a meaningful sandbox possible.

Treat packaging and documentation as part of the product. Debian, Hackage, GitHub releases, installers, binaries, manuals, and examples converted good code into usable infrastructure.

Expect maintenance to dominate. New formats make release notes. Compatibility fixes, issue triage, tests, code signing, and careful review keep users trusting the next release. A long-lived tool is mostly a promise that yesterday’s documents will still work tomorrow.

Deterministic Conversion Still Has a Future

MacFarlane closes his retrospective by asking whether language models might eventually replace conventional document converters. A model can already infer formatting intent and translate a simple document into another syntax. It may even handle ambiguous human-authored markup more naturally than a deterministic grammar.

For production pipelines, however, Pandoc retains important advantages. It is efficient, local, testable, and deterministic. The same input, version, and options produce the same result. Failures can be reproduced. Builds can be pinned. Sensitive documents need not leave the machine. A filter can be inspected as code instead of inferred from a prompt.

The two approaches may become complementary. A model can help repair ambiguous input, design a template, or draft a Lua filter. Pandoc can then perform the repeatable conversion. Semantic inference is useful at the uncertain edges; deterministic machinery is valuable at the center of a build.

After twenty years, Pandoc is more than a universal converter. It is evidence that a small, coherent architecture can outlive several hosting platforms, packaging eras, markup fashions, and deployment targets. The formats changed. The interfaces multiplied. The central idea kept paying rent.

---

Sources: John MacFarlane — Twenty Years of Pandoc, Pandoc User’s Guide, Pandoc filters, Pandoc Lua filters, Pandoc release history, Pandoc on GitHub, Hacker News discussion