DEV Community: Mohamed Shams El-Deen

Regex isn't the best solution in all cases: Building a Recursive TypeScript Parser for nodejs/doc-kit

Mohamed Shams El-Deen — Thu, 02 Jul 2026 16:34:20 +0000

Introduction
A crucial part of modern documentation is cross-linking. When a developer reads an API doc, clicking a type should navigate them directly to its definition.

During my GSoC 2026 journey redesigning Webpack’s documentation, I realized the issue wasn't just in Webpack's codebase. The root of the problem lived upstream in nodejs/doc-kit, the engine responsible for parsing TypeScript annotations and generating Markdown links.

It started with a simple bug: the parser didn't understand basic generics like Promise<string>. But what began as a quick Regex fix evolved into building a Recursive Descent Parser capable of understanding deeply nested generic trees, operator precedence, and function signatures. Here is the technical breakdown of how I built it.

Phase 1: The Regex Trap

My initial approach PR #668 was naive. I thought, "It's just < >, I can use a Regular Expression 👀" I wrote a simple regex /^([^<]+)<([^>]+)>$/ to capture the base type and the inner type.

It worked perfectly for Promise<string>. But during the code review, a maintainer (Aviv Keller) asked a simple yet destructive question: "What about Promise<string | boolean>?"

Standard string splitting would blindly break the generic in half, resulting in malformed syntax. To patch this, I wrote a temporary utility called splitByOuterUnion that tracked the depth of the brackets, ensuring it only split the | symbol if it wasn't trapped inside < >:

const splitByOuterUnion = str => {
  const result = [];
  let current = '';
  let depth = 0;

  for (const char of str) {
    if (char === '<') {
      depth++;
    } else if (char === '>') {
      depth--;
    } else if (char === '|' && depth === 0) {
      result.push(current);
      current = '';
      continue;
    }
    current += char;
  }

  result.push(current);
  return result;
};

It was a smart patch, but deep down, I knew it was just a band-aid.

Phase 2: The Breaking Point

A few weeks later, we needed to support highly complex types spanning multiple parameters, nested generics, and inline functions, such as:
(str: string[]) => Promise<Map<string, number & string>, Map<string | number>>

Regex was officially useless. You cannot parse infinitely nested mathematical structures with Regular Expressions. We needed a custom parser PR #763.

I deleted the regex approach and designed a top-down Recursive Descent Parser. Instead of trying to match the whole string at once, the parser breaks it down layer by layer, from the weakest operator to the strongest.

Here are the 3 pillars of this architecture:

Pillar 1: Depth Tracking (`walkAtDepthZero`)

The foundation of the parser is an iterator that walks through the string character by character, keeping track of bracket depth () {} <> []. It takes a callback onToken that only executes when depth === 0 (meaning we are at the root scope, not trapped inside a nested generic or function parameter).

const TYPE_OPENERS = new Set(['<', '(', '{', '[']);
const TYPE_CLOSERS = new Set(['>', ')', '}', ']']);

const walkAtDepthZero = (str, onToken) => {
  let depth = 0;
  for (let i = 0; i < str.length; i++) {
    const char = str[i];
    if (TYPE_OPENERS.has(char)) {
      depth++;
    } else if (TYPE_CLOSERS.has(char) && !isArrowTail(str, i)) {
      depth--;
    }

    if (depth === 0) {
      const skip = onToken(i, char);
      if (skip === true) {
        return;
      }
      if (typeof skip === 'number') {
        i += skip;
      }
    }
  }
};

Pillar 2: Operator Precedence

In TypeScript, the arrow function => wraps around unions |, which wrap around intersections &. Our parser needs to find the weakest link to split the string correctly.
Using walkAtDepthZero, I built findTopLevelOperator to scan the root scope and locate the first occurrence of these operators, respecting their precedence:

const findTopLevelOperator = str => {
  let arrowIdx = -1;
  let unionIdx = -1;
  let intersectIdx = -1;

  walkAtDepthZero(str, (i, char) => {
    if (char === '=' && str[i + 1] === '>') {
      if (arrowIdx === -1) {
        arrowIdx = i;
      }
      return 1; // skip '>'
    }
    if (char === '|' && unionIdx === -1) {
      unionIdx = i;
    } else if (char === '&' && intersectIdx === -1) {
      intersectIdx = i;
    }
  });

  if (arrowIdx !== -1) {
    return { op: '=>', index: arrowIdx, width: 2 };
  }

  if (unionIdx !== -1) {
    return { op: '|', index: unionIdx, width: 1 };
  }

  if (intersectIdx !== -1) {
    return { op: '&', index: intersectIdx, width: 1 };
  }

  return null;
};

Pillar 3: The Recursive Engine (`parseType`)

This is where the magic happens. parseType acts as the orchestrator. It uses findTopLevelOperator to split the string into left and right chunks. Then, it calls itself recursively on those smaller chunks until it strips away all arrays, generics, and parentheses, finally reaching the base types to generate the Markdown URLs.

export const parseType = (typeString, transformType) => {
  const trimmed = stripOuterParentheses(typeString);
  if (!trimmed) {
    return null;
  }

  const op = findTopLevelOperator(trimmed);
  if (op) {
    if (op.op === '=>') {
      const left = trimmed.slice(0, op.index).trim();
      const right = trimmed.slice(op.index + op.width).trim();
      const sig = parseFunctionSignature(left, transformType);
      return `${sig} =&gt; ${resolveOr(right, transformType)}`;
    }

    // Union / intersection
    const parts = splitByOuterSeparator(trimmed, op.op);
    const joiner = op.op === '|' ? ' | ' : ' & ';
    return parts.map(p => resolveOr(p, transformType)).join(joiner);
  }

  // Strip a trailing `[]` for now; reapply on the way out.
  const isArray = trimmed.endsWith('[]');
  const core = isArray ? trimmed.slice(0, -2).trim() : trimmed;
  const arrayTail = isArray ? '[]' : '';

  // Generic: `Base<...>`.
  const ltIdx = core.indexOf('<');
  if (ltIdx !== -1 && core.endsWith('>')) {
    const baseType = core.slice(0, ltIdx).trim();
    const innerType = core.slice(ltIdx + 1, -1).trim();
    const inner = splitByOuterSeparator(innerType, ',')
      .map(arg => resolveOr(arg, transformType))
      .join(', ');
    return `${formatType(baseType, transformType)}&lt;${inner}&gt;${arrayTail}`;
  }

  // Plain base type.
  if (!core.length) {
    return null;
  }

  const url = transformType(core);
  if (!url) {
    return null;
  }

  return `[\`<${core}>\`](${url})${arrayTail}`;
};

What I Learned

Cross-Repo Impact: Fixing an issue in Webpack is great, but fixing the underlying tool (nodejs/doc-kit) impacts the entire ecosystem.
Regex is not a Parser: Regex is for pattern matching, not syntax analysis.
Think Like a Compiler: Understanding Abstract Syntax Trees (AST) and operator precedence isn't just academic theory; it's practically required to build reliable developer tooling.

Conclusion

We successfully replaced the regex fallback with the recursive parser. Now, Webpack's documentation (and any other project using nodejs/doc-kit) can accurately render and link extreme edge cases of TypeScript signatures.

Writing a parser from scratch taught me that sometimes the "quick fix" is just a delay. Taking the time to understand the root of the problem and engineering a mathematically sound solution is always worth the effort 🤍

Beyond Missing Exports: Building an Early Garbage Collector for Webpack’s TypeDoc AST

Mohamed Shams El-Deen — Wed, 17 Jun 2026 19:20:29 +0000

Introduction
Great documentation relies on an unbroken chain of references. As I'm responsible for the API Markdown Generation Phase, one of my tasks was to import internal types and interfaces that were missing from the generated documentation because they weren't explicitly exported.

The initial thought was simple: drop in typedoc-plugin-missing-exports and let it resolve the missing links. However, integrating a plugin into a massive codebase is rarely a plug-and-play operation. It turned into an architectural challenge involving Abstract Syntax Tree (AST) manipulation, memory optimization, and avoiding recursive explosions.

The Complexity of Resolution: 3 Experiments
To understand how TypeDoc handled these missing exports, I isolated the plugin's behavior through three distinct experiments:

Experiment 1: The Shallow Pass

Running the plugin with its default configuration recovered ~135 types. The downside? It isolated them in an <internal> module. Our documentation tool, nodejs/doc-kit, created markdown files for them and dumped them into an <internal> folder. We still needed to categorize them manually. Furthermore, TypeDoc didn’t traverse them deeply, leaving the documentation structurally inaccurate. It also included generic Node/JavaScript environment types, which we don't want since we aren't building Node/JS docs.

Experiment 2: Filtering the Noise (The Illusion of 60 Types)

By enabling excludeExternals: true, It stripped out the generic Node/JavaScript environment types. On the surface, this seemed successful, reducing the rendered payload in the docs to ~60 Webpack-specific types.
However, this was an illusion. In memory, the plugin was still extracting ~600 types and trapping them in the <internal> namespace. TypeDoc only rendered 60 of them because of its rendering and visibility rules, ignoring the deeper and nested dependencies. The rest were just eating up memory.

Experiment 3: The Recursive Explosion

To fix the shallow pass and map the types back to their original subsystems, I used placeInternalsInOwningModule: true. This broke the types out of the <internal>, but resulted in a recursive chain. The plugin began extracting nested internal interfaces, iterating continuously until it rendered ~630 inline types. The output was completely flooded with noise, making the developer experience terrible.

The Code Review: Rethinking the Architecture
I initially submitted a Pull Request based on Experiment 2, accepting the <internal> folder as necessary to avoid the recursive explosion of Experiment 3. But during the code review, the maintainers pointed out a fundamental flaw: keeping types in an <internal> module broke the logical grouping of Webpack's subsystems.

We needed the types in their respective modules, but we couldn't afford the ~600 noise types. We needed a custom intervention.

The Solution: Early Garbage Collection via AST Hooks
Instead of allowing TypeDoc to build an enormous AST and then trying to filter it during the routing phase (which wastes memory and CPU cycles), I implemented what I call "Early Garbage Collection" (similar to the V8 engine's Garbage Collector 😅).

By hooking directly into Converter.EVENT_RESOLVE_END, I could intercept the AST right after the initial resolution but before TypeDoc began assigning categories, building URLs, or allocating significant memory for the final output.

The logic was clean and focused on Separation of Concerns:

Locate the <internal> module in the AST (which secretly held all ~600 raw types).
Iterate through its child nodes.
Evaluate each node using a custom categoryForReflection utility.
Destroy the noise: Use project.removeReflection(child) to immediately sever the references of over 300+ unneeded nodes, allowing Node.js to garbage collect them and free up memory.
Lift the essentials: Merge the remaining ~300 crucial types directly into the root scope using project.mergeReflections.

By doing this, I rescued ~240 vital types that were previously hidden in Experiment 2's shallow pass, bringing the total of correctly routed, fully visible types to ~300 all without the recursive noise of Experiment 3!

The Implementation:

app.converter.on(Converter.EVENT_RESOLVE_END, context => {
  const project = context.project;
  const internalModule = project.children?.find(c => c.name === '<internal>');

  if (internalModule) {
    const importantTypes = [];

    internalModule.children.forEach(child => {
      // Evaluate before routing to prevent memory waste
      if (categoryForReflection(child)) {
        importantTypes.push(child);
      } else {
        // Early Garbage Collection
        project.removeReflection(child);
      }
    });

    internalModule.children = importantTypes;
    project.mergeReflections(internalModule, project); // Lift to root scope
  }
});

What I Learned
Stepping back from the code, here are the core engineering lessons from this PR:

Handle ASTs Early: "Garbage Collection" isn't just a V8 background task. Actively dropping unneeded AST nodes prevents massive downstream memory overhead.
Hooks over Hacks: Understanding compiler lifecycles (like EVENT_RESOLVE_END) allows for clean interventions rather than messy downstream workarounds.
Reviews Shape Architecture: Maintainer feedback shifted my focus from simply "closing the issue" to building a structurally sound solution that fits the system.
Completeness ≠ Usability: Extracting 600+ types is just noise. Good engineering is finding the sweet spot between capacity and actual developer experience.

Conclusion
What started as a simple plugin integration evolved into a lesson on compiler hooks and AST lifecycle management. By shifting the filtering logic earlier in the execution pipeline, we prevented memory overhead, exposed the true required types, avoided a recursive cascade of useless noise, and delivered clean, deeply-linked documentation for the Webpack community.

Redesigning Webpack's Documentation: My GSoC 2026 Journey

Mohamed Shams El-Deen — Sun, 14 Jun 2026 15:28:55 +0000

Hello, World! 😅

I am participating in Google Summer of Code (GSoC) 2026, working with the Webpack to completely redesign it's Documentation.

Description

Webpack is an industry-standard JavaScript module bundler used by millions of developers to build modern web applications. It plays a foundational role in today’s frontend ecosystem and is commonly used in projects built with frameworks such as React, Vue, Angular, and many others.

Problem

Webpack’s code changes a lot. Every time a developer writes or changes an API, someone from the team has to update the webpack documentation manually.

Solution

So, we will build an automated tool using GitHub Actions (that works as CI/CD and update documentation automatically), taking the advantage of the TypeScript compiler (that gives us APIs in types.d.ts), the typedoc-plugin-markdown (that converts TypeScript APIs to markdown), and the Node.js tool doc-kit (that adds links and customized UI).

The Roadmap

From the previous few months, I started tackling 5 major milestones:

M1: Completing the custom TypeDoc theme for accurate Markdown generation.

M2: Customizing the doc-kit output and UI components to match Webpack's brand.

M3: Building an end-to-end CI/CD automation pipeline.

M4: Implementing core pages (Home, Download, Guides).

M5: Expanding automation to fetch Loader and Plugin READMEs dynamically.

What is this series about?

I created this series to document my journey. No boring updates or simple typo fixes. I will write about the complex, interesting, and architecturally significant Pull Requests.

Wait for the first PR 🔥