DEV Community: frank

Which AI Works: A Practical AI Tools Directory

frank — Sun, 27 Sep 2026 00:22:30 +0000

Which AI Works is an AI tools directory for discovering products by task, comparing categories, and building a shortlist to test against your workflow. Use search and rankings to find candidates, then verify them on a representative task.

On the Which AI Works homepage, products are arranged around categories, and the search box accepts product, category, and feature terms. There is also a popularity leaderboard, individual product pages, and a compact library of builder resources. Here is a practical way to use those parts—and keep discovery separate from product evaluation.

Start with the job, not the model

Before opening a directory, write down the outcome you want in one sentence. “Turn customer calls into searchable notes” is more useful than “find an AI app.” “Generate a first draft of a React form from a design” is more useful than “find the best coding model.” A concrete job gives you something to compare against when product pages use similar language.

Then open the Which AI Works homepage. The directory presents product discovery alongside categories such as AI, analytics, automation, design, and developer tools. At the time of review, the homepage displayed 15 categories and a search box whose prompt includes products, categories, and features. Those are useful entry points when you have either a broad area (“design”) or a capability in mind (“transcription”).

A repeatable way to find and compare tools

Use this short workflow when you are evaluating a category you have not explored before:

Define the task and the constraint. Note the input you have, the output you need, and any non-negotiable limits: budget, data sensitivity, integrations, latency, or human review.
Search by task language. Try the phrase you would use to describe the work, then try a feature or category term. Start at Search or browse the category links from the homepage.
Open more than one product detail page. Read the overview and key features, then follow the product’s own website link. Similar tag labels do not guarantee similar workflows or pricing.
Check the vendor’s current documentation and terms. Confirm what the product actually accepts and returns, what its free or paid plan includes, and how it handles your data. A directory listing is a discovery aid, not an independent security review.
Test one representative task. Use a small, non-sensitive sample and a clear success condition. For example, check whether a transcription tool preserves speaker names and timestamps, or whether a coding assistant respects the project’s existing conventions.
Record the result. Keep the product, test input, outcome, price checked, and date. That makes a later comparison fair when product features or plans change.

This process is intentionally modest. It does not promise that browsing a directory will identify a universal winner. It helps you narrow the field, then gives you a repeatable way to make the final choice against your own work.

What the rankings can—and cannot—tell you

The Which AI Works rankings show a product leaderboard with All Time and month-specific views, plus category filters. The page describes the ordering as based on real catalog engagement. That makes the leaderboard useful for noticing what other visitors are exploring and for spotting products you might otherwise miss.

Treat that signal as attention, not as a quality score. Engagement does not tell you whether a tool is accurate on your task, handles private data appropriately, fits your budget, or remains reliable after an update. A product near the top of a list can be a good candidate for evaluation; the ranking alone is not a reason to adopt it. When you compare positions, keep the time window and category in view, and verify the product’s current behavior on its own site.

Product pages are the bridge to evaluation

A directory is most useful when it shortens the path from “I have a problem” to “I can test a relevant product.” On a listing’s detail page, use the overview and feature description to understand its stated purpose, note its categories, and follow the outbound link to inspect the product directly. For example, the Best Jev AI listing describes a browser playground and an independent API gateway; the page also separates its overview, features, and usage information. Those details help frame what to verify next, but they are still the listing’s published description—not proof of a benchmark result.

For a fair comparison, choose two or three candidates and give each the same task, inputs, and time budget. Check the output against a short rubric you write first. If the tools handle sensitive information, review their data policies before uploading anything real. The right product is the one that meets your requirements consistently, not necessarily the one with the most impressive demo.

A useful extra for builders

Which AI Works also maintains a builder resources page with links to primary documentation for technologies used to build and ship products, including Next.js, Cloudflare, Drizzle, and Stripe. It is a compact reference shelf rather than a general tutorial library. If you are evaluating developer tools or assembling an application stack, it gives you a direct route to the underlying platform documentation.

The site also has a path for creators who want to submit a product, with submission and pricing information under Pricing. That matters because directories serve two audiences: people looking for a tool and teams trying to make a product discoverable. As a reader, pay attention to labels such as promoted or sponsored when they appear, and use the same evaluation criteria for every candidate.

When to use Which AI Works

Use the directory when you need a starting point, a category overview, or a shortlist of products to investigate. It is especially useful when your task crosses categories—such as automating a design handoff or adding AI to an analytics workflow—because a task-first search can reveal options that a model-only list would miss.

Use vendor documentation, trials, and your own tests to answer the questions a directory cannot settle: Does it work on your data? Does it integrate with your stack? Are the limits and terms acceptable? How often does it fail on edge cases? Those answers depend on your workflow and can change over time.

A good AI tools directory should make discovery faster while leaving room for judgment. Which AI Works offers category browsing, task-oriented search, engagement-based rankings, product detail pages, and builder references in one place. Start with the job you need done, use the directory to find plausible candidates, and let a small, dated test—not a headline or leaderboard position—make the final call.

Building Reliable Decision Workflows with Jev AI: A TypeScript Guide

frank — Wed, 23 Sep 2026 15:17:24 +0000

Many product workflows do not need another paragraph from an AI model. They need a bounded answer that application code can use: which queue should receive a ticket, how urgent is it, or should a person review the next step?

Jev is designed for that narrower job. Send a piece of application state with questions whose answer shapes are defined in advance, then use the typed results in your own code. It is a decision API, not a chatbot or a video-generation model.

This guide follows the learning path in the Hugging Face step-by-step Jev guide—define a decision, prepare state, choose a question type, experiment, then integrate server-side. It extends that path with a typed TypeScript service, failure handling, three architecture diagrams, a comparison of the five sites supplied for this review, and a clearly dated benchmark snapshot. The API example below follows the endpoint in the current TypeSafe API reference.

A production shape for a decision workflow

Keep the model behind a server-side boundary. Your application validates its answer and decides whether the next step is an automatic, reversible action, a human review, or a separate generative-model task.

Figure 1. A decision architecture that keeps permissions and execution in application code.

What Jev returns

The request contains three top-level fields:

state: the text or structured application data to evaluate.
model: a model alias such as jev-latest.
questions: named questions with explicit answer types.

TypeSafe documents three primitives:

Question	Use it for	Result
`choice`	Choose a route from a fixed set	Selected option, option probabilities, and confidence
`score`	Rate the state against ordered levels	Probability-weighted score, level probabilities, and confidence
`noul`	Judge one yes/no proposition	A yes probability from 0 to 1

Several questions can share one request. They are evaluated independently against the same state, so a question in that request cannot depend on another answer. If a later decision needs an earlier result, express that dependency in application code. See the TypeSafe introduction and API reference for the request and response contract.

Call the API from a TypeScript server

Store the key in a server-side secret such as TYPESAFE_API_KEY. The official endpoint is POST https://api.typesafe.ai/v1/systemone with a Bearer token. Do not copy API keys into browser code or an article.

Endpoint note: the Hugging Face guide uses https://thejevai.com/v1/systemone in its example, while TypeSafe's current API reference documents https://api.typesafe.ai/v1/systemone. These are distinct service hosts. Use the endpoint and key issued by the provider you have selected, and verify that route's operator and data terms.

The Node.js 18+ example below evaluates a support ticket with one routing question, one urgency score, and one human-review check. It validates the fields the application will use before branching on them. enqueueForHumanReview and routeToTeam are application functions you would implement around your own queue and ticket system.

const ROUTES = ["billing", "delivery", "account", "technical", "other"] as const;
type Route = (typeof ROUTES)[number];

type TriageResult = {
  model: string;
  answers: {
    route: { type: "choice"; choice: Route; confidence: number };
    urgency: { type: "score"; score: number; confidence: number };
    needs_human: { type: "noul"; noul: number };
  };
};

function isRecord(value: unknown): value is Record<string, unknown> {
  return typeof value === "object" && value !== null;
}

function isProbability(value: unknown): value is number {
  return typeof value === "number" && Number.isFinite(value) && value >= 0 && value <= 1;
}

function isTriageResult(value: unknown): value is TriageResult {
  if (!isRecord(value) || typeof value.model !== "string" || !isRecord(value.answers)) return false;
  const { route, urgency, needs_human } = value.answers;
  return (
    isRecord(route) &&
    route.type === "choice" &&
    typeof route.choice === "string" &&
    ROUTES.includes(route.choice as Route) &&
    isProbability(route.confidence) &&
    isRecord(urgency) &&
    urgency.type === "score" &&
    typeof urgency.score === "number" &&
    Number.isFinite(urgency.score) &&
    urgency.score >= 0 &&
    urgency.score <= 2 &&
    isProbability(urgency.confidence) &&
    isRecord(needs_human) &&
    needs_human.type === "noul" &&
    isProbability(needs_human.noul)
  );
}

function retryAfterMs(value: string | null): number | undefined {
  if (!value) return;
  const seconds = Number(value);
  if (Number.isFinite(seconds) && seconds >= 0) return seconds * 1_000;
  const date = Date.parse(value);
  return Number.isNaN(date) ? undefined : Math.max(0, date - Date.now());
}

async function requestJev(body: unknown): Promise<unknown> {
  const apiKey = process.env.TYPESAFE_API_KEY;
  if (!apiKey) throw new Error("TYPESAFE_API_KEY is not configured");

  for (let attempt = 0; ; attempt++) {
    const response = await fetch("https://api.typesafe.ai/v1/systemone", {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
      },
      signal: AbortSignal.timeout(10_000),
      body: JSON.stringify(body),
    });

    if (response.ok) return response.json();

    const retryable = response.status === 429 || response.status === 529;
    if (!retryable || attempt >= 2) {
      throw new Error(`Jev request failed with HTTP ${response.status}`);
    }

    const hintedDelay = retryAfterMs(response.headers.get("Retry-After"));
    const backoff = Math.min(250 * 2 ** attempt, 2_000) + Math.random() * 200;
    const delay = hintedDelay ?? backoff;
    if (delay > 5_000) throw new Error("Jev retry window exceeds the request budget");
    await new Promise<void>((resolve) => setTimeout(resolve, delay));
  }
}

export async function triageTicket(ticket: {
  id: string;
  message: string;
  plan: string;
  failedAttempts: number;
}) {
  const body = {
    model: "jev-latest",
    // Send only fields needed for these decisions; keep ticket.id in our system.
    state: {
      message: ticket.message,
      plan: ticket.plan,
      failed_attempts: ticket.failedAttempts,
    },
    questions: {
      route: {
        type: "choice",
        instructions: "Which team should handle this support ticket?",
        criteria: {
          billing: "Payment, invoice, or refund issue",
          delivery: "A shipment or delivery issue",
          account: "Login, access, or account settings",
          technical: "A product bug or integration failure",
          other: "None of these categories clearly fit",
        },
      },
      urgency: {
        type: "score",
        instructions: "How urgent is the issue for the customer?",
        criteria: [
          "Routine; no near-term impact is described",
          "Time-sensitive; follow up during the current business day",
          "Service is blocked or a deadline is at immediate risk",
        ],
      },
      needs_human: {
        type: "noul",
        instructions: "Should a person review this ticket before an automated action is taken?",
        criteria: {
          true: "The facts are ambiguous or the next action could materially affect the customer",
          false: "The next step is a reversible, low-risk routing action",
        },
      },
    },
  };

  let payload: unknown;
  try {
    payload = await requestJev(body);
  } catch {
    // Log a redacted error in production; do not log the API key or raw ticket state.
    await enqueueForHumanReview(ticket.id, { reason: "jev_unavailable" });
    return { status: "review" as const };
  }

  if (!isTriageResult(payload)) {
    await enqueueForHumanReview(ticket.id, { reason: "invalid_jev_response" });
    return { status: "review" as const };
  }

  const { route, urgency, needs_human } = payload.answers;
  const reviewRequired = needs_human.noul >= 0.8 || route.confidence < 0.75;

  if (reviewRequired) {
    await enqueueForHumanReview(ticket.id, { route, urgency, needs_human });
    return { status: "review" as const, route, urgency };
  }

  await routeToTeam(ticket.id, route.choice, urgency.score);
  return { status: "routed" as const, route, urgency };
}

The retry loop only retries the documented transient statuses, 429 and 529, and stops when the delay exceeds this example's budget. Timeouts, invalid responses, missing credentials, and other HTTP errors enter the application's review path. Production services should add structured, redacted error logging and align timeouts and retry budgets with their own latency objectives.

The 0.8 and 0.75 values are demonstration thresholds, not Jev defaults. confidence is derived from an answer distribution; it is not a guarantee of correctness. Calibrate decision thresholds on examples from your own workload. For consequential actions such as refunds, account access, or deletion, require application permissions or human approval regardless of model confidence.

Treat the provider as part of your architecture

The API contract, key issuer, billing, and data path depend on which service endpoint you choose. The TypeSafe API reference documents the direct endpoint used above. The five sites below were supplied for this comparison; their public pages describe service entry points, playgrounds, or gateways rather than five independently benchmarked models.

Figure 2. Direct API access and Jev-branded services are separate routes to evaluate.

Site	Public positioning observed for this review	Check before integrating
thejevai.com	Advertises a free unlimited playground and API plans; its page says the service is independently operated and not affiliated with TypeSafe.	Confirm the endpoint and key issuer for your account, plus the current usage limits and data terms.
jevaimodel.net	Describes a playground and API credits around the TypeSafe model, and presents itself as an independent service.	Check credit accounting, required account scope, upstream provider, and request retention.
jevaimodel.org	Presents a Jev decision-model workbench with playground, API, and workflow material.	Verify the actual API host, model provider, processing region, and billing terms.
bestjevai.com	Describes an OpenRouter-powered playground and API gateway.	Confirm the routed model, gateway markup, key ownership, and gateway data policy.
jevmodel.net	Advertises a playground and API access plans.	Define what “unlimited” covers and verify the service operator, rate limits, and retention terms.

This is a feature and positioning comparison based on public site descriptions, not a security or compliance audit. Site terms, endpoints, and pricing can change. A familiar name or compatible request shape does not establish common ownership or identical privacy behavior. For production, inspect the exact endpoint, key issuer, upstream provider, region, retention and logging policy, rate limits, price, support, and terms before sending sensitive state. Use synthetic examples during initial evaluation.

Evaluate before automation, then roll out gradually

A successful playground result only proves that one example produced an answer. Build a labeled evaluation set that includes routine cases, boundary cases, ambiguous inputs, and cases where the correct outcome is human review. Track per-class precision and recall, calibration, review rate, p95 latency, timeout and overload rates, and cost per accepted decision.

Run a shadow phase first: record what Jev would have selected without taking the action, then compare it with the human or rules-based outcome. If that meets your acceptance criteria, canary a small percentage of eligible, low-risk traffic. Keep a safe fallback and a rollback path. Revisit the evaluation set as products, rubrics, languages, and request distributions change.

Figure 3. A measured release loop feeds reviewed production outcomes back into the evaluation set.

Six systems in a JevBench snapshot

“Top six” only has meaning after naming a scoring rule and a measurement date. The JevBench v1.3.0 results combine intelligence, calibration, speed, and cost at 25% each using a geometric mean. The table below reproduces the first six systems in the results file as reviewed on September 23, 2026; the run covers 534 frozen decisions per system.

Rank	System	JevBench score	Evaluated setup
1	Jev 1.13.0	74.4	TypeSafe production API
2	SemIf (Qwen3.5-4B)	73.1	Self-hosted GPU
3	djev (Maisa, diffusion-gemma)	73.0	Hosted API preview
4	Winnow-12B Q8	71.2	Self-hosted GPU
5	reflex 4B	70.3	Self-hosted GPU
6	jqv (Qwen3-32B, zero-shot)	68.6	Self-hosted GPU

Treat this as a dated benchmark snapshot, not a universal buying guide or a ranking of the five sites above. The benchmark notes that some self-hosted and demo latencies are adjusted by ×2, plus 0.15 seconds on the benchmark author's own servers, to approximate production load. That is an assumption, not a direct measurement. Your result may differ with your hardware, request mix, language, provider, concurrency, and the relative weight you give to accuracy, calibration, speed, and cost. Re-run your own workload before selecting a production route.

When Jev fits—and when it does not

Use a typed decision API when the answer space is known in advance and your application needs a classification, score, or focused yes/no judgment. Use a generative model for drafting, explanations, open-ended interaction, and work that needs extended reasoning. A reliable system can use both: Jev chooses among allowed paths, and application code decides whether to proceed, ask a person, or call a generative model.

The useful boundary is simple: a model can provide a decision signal; your application still owns the policy and the action.

References

A DOCX Style Map Is an API Contract, Not a Font Heuristic

frank — Thu, 17 Sep 2026 03:10:40 +0000

A paragraph can look like a heading without declaring a heading style. A converter has to choose whether to preserve the document's explicit structure or guess the author's intent. Those are different products, and mixing them silently makes migrations hard to audit.

I tested a small example on September 17, 2026, using docx 9.7.1 to generate inputs, Mammoth 1.12.0 to read them, and Turndown 7.2.4 to serialize HTML as Markdown. The interesting result was not just a missing heading: a table survived the first stage and disappeared structurally in the second.

Start with two independent signals

The semantic fixture uses a paragraph style:

new Paragraph({
  heading: HeadingLevel.HEADING_1,
  children: [new TextRun('Real heading')]
})

The visual fixture uses direct run formatting:

new Paragraph({
  children: [new TextRun({
    text: 'Visual heading', bold: true, size: 32
  })]
})

These are structural test fixtures, not a claim that two rendered pages are pixel-identical. Mammoth returned an h1 for the first and a paragraph containing strong for the second. With ATX headings enabled, the Markdown became # Real heading and **Visual heading** respectively.

That distinction follows the separation between paragraph properties and run properties in WordprocessingML. Reading a font size is not the same as reading a section level.

A custom style needs an agreed meaning

The next fixture used a custom paragraph style named Section Label, with bold 16pt formatting and outline level 0. Mammoth's default conversion produced a paragraph and an unknown-style warning. Adding this mapping changed that paragraph to an h1:

const result = await mammoth.convertToHtml(
  { buffer },
  { styleMap: [
    "p[style-name='Section Label'] => h1:fresh"
  ] }
)

Mammoth documents this mapping mechanism. The map is only justified when the document template actually uses that style for first-level headings. It is not permission to infer a heading from every unfamiliar style name.

This suggests treating template changes like schema changes: version the style map, retain unknown-style warnings, and keep fixtures from the templates you support.

Test the intermediate representation

My difficult fixture also contained a two-level numbered list and a 2-by-2 table. Mammoth's HTML contained both nested lists and a table. Default Turndown, without a table plugin, preserved the list but emitted the cell texts as separate paragraphs. All four cell values survived; their row/column relationship did not.

This is why a text-presence assertion is insufficient. Check the HTML table first, then check the Markdown representation expected by your destination. Turndown exposes plugins, but enabling an extension still requires a new compatibility test.

I reproduced these fixtures in MDFold's Word-to-Markdown converter, which I develop. The live table output had the same limitation. This is a boundary to review, not evidence of lossless conversion.

Boundaries that change the acceptance criteria

An independent comparison with LibreOfficeDev 26.8.0.0.alpha0's HTML exporter produced an h1 for the built-in heading and a visually formatted paragraph for the custom style. For an empty Heading 1, LibreOffice retained an empty heading with line breaks while Mammoth omitted it. This development build and these small fixtures are not a ranking of converter quality.

A deliberately invalid DOCX produced a ZIP error locally. In the live UI, selecting it after a valid file displayed an error but left the previous output available. That is a separate state-management defect: successful old text must not be mistaken for a successful new conversion. It was observed, not fixed, in this test.

My acceptance checks now separate three questions:

Did the reader recognize the declared structure and surface warnings?
Did serialization preserve the structures the destination supports?
Does the displayed result still belong to the currently selected input?

Images, tracked changes, footnotes, and merged cells were outside this experiment. A green result on these fixtures says nothing about those features.

Would you make visual heading inference opt-in, or show proposed headings for review before changing the output?

HTML-to-Markdown Round Trips Are a Semantic Contract, Not a Backup

frank — Mon, 14 Sep 2026 14:20:01 +0000

An HTML-to-Markdown conversion can preserve every visible word and still lose the document.

The reason is structural: HTML can encode presentation, application metadata, arbitrary element attributes, interactive behavior, and table spans. Markdown has a deliberately smaller document model. Conversion is therefore a policy for choosing information, not the inverse of HTML parsing.

I tested a browser-local converter with Node.js 25.3.0, Turndown 7.2.4, and Marked 18.0.7. Each fixture records the input HTML, emitted Markdown, and HTML rendered from that Markdown. The goal was not visual similarity. It was to identify which semantic invariants survived.

Start with a case that should work

This input uses structures with direct Markdown counterparts:

<h1>Release notes</h1>
<p>Ship the <strong>reviewed</strong> draft with the
  <a href="https://example.com/spec">source specification</a>.</p>
<ul>
  <li>Preserve hierarchy
    <ul><li>Verify the final link</li></ul>
  </li>
</ul>

The output was:

# Release notes

Ship the **reviewed** draft with the [source specification](https://example.com/spec).

-   Preserve hierarchy
    -   Verify the final link

Rendering it again restored one h1, one strong, the same link target, and two list levels. The HTML whitespace and tag spelling changed, but the document meaning under test survived.

That distinction matters. Byte equality is neither expected nor useful here. A migration test should ask whether the required structure survived.

One difficult fixture exposes the many-to-one mapping

Now add presentation metadata, inline elements without portable Markdown equivalents, and a merged table cell:

<section class="release-card" style="display:grid" data-build="42">
  <h2 style="color:red">Styled release</h2>
  <p><mark>Highlighted</mark> H<sub>2</sub>O and x<sup>2</sup>.</p>
  <table>
    <tr><th>Item</th><th>Status</th></tr>
    <tr><td rowspan="2">Parser</td><td>Ready</td></tr>
    <tr><td>Checked</td></tr>
  </table>
</section>

The actual Markdown was:

## Styled release

Highlighted H2O and x2.

Item

Status

Parser

Ready

Checked

The text order survived. The class, inline style, build metadata, highlight, subscript, superscript, table grid, and rowspan did not. Re-rendering produced a heading followed by plain paragraphs. No later converter can infer which paragraphs used to be cells.

The Turndown documentation explains why: recognized elements use conversion rules; the default rule for an unrecognized element emits its text content. Table support is supplied by a GFM plugin or custom rules rather than the base CommonMark rule set.

GFM tables still would not solve every case. The GFM specification defines a header row, delimiter row, and data rows whose cells contain inline content. It does not provide HTML-style row spans, column spans, or arbitrary block structure inside cells.

Malformed HTML adds a parsing stage you may forget

I also tested invalid nesting:

<p>Before<div>Inside</div>After</p>
<ul><li>One<li>Two</ul>

It became:

Before

Inside

After

-   One
-   Two

The HTML parser first constructed a corrected DOM tree. The converter then traversed that tree. The WHATWG parsing algorithm defines error-recovery behavior, so the converter may preserve the browser's interpretation rather than the author's exact source boundaries.

If a failed migration is inspected only after rendering, these stages become indistinguishable:

the source HTML was already invalid;
the HTML parser repaired it;
the conversion rule flattened a node;
the destination Markdown dialect interpreted the output differently.

Keep the original fixture and every intermediate representation if you need to locate the first semantic change.

Boundary behavior needs exact output, not a green message

The fourth fixture combined code metadata, non-breaking spaces, a line break, a script, and a style block:

<style>.release { color: red }</style>
<script>window.__shouldNotRun = true</script>
<pre><code class="language-js">if (a &lt; b) {
  console.log("x");
}</code></pre>
<p>alpha&nbsp;&nbsp;beta<br>gamma</p>

With the tested rules, script and style content were removed. The code and its js language survived, <br> became a Markdown hard break, and both U+00A0 characters remained:

```js
if (a < b) {
  console.log("x");
}
```

alpha  beta  
gamma

Those observations are version- and rule-specific. They are not proof that arbitrary HTML is safe, that every class becomes a language, or that every renderer treats non-breaking spaces identically.

Test a semantic contract

Instead of comparing HTML strings, define the structures your migration promises:

const expected = {
  headings: [{ level: 1, text: "Release notes" }],
  links: [
    { text: "source specification", href: "https://example.com/spec" },
  ],
  listDepth: 2,
  code: { language: "js", includes: 'console.log("x")' },
};

For a real content migration, I would assert:

heading count, level, and order;
link text and destination;
list item count and nesting depth;
code text, fence, and language;
image source, alt text, and relative-URL base;
table row and column counts;
first and last meaningful text;
an explicit inventory of discarded scripts, styles, and metadata.

I repeated the four public, non-sensitive fixtures on MDFold's HTML-to-Markdown tool. Its browser output matched the local Turndown script exactly. Headings, links, nested lists, code fences, and the tested language marker survived. The tested table flattened to text, so table structure remains a manual-review boundary in the current base converter. I also verified the workflow at a 390px viewport with no horizontal overflow.

That last limitation is why a converter's success message should never be the acceptance test.

Preserve the source when the source matters

Markdown is a good derived format for editing, diffing, documentation, and long-term text maintenance. It is not a forensic copy of a webpage.

If you must preserve CSS, interactive controls, form behavior, arbitrary attributes, complex tables, or exact invalid source spelling, retain the original HTML alongside the Markdown. Record the converter version, options, plugins, fixtures, and warnings so the derivation can be reproduced.

The engineering question is not “Did the converter finish?” It is “Which semantic contract did it satisfy, and which information did we intentionally abandon?”

When visual fidelity and semantic maintainability conflict in a migration, which one belongs in your acceptance criteria?

Source Positions Are a Coordinate System, Not Just Line and Column

frank — Fri, 04 Sep 2026 06:24:16 +0000

A Markdown linter can detect the right problem and still highlight the wrong character.

The usual cause is not parsing. It is a coordinate-system mismatch: one component counts UTF-8 bytes, another counts Unicode code points, and JavaScript indexes UTF-16 code units.

I tested this with Node.js 25.3.0, unified 11.0.5 + remark-parse 11.0.0, commonmark.js 0.31.2, markdown-it 15.0.1 (commonmark preset), and Marked 18.0.11.

One emoji, three valid offsets

Consider this source:

# A😀B

The position immediately before B is:

Unit	Offset
JavaScript UTF-16 code units	5
Unicode code points	4
UTF-8 bytes	7

const source = "# A😀B";
const utf16 = source.indexOf("B");
const codePoints = [...source.slice(0, utf16)].length;
const bytes = Buffer.byteLength(source.slice(0, utf16), "utf8");

console.log({ utf16, codePoints, bytes });
// { utf16: 5, codePoints: 4, bytes: 7 }

None of those numbers is universally wrong. The bug appears when an API calls all of them offset.

Positions should be half-open ranges

The unist specification defines a position with start and end points. Lines and columns are one-based, offsets are zero-based, and end points to the first character after the source region. Its definition of a character is a UTF-16 code unit.

remark-parse produced this position for the A😀B text node:

{
  "start": { "line": 1, "column": 3, "offset": 2 },
  "end":   { "line": 1, "column": 7, "offset": 6 }
}

That makes source recovery unambiguous:

source.slice(node.position.start.offset, node.position.end.offset)

Treating end as inclusive introduces an off-by-one error.

Four parsers, four position capabilities

The same input produced materially different public metadata:

Parser	Block positions	Inline positions	Absolute offset
remark-parse 11.0.0	yes	yes	UTF-16
commonmark.js 0.31.2	yes	no	no
markdown-it 15.0.1	line ranges	no	no
Marked 18.0.11	no standard field	no	no

commonmark.js reported the heading as [[1,1],[1,6]], but its A😀B text node had no sourcepos. markdown-it reported [0,1] on the inline token, while every inline child had a null map. Marked exposed raw, not a unique source location.

This is a capability difference, not a rendering-quality ranking. Block ranges are enough for scroll synchronization. Character-accurate quick fixes need inline ranges and explicit offsets.

Do not reconstruct positions with indexOf(token.raw). Repeated text makes the result ambiguous. Accumulating raw.length also breaks when a parser normalizes newlines, ignores a BOM, or merges text nodes.

Test failure syntax, not only valid syntax

I used four fixtures:

const fixtures = {
  normal: "# Title\n\nA paragraph.",
  difficult: "# A😀B\n\nUse **e\u0301** and `code`.",
  failure: "# Broken [link](<oops\n\nTail",
  boundary: "\uFEFF# Zero\u200BWidth\r\n\r\nEnd",
};

The failed link never became a link node. remark kept the failed construct in one text node; commonmark.js split it into several text nodes; markdown-it still only identified the containing line. A diagnostic implementation cannot assume malformed syntax has the AST shape of successful syntax.

The boundary fixture exposed another divergence. With a leading BOM, remark still recognized the heading, while commonmark.js parsed the first line as a paragraph. The zero-width character remained part of the text, and CRLF consumed two UTF-16 code units.

When the first question is whether a real file is Markdown, plain text, or affected by encoding, this Markdown file structure guide is the contextual checklist I use. It does not imply that parsers normalize those boundaries consistently.

A practical range contract

For a JavaScript editor, I would make the unit part of the field name:

type SourcePoint = {
  offsetUtf16: number;
  line: number;
  columnUtf16: number;
};

type SourceRange = {
  start: SourcePoint;
  end: SourcePoint; // exclusive
  sourceVersion: string;
};

If a Rust or Go service returns byte offsets, call the field offsetUtf8Bytes and convert against the exact same source text. sourceVersion matters because every range becomes suspect after formatting or editing.

For occasional lookups, scan from the start to compute line and column. For many diagnostics, precompute every line-start offset and use binary search. Decide how CRLF is counted and encode that decision in tests.

What I would regression-test

an emoji before the target;
a decomposed character such as e\u0301;
LF and CRLF;
a leading BOM;
a zero-width character;
repeated identical text;
malformed links and emphasis;
generated AST nodes with no source range.

The invariant is stronger than a snapshot: slicing the original source with a node's half-open offsets should reproduce that node's source spelling whenever the node genuinely came from one continuous region.

Source positions are a protocol between parser, analyzer, editor, and formatter. Define the unit, range semantics, and source version before trusting the number.

After a formatter rewrites the document, would you invalidate every diagnostic and reparse, or maintain a source map from the previous version?

Snapshot Tests Lie About Markdown: Assert Structure Instead

frank — Wed, 02 Sep 2026 08:22:56 +0000

A Markdown parser upgrade can pass every “does it render?” check and still change the document.

The dangerous regressions are quiet: a malformed link starts auto-linking a URL, a list changes nesting, an invisible character survives normalization, or a plugin turns previously literal text into an extension node.

An HTML snapshot will notice some of those changes. The problem is that it also notices quote style, optional closing tags, whitespace, attribute order, and renderer-specific formatting. Once a dependency upgrade changes hundreds of snapshots, “update all” becomes tempting—and that can approve the semantic regression with the noise.

I ran a small fixed-version experiment to separate structure from serialization.

The experiment

Environment:

Node.js 25.3.0
commonmark.js 0.31.2
markdown-it 15.0.0 with the commonmark preset
Marked 18.0.9 with default options

The corpus deliberately includes four classes of input:

const cases = [
  { id: "normal", source: "# Release\n\n- parse\n- render\n" },
  { id: "difficult", source: "Read [the *nested* case](https://example.com/a_(b)).\n" },
  { id: "failure", source: "Broken [link](<https://example.com\n" },
  { id: "edge-nul", source: "left\u0000right\n" },
];

Instead of first comparing the entire HTML string, the harness extracts the contract we actually care about:

function structuralFacts(html) {
  return {
    heading: /<h1(?:\s|>)/.test(html),
    list: /<ul(?:\s|>)/.test(html) && /<li(?:\s|>)/.test(html),
    link: /<a\s+href=/.test(html),
    replacement: html.includes("�"),
  };
}

All 12 parser-fixture assertions passed after the expected capability differences were made explicit.

The differences were the useful result

All three parsers produced the same structures for the normal document and the nested-emphasis link.

For the unclosed link destination, commonmark.js and markdown-it emitted literal text. Marked did not create the intended Markdown link either, but its default URL auto-linking created an anchor for the URL inside the broken syntax.

For the U+0000 case, commonmark.js and markdown-it emitted U+FFFD, matching CommonMark's input-character rule. The HTML string returned by Marked still contained U+0000.

That does not make differential testing a vote. Two implementations agreeing does not establish the product contract. The correct reference depends on the layer:

CommonMark core: the pinned CommonMark spec examples
GFM or another extension: that extension's spec and the enabled configuration
product behavior: an explicit, reviewed project decision
unresolved divergence: a report, not an automatically accepted snapshot

Start with upstream conformance fixtures

The CommonMark specification repository embeds more than 500 examples that act as conformance tests. Its test tool can dump them as JSON records containing the Markdown input, expected HTML, section, and example number.

That is a much stronger base than a hand-written “Markdown basics” file:

for (const example of commonmarkSpec.tests) {
  assert.equal(
    render(example.markdown),
    example.html,
    `CommonMark example ${example.number}`
  );
}

But conformance is not the whole product. Those fixtures do not cover your sanitizer, editor transactions, plugin ordering, export template, or platform-specific extensions.

Add one permanent fixture per real bug

Every parser bug should leave behind:

the smallest input that reproduces it;
parser, plugin, preset, and option versions;
the expected structural facts;
the issue or commit that explains why the expectation exists;
the boundary: spec requirement, extension, or product policy.

Do not delete that fixture when the implementation changes. If the intended behavior changes, update the contract in a reviewable change that explains why.

Use snapshots as evidence, not as the oracle

Raw HTML snapshots are still useful when exact serialization is a public API. Keep them beside structural assertions.

For most editor and publishing systems, stronger assertions target:

node types and nesting;
heading levels;
link and image destinations;
literal text preservation;
source spans when diagnostics depend on them;
sanitizer results after rendering;
stable IDs or attributes that downstream code consumes.

When a table fixture is needed, I use a visual generator only to create the source, then test the generated Markdown in the target parsers. In a September 2 check, the MDFold Markdown Table Generator escaped Maya | UX as Maya \| UX; that proves the current generator's output for this fixture, not universal table support. The same page also stayed within a 390-pixel viewport without page-level horizontal overflow.

Property tests need real invariants

“Every parser must produce identical HTML” is not a valid property across Markdown dialects.

Better properties include:

parsing bounded input never crashes;
the same version and options produce a deterministic result;
the AST contains no parent-child cycles;
disabling raw HTML produces no raw-HTML nodes;
bounded input size and nesting do not produce unbounded work.

Property-based testing tools combine generated inputs with predicates. Record the seed for every failure, shrink it, and add the minimal result to the permanent regression corpus. Otherwise the interesting failure disappears after the random run.

Keep pathological tests separate

Large unmatched delimiter runs, deep brackets, nested lists, and unclosed HTML comments test complexity rather than ordinary correctness. markdown-it's own repository separates CommonMark fixtures, implementation fixtures, and pathological tests; the pathological suite uses an isolated worker and a timeout.

That separation is important. Run fast semantic fixtures on every commit. Run larger complexity suites nightly or before releases, using a stable input scale and a generous budget. A noisy wall-clock threshold on shared CI is not a reliable algorithm test.

A practical test layout

fixtures/
  commonmark/      # upstream, versioned conformance data
  regressions/     # one directory per real bug
  extensions/      # GFM tables, task lists, footnotes...
  security/        # raw HTML, protocols, remote resources
  pathological/    # deep, long, and unclosed inputs

For dependency upgrades, generate a report with three sections: new differences, removed differences, and unclassified differences. Block the upgrade on the last section. That makes “update snapshots” a reviewed decision instead of a reflex.

Sources

CommonMark 0.31.2 specification and examples: https://spec.commonmark.org/0.31.2/
CommonMark spec test tooling: https://github.com/commonmark/commonmark-spec
markdown-it test layout and pathological cases: https://github.com/markdown-it/markdown-it/tree/master/test
fast-check property model: https://fast-check.dev/docs/core-blocks/properties/

What does your Markdown suite protect today: the document's meaning, or one renderer's current whitespace?

Regex Cannot Sanitize Markdown: Put the Security Boundary After Parsing

frank — Mon, 31 Aug 2026 07:10:45 +0000

Deleting <script> from Markdown with a regular expression is not an HTML security boundary. The browser eventually consumes a DOM, not the original Markdown string. A regex that matches one attribute spelling can miss another spelling the HTML parser accepts, while an aggressive replacement can damage text that was supposed to remain inside a code span.

The reliable order is:

Markdown source
  -> pinned parser
  -> HTML sanitizer for the real output policy
  -> controlled DOM sink
  -> CSP / Trusted Types as defense in depth

A regex that passes the easy test

I tested this deliberately narrow cleaner:

function naiveRegexClean(html) {
  return html
    .replace(/<script\b[^>]*>[\s\S]*?<\/script>/gi, '')
    .replace(/\son\w+="[^"]*"/gi, '')
    .replace(/javascript:/gi, '');
}

It removes a plain <script> and a double-quoted onerror. It does not remove the same event attribute when it is unquoted or single-quoted:

Input	Result
`<img src=x onerror="alert(1)">`	event attribute removed
`<img src=x onerror=alert(1)>`	event attribute remains
`<img src=x onerror='alert(1)'>`	event attribute remains

Replacing javascript: also turns the URL into a different string. It does not prove that the resulting URL matches the application's protocol or navigation policy.

The point is not that every regex is short. The point is that string matching and browser HTML parsing operate at different abstraction levels.

The fixed-version test

I used Node.js 25.3.0, Marked 18.0.7, DOMPurify 3.4.12, and MDFold's current browser-side Markdown-to-HTML path. The fixture covered ordinary formatting, raw scripts, quoted and unquoted event handlers, Markdown and raw-HTML javascript: links, inline code, a remote image, and a fixed-position style.

Marked preserved the raw HTML in its generated output. That is expected: the Marked documentation explicitly warns that it does not sanitize output HTML.

The real browser preview then sanitized the parsed HTML. It:

preserved bold text and an HTTPS link;
removed the script element and event handler;
removed unsafe href values while preserving link text;
preserved the attack-shaped string inside inline code as escaped text;
preserved the remote image URL;
preserved the tested style attribute.

This is more informative than checking whether an alert happened to appear. The test inspected the resulting elements, attributes, URLs, and text nodes.

Parsing, sanitizing, and resource policy are separate

CommonMark defines how raw HTML participates in Markdown parsing. It does not define a web application's XSS policy. OWASP recommends output encoding when data should remain text and HTML sanitization when authors are allowed to provide HTML. DOMPurify works on a parsed, inert DOM and applies an allow-list to elements and attributes.

An application that does not need raw HTML can reject HTML nodes earlier. That reduces the surface but does not answer every downstream question. Images can still cause network requests. Links can still leave the site. Plugins can still generate HTML. The final sink still owns the final policy.

The two preserved values in my test show the remaining boundaries:

A sanitized remote image can still make a request. XSS prevention and privacy are different controls.
A permitted style can still obscure content. Script safety and visual-integrity policy are different controls.

Do not modify sanitized HTML afterward

This sequence destroys the boundary:

const clean = DOMPurify.sanitize(dirty);
target.innerHTML = clean.replace('USER', untrustedValue);

Every untrusted value needs the defense for the context where it is inserted. Prefer safe sinks such as textContent for plain text. If a CMS or template engine reparses or mutates the HTML later, validate at that boundary too.

A practical review checklist

Pin parser and sanitizer versions separately.
Disable raw HTML when the product does not need it.
When HTML is required, start with a narrow allow-list.
Define protocol rules for href and resource rules for src.
Decide whether inline styles belong in the content model.
Avoid post-sanitization string concatenation.
Inspect the final DOM and network behavior, not only the source string.
Retest in the destination CMS, browser, and mobile layout.

I reproduced the browser portion with the MDFold Markdown-to-HTML converter. At a 390-pixel viewport the tested preview stayed within the page width, but that result covers this fixture and current version only.

The engineering question is not "which regex catches every payload?" It is "which component owns the policy for this exact output sink?"

Primary sources

Invisible Unicode Characters Can Break Markdown Without Looking Different

frank — Sat, 22 Aug 2026 22:44:45 +0000

Two Markdown lines can look identical and still produce different HTML. Before blaming the parser, inspect the code points.

# heading
#heading

The second line contains U+200B ZERO WIDTH SPACE after #. CommonMark requires the ATX marker sequence to be followed by a space, tab, or line ending, so this is a paragraph rather than a heading.

Make the invisible input observable

const codePoints = value => [...value].map(char =>
  `U+${char.codePointAt(0).toString(16).toUpperCase().padStart(4, '0')}`
)

console.log(codePoints('#\u200Bheading'))
// U+0023 U+200B U+0068 ...

When the source came from an unknown editor, first verify that it is actually a text file and inspect its encoding. This guide to opening Markdown files is a useful checklist; the important step is inspection, not conversion.

What three fixed parser versions produced

I tested Node.js 25.3.0 with Marked 18.0.7, markdown-it 14.1.0, and commonmark.js 0.31.2. These are observed outputs under default options, not claims about every Markdown platform.

Input	Marked	markdown-it	commonmark.js
`# heading`	heading	heading	heading
leading U+FEFF before `#`	paragraph, FEFF retained	paragraph, FEFF omitted in output	paragraph, FEFF omitted in output
U+200B after `#`	paragraph	paragraph	paragraph
U+00A0 after `#`	heading	paragraph	paragraph
fullwidth `＃` or `＊`	plain text	plain text	plain text

The NBSP result is the useful warning: a character that looks like a space creates a real cross-parser difference. “Whitespace” is not one universal Markdown category.

A BOM is positional

UTF-8 uses EF BB BF as its optional signature. After decoding, that is U+FEFF. Unicode says a recognized initial BOM should be removed before text processing. The same code point in the middle of text is not automatically a BOM and must not be globally deleted.

In the experiment, a leading U+FEFF prevented # from being the first character seen by the Markdown grammar. An internal alpha\uFEFFbeta remained present in all three outputs. Fix BOM handling at the decoding boundary, not with replaceAll('\uFEFF', '').

Zero width does not mean zero semantics

U+200B is a format control used to indicate a word or line-break opportunity. Its effect depends on position:

after #, it prevents ATX-heading recognition;
inside **bo\u200Bld**, all three parsers still create <strong>, while preserving U+200B in the text node.

So a blanket “remove invisible characters” rule is both too broad and too weak.

Look-alike punctuation is different punctuation

＃ is U+FF03, not U+0023. ＊ is U+FF0A, not U+002A. All three parsers treated the fullwidth examples as ordinary text.

NFKC normalization can map some compatibility characters to ASCII, but silently normalizing user source changes data. A safer editor warns, previews the proposed change, and lets the author approve it.

Bidirectional controls require a different review model

UAX #9 distinguishes logical order from display order. Directional controls can change what reviewers see while parsers continue reading logical code-point order.

With U+202E inside a Markdown URL, none of the three parsers simply discarded it. Marked and commonmark.js percent-encoded it; markdown-it serialized the host differently through IDNA processing. The safe review target is therefore not just the visible source. Inspect the code points, parsed href, and final resolved URL.

A practical ingestion policy

Handle an initial BOM during byte decoding.
Preserve raw bytes in regression fixtures.
Flag Cf characters and non-ASCII look-alike punctuation near Markdown delimiters.
Show code point, line, column, and Unicode name in diagnostics.
Preview repairs instead of silently rewriting source.
Validate links after parsing and URL resolution.

Invisible-character bugs are boundary bugs: the decoder, editor, Markdown grammar, HTML serializer, and browser may each make a different decision. Good tooling makes those decisions observable.

Would you prefer an editor that automatically cleans invisible characters, or one that always preserves the source and only warns?

Markdown Images Are Network Requests, Not Embedded Assets

frank — Thu, 20 Aug 2026 06:08:34 +0000

The line ![diagram](../assets/flow.png) does not embed an image. It creates an image node whose destination still has to be resolved, fetched, allowed, and rendered by another system.

That distinction explains why a document can parse perfectly and still show broken images—or make network requests its author did not account for.

Parsing and URL resolution are separate stages

I fixed the experiment to Node 25.3.0 and Marked 18.0.7 with GFM enabled:

const html = marked.parse('![diagram](../assets/flow.png "Flow")');
// <p><img src="../assets/flow.png" alt="diagram" title="Flow"></p>

Marked preserves the destination. A browser or publisher later resolves it against a base URL:

new URL("../assets/flow.png", "https://docs.example.test/guides/setup/").href
// https://docs.example.test/guides/assets/flow.png

new URL("../assets/flow.png", "https://docs.example.test/guides/setup.html").href
// https://docs.example.test/assets/flow.png

Same Markdown, different request. The parser did its job in both cases.

Six cases that need different policies

Input	Parser output	What the host must decide
`../assets/flow.png`	relative `src`	base URL and asset copy rules
`/assets/logo.svg`	root-relative `src`	which origin owns the root
`https://.../pixel.png?doc=42`	absolute `src`	remote trust and request privacy
`data:image/...`	data URL	sanitizer and CSP policy
empty alt	`alt=""`	decorative or accessibility defect
empty destination	`src=""`	reject, rewrite, or host behavior

CommonMark defines image syntax and maps the description to alt text. It does not upload files or promise that a URL exists.

Remote images are third-party dependencies

When HTML contains a remote <img src>, the browser requests that resource. The remote server observes the request; the page's Referrer Policy controls how much referring-page information is sent. Query parameters can also encode a document or campaign identifier.

Treat remote images like any other external dependency:

allow only reviewed schemes and hosts;
proxy or self-host assets when appropriate;
set a deliberate Referrer Policy;
restrict image sources with CSP img-src;
sanitize final HTML after AST transformations;
never put sensitive identifiers in image URLs.

Parser configuration cannot replace those controls.

Relative paths need a publishing contract

A stable workflow defines:

where the Markdown file lives;
what base URL the final page uses;
how referenced assets are copied;
whether root-relative paths are allowed;
what happens when an image is missing.

If you are debugging a real document, this Markdown image path guide provides a concrete path checklist. The important step is to test the published URL, not just the editor preview.

Data URLs trade path stability for other costs

Data URLs avoid a separate file lookup, but they make source files larger and harder to review, prevent independent caching, and may be blocked by a sanitizer or CSP. They are reasonable for small controlled assets, not a universal fix for image portability.

Validate the final request graph

My recommended pipeline is:

parse Markdown to an AST;
inspect image destinations;
resolve them against the declared publication base;
validate scheme, host, path, and query;
copy or rewrite managed assets;
sanitize rendered HTML and apply CSP/referrer policy;
open the public page and verify actual responses and layout.

The AST tells you what the author referenced. URL resolution tells you where it points. A browser test tells you what really happened.

Should Markdown hosts block cross-origin images by default, or preserve compatibility and require each application to opt into a stricter policy?

Primary references

Your Markdown Parser Is Not Your XSS Boundary

frank — Wed, 19 Aug 2026 02:36:27 +0000

A Markdown parser can produce exactly the right HTML and still leave your application exposed to XSS. Parsing answers what the input means. Sanitization decides which parts of that meaning are allowed to reach an HTML sink.

I tested that boundary with Node.js 25.3.0, Marked 18.0.7, DOMPurify 3.4.12, and jsdom 30.0.1. The important comparison is not a screenshot. It is the HTML before and after sanitization.

The smallest useful pipeline

const rendered = marked.parse(markdown)
const sanitized = DOMPurify.sanitize(rendered, {
  USE_PROFILES: { html: true },
  SANITIZE_NAMED_PROPS: true,
})

This deliberately keeps two responsibilities separate. Marked parses Markdown. DOMPurify applies an allow-list to the HTML structure that will approach the browser.

Five cases that expose the boundary

1. Normal content survives

# Hello

[Safe](https://example.com)

The heading and HTTPS link survive both stages. A sanitizer should preserve allowed document structure, not flatten every document to text.

2. Raw HTML is valid Markdown, not necessarily safe HTML

<img src=x onerror="alert(1)">

Marked 18.0.7 returns the element and its event attribute unchanged. DOMPurify returns:

<img src="x">

The parser did not fail. CommonMark supports raw HTML. The unsafe step would be treating syntactic validity as authorization to insert every attribute.

3. URL schemes need their own policy

[click](javascript:alert(1))

Rendered HTML:

<p><a href="javascript:alert(1)">click</a></p>

Sanitized HTML:

<p><a>click</a></p>

An element allow-list alone is insufficient. URL-bearing attributes need scheme validation.

4. Code examples must not be cleaned as attacks

```html
<img src=x onerror="alert(1)">
```

The parser escapes the payload inside pre > code. A regex that removes attack-looking source before parsing would damage legitimate security documentation. Context has to be established first.

5. XSS defenses extend beyond `script`

<form id="attributes"><input name="action"></form>

With SANITIZE_NAMED_PROPS, the result becomes:

<form id="user-content-attributes"><input name="user-content-action"></form>

This addresses DOM clobbering: attacker-controlled names can interfere with properties that application code expects to resolve normally.

Put sanitization after the last unsafe transform

A practical pipeline is:

untrusted Markdown
  -> parser
  -> Markdown/HTML AST transforms
  -> sanitizer
  -> serializer
  -> matching HTML sink

The rehype-sanitize documentation makes the ordering rule explicit: sanitize after the last unsafe operation, because a later plugin can reintroduce unsafe properties. DOMPurify's current threat model adds another constraint: do not sanitize and then freely post-process the result. The policy and the sink must stay aligned.

Turning off raw HTML is a useful reduction in attack surface, but it is not a universal sanitizer. Plugins, link protocols, generated IDs, and later transforms still deserve explicit policies.

What I verify in a conversion workflow

When checking Markdown to HTML, I separate three questions:

Did normal Markdown preserve the intended structure?
Did fenced examples remain inert code?
Is untrusted output safe for this application's sink and policy?

The first two are conversion checks. The third belongs to the embedding application. A converter producing structurally correct HTML does not automatically promise that arbitrary input is safe to inject into another site's DOM.

Engineering checklist

Treat external Markdown as untrusted by default.
Disable raw HTML where the product does not need it.
Sanitize the final HTML tree with a maintained allow-list sanitizer.
Model URL schemes, id, name, and styling capabilities explicitly.
Do not run arbitrary HTML-mutating plugins after sanitization.
Test AST shape, rendered HTML, and sanitized HTML separately.
Pin and update sanitizer versions; security fixes are part of the boundary.

The design question I keep coming back to is this: should Markdown renderers disable raw HTML by default, or should they expose it only behind an explicit host-supplied security policy?

Primary sources

Build a Markdown Heading Linter on the AST, Not With Regex

frank — Sat, 15 Aug 2026 22:13:26 +0000

A heading linter sounds like a one-line regex:

/^#{1,6}\s+(.+)$/gm

It works until a documentation page contains a fenced example:

# Real heading

```md
## This is code, not navigation
```

The regex finds two headings. A Markdown parser finds one heading and one code block.

That difference is why I prefer to put document automation behind an AST boundary:

source -> parser -> mdast -> transformer -> hast -> HTML

A small, testable plugin

For this experiment I pinned unified@11.0.5, remark-parse@11.0.0, remark-rehype@11.1.2, rehype-stringify@10.0.1, and unist-util-visit@5.1.0 on Node 25.3.0.

The plugin has two jobs:

collect headings for a table of contents;
report a diagnostic when heading depth jumps by more than one level.

function headingAudit() {
  return (tree, file) => {
    const headings = []
    let previousDepth = 0

    visit(tree, 'heading', node => {
      const title = textOf(node)

      if (previousDepth && node.depth > previousDepth + 1) {
        file.message(
          `Heading jumps from h${previousDepth} to h${node.depth}`,
          node
        )
      }

      headings.push({depth: node.depth, title})
      previousDepth = node.depth
    })

    file.data.headings = headings
  }
}

The parser decides what a heading is. The plugin only evaluates heading nodes. This keeps syntax decisions separate from project policy.

Four cases worth keeping in CI

1. Normal structure

# Guide

## Install

Result: two heading nodes, no diagnostic, and two TOC entries.

2. Inline markup and duplicate labels

# Guide

## *API* Reference

## API Reference

The first heading does not contain one flat text node. Its children include an emphasis node, so a robust text extractor must walk descendants. Both visible labels normalize to the same slug; the second needs a stable suffix such as api-reference-1.

3. Valid Markdown that violates a project rule

# Guide

### Internals

Parsing and HTML rendering succeed, but the plugin reports:

Heading jumps from h1 to h3

This is a useful distinction. A parser error means the syntax cannot be interpreted under the configured grammar. A linter diagnostic means the syntax is valid but conflicts with a documentation policy.

4. A heading marker inside code

# Real heading

```md
## Not a heading
```

The top-level mdast nodes are heading and code. The TOC contains only Real heading. No special-case regex is needed because the parser has already resolved the context.

Keep plugins on the right side of the bridge

Here is the complete processing order:

const file = await unified()
  .use(remarkParse)
  .use(headingAudit)
  .use(remarkRehype)
  .use(addHeadingIds)
  .use(rehypeStringify)
  .process(markdown)

headingAudit expects mdast and therefore runs before remarkRehype. addHeadingIds writes HTML properties and therefore runs on hast after the bridge. Plugin order is part of the contract, not cosmetic configuration.

When checking the same samples with a Markdown-to-HTML converter, I compare structural output—headings, code blocks, links—not only whether the preview looks plausible. Two pages can look similar while exposing different trees to a TOC generator, sanitizer, or editor.

What an AST does not solve

An AST removes a lot of context guessing, but it does not make a pipeline automatically safe:

two plugins can mutate the same nodes;
generating a TOC before another plugin rewrites headings creates stale data;
slug behavior is not defined by CommonMark and differs across platforms;
raw HTML still needs an explicit trust and sanitization policy;
syntax-extension plugins can change what inputs the parser accepts.

The practical rule I use is to document every plugin's input tree, output tree, mutations, diagnostics, and ordering constraints. Then test both the tree shape and the serialized output.

CommonMark 0.31.2 supports the underlying separation: block structure is resolved before inline structure. Unified and remark make that structure available as mdast and provide a plugin pipeline around it. The experiment above only claims the pinned versions and inputs listed here; it is not proof that every Markdown dialect shares the same tree or slug rules.

Where would you draw the boundary between plugin freedom and syntax stability in a long-lived documentation pipeline?

Markdown Dialects Need a Capability Matrix, Not Just a Name

frank — Sat, 15 Aug 2026 03:41:50 +0000

Two tools can both claim to support “Markdown” and still disagree about tables, task lists, bare URLs, and even the HTML element used for strikethrough.

That is usually not a parser bug. It is a contract problem.

“Markdown support” often collapses three separate layers into one label:

a base syntax such as CommonMark;
extensions such as GitHub Flavored Markdown;
platform-specific configuration, sanitization, and post-processing.

I ran a small differential test to make those boundaries concrete.

Fixed environment

Tested on August 15, 2026:

commonmark 0.31.2
markdown-it 14.3.0, default preset, no third-party plugins
marked 18.0.7, default options

The comparison records generated HTML, not screenshots. That distinction matters: two results can look similar while producing different DOM structures for accessibility, export, or later transforms.

Minimal setup:

import * as commonmark from "commonmark";
import markdownit from "markdown-it";
import { marked } from "marked";

const source = "before ~~deleted~~ after";
const reader = new commonmark.Parser();
const writer = new commonmark.HtmlRenderer();

console.log(writer.render(reader.parse(source)));
console.log(markdownit().render(source));
console.log(marked.parse(source));

These are versioned defaults, not permanent labels for the libraries. Options and plugins can change the result.

1. A pipe table is not base CommonMark

Input:

| A | B |
|---|---|
| 1 | 2 |

Observed structure:

Parser	Result
commonmark 0.31.2	One paragraph; pipes remain text
markdown-it 14.3.0	A `table` with `thead` and `tbody`
marked 18.0.7	A `table` with `thead` and `tbody`

CommonMark 0.31.2 does not define pipe tables. GFM adds them as an extension.

When I need a controlled table input for a compatibility test, I use a Markdown table generator to avoid accidental delimiter mistakes, then verify the output with the actual target renderer. Generating valid source is not the same as proving platform compatibility.

2. A pipe inside a code span is a hard case

Input:

| A | B |
|---|---|
| `x|y` | a\|b |

With the tested markdown-it and Marked defaults, the unescaped pipe inside the code span still splits the row. One cell receives the opening backtick plus x; the next receives y plus the closing backtick, instead of one code span.

GFM’s table examples require escaping that pipe even inside a code span:

| A | B |
|---|---|
| `x\|y` | a\|b |

This is why splitting a table row with source.split('|') is not a parser.

3. A table-like shape can fail correctly

Input:

| A | B |
| 1 | 2 |

All three parsers emitted a paragraph. Without the delimiter row, none guessed that the author intended a table.

For editor diagnostics, checking for a real table node is safer than checking whether the source contains pipes.

4. Strikethrough has two kinds of differences

For:

before ~~deleted~~ after

commonmark keeps the tildes as text;
markdown-it emits <s>;
Marked emits <del>.

For the boundary case:

before ~not deleted~ after

only Marked recognized strikethrough in this test.

There are two contracts here: whether the syntax is recognized, and which HTML structure represents it. Screenshot tests can miss the second one.

5. Task lists may be plain text or form controls

Input:

- [x] shipped
- [ ] pending

commonmark and the markdown-it default produced normal list items containing [x] and [ ]. Marked produced disabled checkbox inputs and marked the first one as checked.

Whether a host then makes those boxes interactive is a UI and security decision, not something the Markdown syntax alone can decide.

6. Angle-bracket autolinks and bare URLs are different features

All three parsers linked this CommonMark form:

<https://example.com/a_(b)>

But for:

Visit https://example.com/a_(b).

only Marked linked the bare URL under the tested defaults. It also excluded the trailing period from the destination.

markdown-it can add this behavior with linkify; that option was deliberately left off so the matrix describes the default preset.

7. Raw HTML policy is not HTML safety

Input:

<xmp>**not emphasis**</xmp>

commonmark and Marked preserved the xmp element. markdown-it escaped the tags because raw HTML is disabled by default. All three still parsed the emphasis inside.

GFM has a tag-filter extension for names such as script, style, iframe, and xmp. That filter is not a complete HTML sanitizer. Marked’s own documentation also warns that its output is not sanitized.

A production pipeline therefore needs two explicit answers:

What HTML can the parser generate?
What HTML does the sanitizer allow?

The resulting capability matrix

For these exact versions and defaults:

Capability	commonmark	markdown-it	Marked
Angle-bracket autolink	Yes	Yes	Yes
Pipe tables	No	Yes	Yes
`~~x~~` strikethrough	No	Yes	Yes
`~x~` strikethrough	No	No	Yes
Task-list checkboxes	No	No	Yes
Bare-URL autolinking	No	No	Yes
Raw HTML enabled by default	Yes	No	Yes
Complete HTML sanitization	No	No	No

“No” is not automatically a defect. Disabling raw HTML can be a deliberate safety boundary. markdown-it can gain more syntax through plugins. The important point is that version and configuration belong in the contract.

A more useful declaration

Instead of only storing:

{ "flavor": "gfm" }

an application could expose something closer to:

{
  "base": "commonmark-0.31.2",
  "extensions": {
    "tables": true,
    "strikethrough": "single-or-double-tilde",
    "taskList": true,
    "extendedAutolink": true,
    "tagfilter": false
  },
  "rawHtml": "parse-then-sanitize"
}

That object is testable, versionable, and much harder to misunderstand than a dialect name.

The open question is whether Markdown ecosystems should standardize a machine-readable capability manifest—or whether every platform will continue documenting its dialect through examples and surprises.

DEV Community: frank

Which AI Works: A Practical AI Tools Directory

Start with the job, not the model

A repeatable way to find and compare tools

What the rankings can—and cannot—tell you

Product pages are the bridge to evaluation

A useful extra for builders

When to use Which AI Works

Building Reliable Decision Workflows with Jev AI: A TypeScript Guide

A production shape for a decision workflow

What Jev returns

Call the API from a TypeScript server

Treat the provider as part of your architecture

Evaluate before automation, then roll out gradually

Six systems in a JevBench snapshot

When Jev fits—and when it does not

References

A DOCX Style Map Is an API Contract, Not a Font Heuristic

Start with two independent signals

A custom style needs an agreed meaning

Test the intermediate representation

Boundaries that change the acceptance criteria

HTML-to-Markdown Round Trips Are a Semantic Contract, Not a Backup

Start with a case that should work

One difficult fixture exposes the many-to-one mapping

Malformed HTML adds a parsing stage you may forget

Boundary behavior needs exact output, not a green message

Test a semantic contract

Preserve the source when the source matters

Source Positions Are a Coordinate System, Not Just Line and Column

One emoji, three valid offsets

Positions should be half-open ranges

Four parsers, four position capabilities

Test failure syntax, not only valid syntax

A practical range contract

What I would regression-test

Snapshot Tests Lie About Markdown: Assert Structure Instead

The experiment

The differences were the useful result

Start with upstream conformance fixtures

Add one permanent fixture per real bug

Use snapshots as evidence, not as the oracle

Property tests need real invariants

Keep pathological tests separate

A practical test layout

Sources

Regex Cannot Sanitize Markdown: Put the Security Boundary After Parsing

A regex that passes the easy test

The fixed-version test

Parsing, sanitizing, and resource policy are separate

Do not modify sanitized HTML afterward

A practical review checklist

Primary sources

Invisible Unicode Characters Can Break Markdown Without Looking Different

Make the invisible input observable

What three fixed parser versions produced

A BOM is positional

Zero width does not mean zero semantics

Look-alike punctuation is different punctuation

Bidirectional controls require a different review model

A practical ingestion policy

Markdown Images Are Network Requests, Not Embedded Assets

Parsing and URL resolution are separate stages

Six cases that need different policies

Remote images are third-party dependencies

Relative paths need a publishing contract

Data URLs trade path stability for other costs

Validate the final request graph

Primary references

Your Markdown Parser Is Not Your XSS Boundary

The smallest useful pipeline

Five cases that expose the boundary

1. Normal content survives

2. Raw HTML is valid Markdown, not necessarily safe HTML

3. URL schemes need their own policy

4. Code examples must not be cleaned as attacks

5. XSS defenses extend beyond script

Put sanitization after the last unsafe transform

What I verify in a conversion workflow

Engineering checklist

5. XSS defenses extend beyond `script`