DEV Community

Cover image for How a missing @ in a filename broke my Netlify build
Osmium
Osmium

Posted on • Originally published at osmiumsilver.github.io

How a missing @ in a filename broke my Netlify build

#webdev #netlify #javascript #troubleshooting

It was fine three months ago :(

Three months ago I deployed my Qwik site to Netlify. And after some while, I pushed an new, unrelated content change and the build died without touching a thing on it’s codebase:

Edge Functions bundling
────────────────────────────────────────────────────────────────

Packaging Edge Functions from .netlify/edge-functions directory:
 - entry.netlify-edge

[Error: ENOENT: no such file or directory, stat '/tmp/tmp-2269-6qAdCcbGEFml/qwik-city-not-found-paths.js'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'stat',
  path: '/tmp/tmp-2269-6qAdCcbGEFml/qwik-city-not-found-paths.js'
}
Node.js v22.22.1
Enter fullscreen mode Exit fullscreen mode

No version bumps in package.json. No config changes. The build itself (client, server) compiled cleanly. If it worked three months ago with the exact same versions, why was it broken now?

I did the usual steps: Cleared cache and ran the build, failed. But build process worked fine on local machine. Upgraded all dependencies with ncu to their latest versions, pushed again, still failed. Until I created a brand new Qwik hello-world project, pushed it to Netlify. Still failed, that’s when I knew there must be something wrong on Netlify’s side.

Searching online for "qwik netlify build error ENOENT file not found" wasn't really helpful, found only one forum post describing the same issue. No replies. Auto-closed after seven days. So I posted on Netlify's answers forum and hoping someone could give a solution. A response did come eventually, but hours later. The site was down and needed to come back up ASAP, so I wasn't going to sit there refreshing a forum tab. I started digging deep.

The First Clue: A Missing @

I noticed the failure was in the Edge Functions bundling step, an additional step after everything else was done. I opened the output directory on my local machine expecting to see the missing file:

.netlify/edge-functions/entry.netlify-edge/
├── @qwik-city-not-found-paths.js  ← Here it is!
├── @qwik-city-plan.js
├── @qwik-city-static-paths.js
├── entry.netlify-edge.js
└── ...
Enter fullscreen mode Exit fullscreen mode

That's when it gets interesting, the file existed. It just had an @ in front of it. Yet the error occurs when attempting to call stat on a file with a filename that is almost identical, except for the missing @?

If the bundler knew this file existed, why was it looking for a file with the wrong name?

Something was stripping the @ somewhere in the process.

Following the Trail

I pulled down the source of @netlify/build from Netlify’s GitHub repo including the package edge-bundler, which responsible for that Edge Functions bundling step. I followed the call chain:

edge_functions/index.ts → bundler.ts → formats/tarball.ts

In tarball.ts, I found the function file list being built:

const files = (await listRecursively(bundleDir.path))
  .map((p) => path.relative(bundleDir.path, p))
  .map((p) => getUnixPath(p))
  .sort()

await tar.create({ cwd: bundleDir.path, file: tarballPath, gzip: true }, files)
Enter fullscreen mode Exit fullscreen mode

the code packs bundled edge functions into a tarball for deployment. It calls tar.create() with a list of filenames. And at that point, the filenames still had their @ prefix intact. The correct name was being passed in. So the issue wasn't in the path calculation. The @ was being stripped inside tar.create() itself.

Reproducing the Bug

To confirm, I put together a minimal reproduction with some help from Claude:

import * as tar from 'tar'
import fs from 'fs'
import os from 'os'
import path from 'path'

const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'test-'))
fs.writeFileSync(path.join(tmpDir, '@test-file.js'), 'export default 1')

// Without fix: ENOENT — @ is stripped
await tar.create(
  { cwd: tmpDir, file: '/tmp/out.tar.gz', gzip: true },
  ['@test-file.js']
)
Enter fullscreen mode Exit fullscreen mode

And i got:

[Error: ENOENT: no such file or directory, stat '.../test-file.js']
Enter fullscreen mode Exit fullscreen mode

The bug was confirmed: node-tar WAS doing something on filenames that result in the removal of the @ symbol.

Root Cause: bsdtar (libarchive)’s @archive syntax

After some digging online, bsdtar/libarchive has a feature where entries starting with @ have special meaning: @archive.tar means "open that archive and include its contents". node-tar, as a tar implementation, inherited this behavior. When it sees a filename starting with @ in the files list, it strips the @ symbol and tries to open the remainder as a tar archive to read entries from.

So when node-tar anything starts with a @ symbol:

→ Interpret this as an archive-include directive
→ It tried to open qwik-city-not-found-paths.js as a tar archive
→ Run stat on qwik-city-not-found-paths.js
→ ENOENT

The Fix

In tarball.ts, where the files array is passed to tar.create(), prefix each entry with ./:

// Before
const files = (await listRecursively(bundleDir.path))
  .map((p) => path.relative(bundleDir.path, p))
  .map((p) => getUnixPath(p))
  .sort()

// After
const files = (await listRecursively(bundleDir.path))
  .map((p) => path.relative(bundleDir.path, p))
  .map((p) => './' + getUnixPath(p))  // ← this line
  .sort()
Enter fullscreen mode Exit fullscreen mode

The Chain Reaction

PR got merged

After the PR was merged into @netlify/build, the maintainer traced the issue further upstream and submitted a PR to node-tar itself — fixing the error handling so that errors from @-prefixed entries are properly catchable instead of becoming unhandled promise rejections that the caller couldn't catch.

Moreover, somehow it also uncovered a CI coverage gap: the test suite for tarball handling had a describe block that was being silently skipped because of a wrong Deno version number in the test configuration. An entire set of tarball tests had been passing CI without actually running.

One missing @, and suddenly there were three fixes flowing in different directions.

Looking Back

Reading someone else's TypeScript to find out why my site wouldn't deploy is not something I'd done before this week. The typical workaround for this kind of issue you would find online, is to duplicate the files without the @. That works. Your build passes. You move on. But somewhere the same bug is still waiting for the next person. And the next. Until someone reads the function that's eating the @.

There's something I've been thinking about since. The experienced developers who maintain these projects, they do this kind of invisible work all the time, and they do it faster than I ever could. They trace a bug, fix it, and the result is that thousands of people simply never encounter the problem. Nobody knows it was ever there.

I'm not a developer by trade. If these projects had been closed-source, my story would have ended at the forum post with no replies, waiting for someone on the other side to notice. But because the code was right there, I could pull it down and find the answer myself. A workaround solves your problem. A root cause fix solves everyone's.

Links

Top comments (0)