DEV Community

Cover image for pax Recovered the File tar Swallowed from the Same Archive
Mustafa ERBAY
Mustafa ERBAY

Posted on Originally published at mustafaerbay.com.tr

pax Recovered the File tar Swallowed from the Same Archive

In the post where the same commit got packed twice,
published on 3 November, something turned up as an aside: macOS tar writes
members into the archive that I never put there, and then doesn't list them
back to me. The direction there was outbound — the Mac packs, the other side
finds extra files.

This post is the inbound face of the same mechanism. The question: if an
archive arriving from the other side holds a real file whose name begins
with ._, what does macOS do?

I tried six different modes. In all six the file never reached disk; in four
tar errored and exited 1, in two it said nothing at all and exited 0. And
pax, from the same machine's /bin, pulled the same file out of the same
archive with its contents intact.

"Why would such a file even exist"

That's the first objection, and it's a fair one. But the ._ prefix is special
only to macOS; on Linux and BSD it's an entirely ordinary name. An editor
backup, a tool's output, a deliberately hidden helper file, or a twin that once
came from a Mac and settled in on the other side — any of them can start with
._, and to the other side they are simply data.

So the lab lives on Linux. Two files: one ordinary, one a real text file whose
name starts with ._. (The Turkish line inside it, BU BIR DUZ DOSYA, means
"this is a plain file" — it matters later, because it is plainly not metadata.)

docker run --rm -v "$PWD":/ark alpine:3.21 sh -c '
  mkdir -p /ark/tersyon/ic && cd /ark/tersyon
  echo gercek > ic/veri.txt
  printf "BU BIR DUZ DOSYA\n" > ic/._veri.txt
  tar czf /ark/tersyon.tgz ic'
Enter fullscreen mode Exit fullscreen mode

Both are in the archive, and on Linux both land on disk:

$ docker run --rm -v "$PWD":/ark:ro alpine:3.21 tar tzf /ark/tersyon.tgz
ic/
ic/._veri.txt
ic/veri.txt
Enter fullscreen mode Exit fullscreen mode

The environment: macOS 26.6.2, bsdtar 3.5.3 - libarchive 3.7.4, and on the
other end busybox tar 1.37.0 inside alpine:3.21 (Docker 27.4.0). I write the
versions down because this whole post hangs on the question "which tool".

The listing is already short

Before extracting anything I wanted to list the archive. Two tools, two
answers:

$ tar tzf tersyon.tgz        $ pax -zf tersyon.tgz
ic/                          ic
ic/veri.txt                  ic/._veri.txt
    → 2 members              ic/veri.txt
                                 → 3 members
Enter fullscreen mode Exit fullscreen mode

This archive wasn't made on a Mac; it was packed on Linux, with nothing called
AppleDouble anywhere in it. tar drops ._veri.txt from the listing anyway.
The loss doesn't begin at extraction — it begins while you're still looking at
the table of contents: the first command you ask about the archive's contents
gives you an incomplete answer.

Six modes, two of them silent

I opened the archive on macOS:

$ tar xzf tersyon.tgz -C chk
ic/veri.txt: Failed to restore metadata: File exists
tar: Error exit delayed from previous errors.
$ echo $?
1
$ find chk -type f
chk/ic/veri.txt
Enter fullscreen mode Exit fullscreen mode

veri.txt is there, ._veri.txt is not. tar didn't see it as a file; it
took it for its neighbour's metadata and tried to apply it. I tried every flag
I could think of:

mode exit code files written message
default 1 1 Failed to restore metadata
--mac-metadata 1 1 same message
--no-xattrs 1 1 same message
-p 1 1 same message
--no-mac-metadata 0 1 none
--no-same-permissions 0 1 none

Six modes, one outcome: in none of them does the file reach disk. I could not
find a flag that persuades tar to write that member out as a plain file.

The last two rows are the ones that bother me. With those flags the command
finishes successfully, you get not a single warning, and the file still isn't
there. The modes that error are being honest; the modes that report success
make the loss invisible. In a script running under set -e, the first four
rows stop you and the last two don't.

A small addendum: the tar(1) man page on this same machine says
--no-mac-metadata is "the default behavior if tar is run as non-root in x
mode"
. I ran the measurement as uid 501, and the default run tried to restore
metadata and failed — it did not behave like the fifth row. That sentence
doesn't hold for the version I have.

The decision is made on the name, not the contents

What's inside ._veri.txt? Plain text. The magic number of the AppleDouble
format — 0x00051607, defined by RFC 1740 — is nowhere in it. tar took it
for metadata anyway. It decides not by looking at the contents but at the
first two characters of the name
.

The prefix itself is one line in Apple's copyfile source:

#define ATTR_FILE_PREFIX   "._"
Enter fullscreen mode Exit fullscreen mode

I isolated the name-based rule with a second experiment: this time I put a lone
._abc.txt into the archive, with no sibling (its contents a 13-byte line).

$ tar xzf yalniz.tgz
ic/._abc.txt: Truncated tar archive: Unknown error: -1
tar: Error exit delayed from previous errors.
$ ls -l ic/._abc.txt
-rw-r--r--  1 mustafaerbay  wheel  13 Nov 11 17:19 ic/._abc.txt
Enter fullscreen mode Exit fullscreen mode

The file is on disk, with its name and its size. But look inside:

$ xxd ic/._abc.txt
00000000: 0000 0000 0000 0000 0000 0000 00         .............
$ tr -d '\000' < ic/._abc.txt | wc -c
       0
Enter fullscreen mode Exit fullscreen mode

All thirteen bytes are zero. tar tried to parse the file as AppleDouble,
decided the contents were a truncated archive, gave up halfway, and left behind
a shell with the right size and nothing in it. That's worse than the first
experiment: there the loss was at least visible in a file count, here both the
file count and the size look correct.

Two experiments, two outcomes, one rule: if the name starts with ._, nobody
looks inside.

Diagram

Three tools in the same box: two lose it, one recovers it

"Open it on Linux" is an easy thing to say, but the framing is wrong. The
problem isn't the operating system, it's the tool's code — and I can show that
inside macOS alone. I had three built-in tools extract the same archive:

tool written to disk ._veri.txt
tar xzf 1 file missing
cpio -id 1 file missing
pax -rzf 2 files with its contents

Here is what pax prints:

$ cd paxcik && pax -rzf ../tersyon.tgz
pax: Unable to set metadata on ic/veri.txt <File exists>
$ find . -type f
./ic/._veri.txt
./ic/veri.txt
$ cat ic/._veri.txt
BU BIR DUZ DOSYA
Enter fullscreen mode Exit fullscreen mode

Same machine, same archive, a message from the same "File exists" family — but
the file is on disk with its contents intact. pax tries to apply the metadata
and, when it can't, writes the member out as a plain file. tar and cpio
give up. All three ship in macOS's /bin or /usr/bin; no Docker, no brew.

The man page contradicts itself

While hunting for the flags in this post I tripped over the local
documentation twice.

First: the same tar(1) page introduces the same pair of flags with two
different mode lists.

$ man tar | col -b | grep -A1 -- '^     --\(no-\)\?mac-metadata$'
     --mac-metadata
             (c, r, u and x mode only) Mac OS X specific.  Archive or extract
--
     --no-mac-metadata
             (x mode only) Mac OS X specific.  Do not archive or extract ACLs
Enter fullscreen mode Exit fullscreen mode

One says "create, append, update and extract", the other says extract only —
even though they're defined as each other's reverse. I tried which one is
right: tar --no-mac-metadata -czf works fine in create mode and produces a
twin-free archive. The line upstream libarchive writes for the same flag is
already "(c, r, u and x mode only)". The second entry's mode list is incomplete
in the version I have.

Second, and more interesting: the COPYFILE_DISABLE variable. The page's
ENVIRONMENT section has five entries (TAR_READER_OPTIONS,
TAR_WRITER_OPTIONS, LANG, TAPE, TZ) and this isn't one of them; it
isn't in the 591-line copyfile(3) page either, which has no ENVIRONMENT
section at all. The name is defined in copyfile.h:

#define COPYFILE_DISABLE_VAR    "COPYFILE_DISABLE"
Enter fullscreen mode Exit fullscreen mode

But the only code in copyfile.c that reads that name is switched off:

#if 0
    /* Temporarily disabled */
    if (getenv(COPYFILE_DISABLE_VAR) != NULL)
Enter fullscreen mode Exit fullscreen mode

So the library isn't the thing listening to the variable. Instead of guessing
where it is, I asked:

$ strings /usr/bin/tar | grep -i 'copyfile_disable\|disable-copyfile'
COPYFILE_DISABLE
disable-copyfile
Enter fullscreen mode Exit fullscreen mode

Both strings live inside the tar binary. It's tar itself that reads the
variable; the definition in the source only supplies the name, and the check
moved out of the library. One more detail: the check looks at presence, not
value
— setting COPYFILE_DISABLE=0 also suppressed the twins, taking the
archive from twelve members down to six. A reader who writes zero to turn it
off gets exactly the opposite.

A "clean" archive still carries your attributes

A side measurement, because the common advice stops short here. An archive
built with COPYFILE_DISABLE=1 or --no-mac-metadata has no ._ members —
but that doesn't mean the attributes are gone. I scanned all three archives:

$ for a in varsayilan kapali nomac; do echo -n "$a: "; gzcat $a.tgz | strings | grep -c 'LIBARCHIVE.xattr'; done
varsayilan: 9
kapali: 9
nomac: 9
Enter fullscreen mode Exit fullscreen mode

Nine headers in each, belonging to three distinct attribute names (six
com.apple.provenance, two of my own com.ornek.not, one
com.apple.quarantine). So the archive I called clean carries the quarantine
tag of a downloaded file to the other side; only the AppleDouble members are
gone.

Do those headers do anything? I extracted two archives with two tools and
compared:

archive extracted by did com.ornek.not come back
default (AppleDouble present) tar yes
default (AppleDouble present) pax yes
--no-mac-metadata (pax header only) tar no
--no-mac-metadata (pax header only) pax no

On the macOS side what restores attributes is the AppleDouble member; the
archive format's extended headers (LIBARCHIVE.xattr.* — these belong to the
pax format, not to the /bin/pax tool) travel but are never applied.

Linux only reads those headers if you ask explicitly. A plain tar -xzf says
"unknown" to each of them; with --xattrs --xattrs-include='*' it tries, and
runs into a different wall:

tar: setxattrat: Cannot set 'com.apple.quarantine' extended attribute for
file 'kaynak/indirilen.txt': Operation not supported
Enter fullscreen mode Exit fullscreen mode

My first reading was "this filesystem doesn't support extended attributes".
Trying three names on the same file showed otherwise:

user.test              → OK
com.apple.quarantine   → Operation not supported
security.foo           → Operation not permitted
Enter fullscreen mode Exit fullscreen mode

The filesystem takes attributes fine; what refuses is Linux's namespace rule.
A user attribute's name has to start with user., so com.apple.* can't be
written on any Linux filesystem. The names and values travel to the other side,
but they aren't reborn there as Mac attributes.

Cleaning up on the receiving side: dot_clean and its three modes

If you've ended up with a tree full of ._ files, macOS has a tool for it and
its man page describes it in one line: "Merge ._ files with corresponding
native files."* I ran all three modes over the same tree:

command twin files did com.ornek.not land on the file
dot_clean <dir> (default) deleted no
dot_clean --keep=dotbar <dir> deleted yes
dot_clean -m <dir> deleted no

No surprise for -m: the man page says "Always delete dot underbar files" —
a delete mode. The one to watch is the default. Its verbose output claims it
did the job:

$ dot_clean -v n2
Merging directory altdizin
merging ._ic.txt & ic.txt
Using the most recent stored info
Deleting: ._ic.txt
Enter fullscreen mode Exit fullscreen mode

Yet afterwards xattr -l shows no com.ornek.not on the file. By the
documentation it should be there: DESCRIPTION says the most recent copy wins
for an attribute that is also present on the native file, and the
--keep=mostrecent entry says that otherwise the information stored in the
AppleDouble file is used. This attribute wasn't on the native file, so by the
rule it should have landed. With --keep=dotbar the output changes
("Replacing ic.txt's EAs with ._ic.txt's EAs") and the attribute arrives. The
default mode doesn't do what its own documentation describes; cleaning up and
restoring are not the same mode of the same command.

They pass extension-based filters

The sneaky part about the twins isn't their count, it's that their names look
legitimate. A ._report.mdx matches the *.mdx pattern. I tested this on my
own publishing pipeline; the script that moves the blog's MDX files into the
database finds them like this:

const files = execSync("find src/content/blog -name '*.mdx'", ...)
Enter fullscreen mode Exit fullscreen mode

The pattern catches the twins — not a guess, a measurement:

$ find . -name '*.mdx' | sort
./technology/._gercek.mdx
./technology/gercek.mdx
Enter fullscreen mode Exit fullscreen mode

In a pipeline that moves content through an archive, these files get taken for
posts. Mine takes content from git, so the scenario is theoretical for now —
and the measurement came back clean:

$ find ~/IdeaProjects -name '._*' -type f | wc -l
0
Enter fullscreen mode Exit fullscreen mode

Zero. It scanned in two and a half seconds and found none, because I don't
extract Mac-made archives into my own tree. This mechanism doesn't accumulate
locally; the damage happens exactly at the boundary, where one machine packs
and another unpacks. That's the good half of the bad news: the number of places
to check is small and well defined.

Checklist — for the receiving side

  • Don't let tar do the counting. The gap between two listings shows the loss in one line: if pax -zf a.tgz | wc -l and tar tzf a.tgz | wc -l disagree, the archive holds ._ members. In a script, a gate comparing those two numbers catches the pipeline that loses files quietly under --no-mac-metadata.
  • The exit code isn't enough. --no-mac-metadata and --no-same-permissions return 0 and don't write the file; trusting set -e alone hides the loss.
  • Use pax to recover. If you have an extraction that genuinely needs members named ._something on disk, use pax -rzf; tar has no flag that does it.
  • If you clean up, choose the mode. Just getting rid of the twins: dot_clean <dir>. Also wanting the attributes inside them back: dot_clean --keep=dotbar <dir>.
  • Remember the "clean" archive carries the quarantine tag. Even an archive with no ._ members takes attribute names and values along as pax headers; to really strip them you have to clean the source before packing.
  • Watch out for extension-based filters. ._report.mdx matches *.mdx too. Add -not -name '._*' to the find/glob patterns on the receiving side.

Wrapping up

What this measurement left me with isn't the lost file itself; it's that a tool
decided what a file is by looking at its name. Had it looked at the
contents, the plain-text line would have been obvious in the first bytes —
AppleDouble's magic number doesn't match. A name-based rule is fast and simple;
in exchange it's blind to anything that imitates the name.

And there's this: I wanted to write the sentence "macOS behaves this way", and
I couldn't. Of three tools in the same box, two lose the file and one recovers
it. What decides is not the operating system but that binary. Before I write my
next "platform X does it this way", I'll try asking the second tool on the same
platform.

Official Sources

Top comments (0)