DEV Community: Igor

Four ways a file-sync tool eats your data, and how to not

Igor — Sun, 26 Jul 2026 17:15:54 +0000

I've been building a small sync tool — it moves files between a remote project and a local folder, git-style (clone, pull, push, status). The moment you write a tool that touches other people's files on both sides of a network, you inherit a category of bug that's worse than a crash: silently destroying work the user can't get back.

Here are four data-loss traps I hit (or nearly hit), each with the fix. None are exotic. All of them pass a naive test suite while being wrong.

1. `os.WriteFile` truncates before it writes

The obvious way to save a downloaded file:

os.WriteFile(path, data, 0o644)

O_TRUNC means this empties the file, then writes. If the process is killed between those two steps — Ctrl-C, OOM, a laptop lid — you're left with a dense prefix of the new bytes and none of the old. The user's file is now half a file.

The fix is the same trick databases use: write to a temp file beside the destination, then rename:

tmp, _ := os.CreateTemp(filepath.Dir(path), ".tmp-*")
tmp.Write(data)
tmp.Close()
os.Rename(tmp.Name(), path) // atomic on the same filesystem

rename(2) is atomic within a filesystem, so any reader sees either the whole old file or the whole new one, never a fragment. Two things that bite here: the temp file must be in the same directory (a temp in /tmp gives you EXDEV: cross-device link the moment someone's on a different mount), and rename swaps the inode, so hard links and xattrs on the destination don't survive — an accepted cost, but know you're paying it.

The second-order damage is the sneaky part. If your write happens before you record the file in your local ledger, then a killed write leaves a fragment your tool doesn't know it wrote. Next run, the tool compares the fragment against the server, sees a difference, and reports "local file changed — use --force to overwrite." You corrupted the file, then blamed the user, then offered them the one destructive flag as the cure. Atomic writes kill the whole chain.

2. Conflict detection on timestamps or etags can't see the case that matters

The tempting cheap conflict check: compare the server's etag (or mtime) to the one you saved. Different? Re-download.

The case this misses is the only one that actually loses data: both sides changed. If the user edited locally and the file changed on the server, an etag comparison just sees "server differs" and happily overwrites the local edit. Gone.

Conflicts have to be decided on the bytes, not on metadata:

local hash == last-synced hash → clean, take the remote
local hash != last-synced hash and remote changed → conflict, touch nothing
only one side changed → apply that side

Metadata is a fine fast path to skip work, but it can never be the thing that authorizes an overwrite. The authorization is the hash.

3. "Delete what's not on the server" deletes things you didn't sync

A --prune/mirror flag is where sync tools cause the worst damage, because deletion is the one operation with no undo. The naive rule — "if it's local but not remote, delete it" — will cheerfully remove:

a file the user created and never synced (it was never yours to delete)
a file the user edited locally (that's a conflict, not a deletion)
a file that legitimately only ever lived locally

The rule has to be: delete only what you can prove you put there and the user didn't touch. That proof is your ledger — the record of "I wrote these exact bytes at this version." No ledger entry → not yours → leave it. Ledger entry but local bytes differ → conflict → leave it.

And a subtle one: if the operation is interrupted halfway, a naive tool sees a partial local tree and concludes the missing files were "deleted on the server," then prunes the real ones. An interrupted run has to record itself as a failure, not as a short success, or your prune logic reads a crash as a batch of user deletions.

4. When the data is a shape you don't recognize, refuse — don't guess

The tool reads a file listing from the server to decide what to sync. What happens when that listing comes back as null because of a server hiccup?

In Go, json.Unmarshal("null", &slice) succeeds and leaves you with a nil slice and no error. Unguarded, your code now believes the project has zero files — and on the next push --prune, "zero files on the server" means "delete everything locally." A transient null just wiped the user's tree.

The defense is a mindset, not a line: untrusted input that will drive a destructive action must be validated for shape, and anything unexpected must stop the operation, not flow through it. I explicitly reject a null listing, a listing that hit the pagination cap (at the boundary, a complete listing and a truncated one are identical — so I refuse rather than risk it), and any decode whose length disagrees with the size the server advertised. A needless refusal costs one retry. A wrongly-trusted listing costs the user's files.

The through-line

All four bugs pass a happy-path test. You only catch them by asking a different question than "does it work?" — namely "if this dies right now, or lies to me right now, what does the user lose?" For anything that writes files or deletes them, that question is the design.

I wrote these up because the tool is open source and the reasoning is the interesting part — it's a single Go binary, standard library only, and every one of these decisions has a test pinning it (including a few that only exist because a mutation showed the guard was silently not guarding). If you want to see them in situ: https://github.com/somework/dsx

What's your worst "the tool ate my files" story? I collect them.

I reverse-engineered Claude's undocumented Design API so a file sync wouldn't cost 665k tokens

Igor — Sun, 26 Jul 2026 16:26:52 +0000

A build log about protocol archaeology, three wrong guesses, and refusing to trust my own mocks.

The token bill that started it

Claude Design (claude.ai/design) is a nice place to keep a design system, but the files live there, on the server. The obvious way to get them onto disk is to ask an agent: "read every file in this project and write it locally." That works. On one large project I measured, it also cost about 665,000 tokens — because every byte of every file has to pass through the model's context on the way out.

That number is silly. Moving bytes from A to B is not a reasoning task. The model is being used as a very expensive cp.

So I wanted a plain CLI that syncs a Claude Design project to a local directory the way git syncs a repo — clone, pull, push, status, diff — where the file bytes never touch a model's context. The same pull that cost 665k tokens should cost one summary line:

pulled 103, unchanged 0, binary 6 (660.1 KB)

There was one problem: the endpoint that Claude Design speaks to is undocumented.

Protocol archaeology

There's no public spec. The endpoint is an MCP server, so the shape of the transport is known (JSON-RPC, tool calls), but which tools exist, what arguments they take, and what they return — that's all unwritten. The only way to learn it is to call it and write down what comes back.

That's fine; archaeology is a legitimate way to learn an API. The trap is what you do with what you learn. It is very easy to call an endpoint a few times, form a belief about how it behaves, encode that belief into a mock, and then test your code against the mock. Everything goes green. You ship. And you've tested nothing except that your code agrees with your own assumptions.

I know this because I got three protocol facts wrong before I measured them, and a green mock would have hidden every one:

write_files returns a map, not a list. I assumed a list of results, one per file. It's keyed by path. A mock returning a list would have passed my tests and mis-parsed the real server.
The "you don't have access to this project" case is an HTTP 403, not a tool-level error. I'd modelled it as an error object inside a normal 200 response. It's a status code — a different code path entirely.
Binary detection is by content, not by file extension. I assumed the server keyed off the filename. It reads the bytes. A .txt full of NULs is binary to the server; an extensionless script is text.

None of these are exotic. They're the kind of thing you're 80% sure about and wrong. So the rule became: mocks test my own handling; the protocol facts get verified against the real server.

Concretely, the repo has a PROTOCOL.md where every documented claim has a corresponding live test (go test -tags=live) that hits the actual endpoint. If Anthropic changes the server and a claim stops being true, that test goes red — instead of the tool silently doing the wrong thing to your files. The mocks still exist, but they only ever test my own logic, never a fact about the world.

The part I was more scared of than tokens

Once bytes stop flowing through a model, the tool is just a sync engine talking to a server — and sync engines lose data. That became the second obsession after the token cost. A few of the decisions:

Every file is written temp-then-rename. A killed process never leaves a half-written file. This matters more than it sounds: a truncated file that also never made it into the tool's ledger looks, on the next run, like you edited it — so the tool would "helpfully" offer to overwrite your work. Atomic writes kill that whole class of bug.
Conflicts are decided on bytes, never on timestamps or etags. If both sides changed, that's a conflict, full stop. An etag comparison literally cannot see the both-sides-changed case.
--prune deletes only what it can prove was ours and unmodified. Untracked file → not ours, leave it. Locally edited → a conflict, don't touch it. The proof is an on-disk ledger, and twice during this project an attempt to make that proof "smarter" taught the prune loop to delete something it shouldn't have. Both were caught by tests, and both are written up in the repo, because a deleted-someone's-file bug is worth a paragraph so the next person doesn't reinvent it.
The credential is read, never written, never printed. The tool reads Claude Code's existing OAuth token so there's no separate login. It will not even wrap that token into an error string. (Refreshing it would rotate the refresh token out from under Claude Code, so it doesn't; an expired token is reported, and the fix is to run any claude command.)

stdlib only

The whole thing is a single Go binary with zero third-party dependencies — go.mod names none. The reason is auditability: everything that parses untrusted bytes from an undocumented server and touches your credential is the Go standard library plus this one program. staticcheck, govulncheck and the release tooling run in CI without ever entering the dependency graph. If you're going to point a tool at an unofficial endpoint with your OAuth token, "you can read all of it" is worth a lot.

The honest caveat

The endpoint is undocumented and unofficial. A server deploy can break it. I can't make that risk zero — but I tried to make it visible rather than hidden: the live tests are the tripwire, and the tool refuses and reports rather than guessing when the server answers in a shape it doesn't recognise.

It's called dsx, a single binary, macOS and Linux, MIT. If you use Claude Design and have felt that token bill — or you just like protocol-archaeology writeups — the code and the PROTOCOL.md are here:

https://github.com/somework/dsx

Happy to answer questions about the protocol or the data-loss reasoning.