My Backup Was Crawling Because I Was Using Someone Else's API Quota

Originally published on my blog — read it there with the full context, code and updates.

The subscription ended, and I had to move a terabyte

The call came in the least convenient way: the OneDrive account holding my
nightly backups had no subscription any more, and I had a 1.5-year Google
subscription sitting unused. Move the backup, keep the history, don't lose a
night.

The first thing I hit was not a technical problem: I could not remember
how the backup was wired. rclone? Duplicati? Something Synology shipped?
On a three-host estate with 78 compose stacks — the one in
Running Production Infrastructure Solo
— "which tool writes my backups" is a question I should have been able to
answer from memory and could not. So step one was reading the machine.

What was actually there

Two rclone configs, which is the kind of detail that bites: the GUI config
that Backrest (the restic front end) uses, plus a separate stale CLI
copy that a cron job or a human shell had been using at some point. Change
one and you have changed half the system.

The migration itself is boring when done in the right order, and I did it
read-first:

rclone.conf                     append [gdrive-*] + [gdrive-*-crypt]
rclone/rclone.conf (CLI copy)   same two remotes
Portainer stack 177             env → the new crypt remote; config :ro → :rw
restic init                     new repo on Google Drive
Backrest SetConfig              dsm-repo URI → the new location

Before touching the plan URIs I checked one property in the source, because
everything depended on it: a restic repository's identity is
guid = sha256(id) — a stable value derived from the repo id, not from
its URL. So repointing a plan at a new backend cannot silently fork the
history. All six plans came out byte-identical to the originals, and the
repo id/guid were untouched.

Then the seed stopped moving, and the log named a stranger

The new repo seeded to 2.88 GB of roughly 1075 GB — 0.27% — and then slowed
to a crawl. The live log had the answer:

'Read requests per minute' ... reason: RATE_LIMIT_EXCEEDED
consumer: projects/202264815644        <- not my project
quota_limit_value: 840000

202264815644 is rclone's own shared OAuth project. My terabyte seed
was drawing on a per-minute read quota pooled with every rclone user on
earth, which is why it degraded from ~750 KB/s to nothing while my NAS, my
disk and my link were all idle. By the time I measured, the seed was fully
stalled — 0 bytes in 90 seconds, 159 errors, still rateLimitExceeded —
and the six nightly plans had all failed with the same wall:

Fatal: unable to save snapshot: unexpected HTTP response (500)
googleapi: Error 403: Quota exceeded for quota metric 'Queries'
  and limit 'Requests per minute' ...

"Slow" and "blocked" look similar in a progress bar and are completely
different problems. Ninety seconds of sampled zeros is what made the call.

You need your own OAuth client — and the console is unavoidable

There is exactly one real fix: stop using the shared client and bring your
own. That means a Google Cloud project, an OAuth consent screen, and a
credential. Two settings cost real time:

• Application type = Desktop app. rclone's redirect is http://127.0.0.1:53682, and that client type permits it without extra configuration.
• Publish the app. Leave it in "Testing" and Google expires your refresh token every 7 days — which turns "working" into a silent weekly outage.

I checked whether I could shortcut the console with the CLI that was already
authenticated here. I could not, and the reason is worth knowing: gcloud
was signed in only as a service account in an unrelated project, and
service accounts cannot create consent screens or OAuth client IDs. The
credential has to belong to a project a human owns. It was a dead end, and I
verified that instead of asserting it.

Three traps in the authorization flow

1. Publishing an unverified app can hard-block a restricted scope. .../auth/drive is restricted, and an app that is published without completing Google's verification review is sometimes blocked outright with no bypass. In my case the bypass existed — the "Google hasn't verified this app" screen, then Advanced → Go to rclone-backup (unsafe) — but that is a coin flip on scope and review state, not a guarantee, and "unsafe" is only a label meaning the app never went through a paid audit.
2. The callback accepts exactly one request. A reload, or a second tab, hits 127.0.0.1:53682 with no code and kills the attempt. My first try died exactly that way (no code returned), and it looks like a credentials problem. One tab. Don't reload it.
3. Finish the consent screen on the machine that runs rclone. The redirect goes to localhost, so a phone or another PC cannot complete it — the browser has to be the one on the same host as the process waiting on that port.

Then: read the token, write client_id + client_secret + token into
both configs (the GUI one and the stale CLI copy), and restore the two
flags that matter for a long unattended upload: use_trash = false and
stop_on_upload_limit = true.

The old side: verify by reading, not by asking

Purging the old remote reported an error — itemNotFound — and the space
came back anyway:

Used:   1.012 TiB  ->  3.800 GiB
Free:   ~0         ->  1.007 TiB
DSM folder: present -> gone      Trashed: 0 B

The error was rclone complaining after the work finished. Also worth
recording: the tool's documentation says personal accounts cannot
permanentDelete, which would have meant emptying the recycle bin by hand —
in practice the trash was already empty and roughly 1 TB was reclaimed with
no browser step. Documentation describes the API; the API is not always what
you get.

One more measured thing, because it changed my next decision: pausing the
file-sync container that shared that machine halved the load average — 16.70
→ 8.89 — which is the difference between "the network is slow" and "the box
is busy".

What I would do differently

1. Own the OAuth client from day one. A shared client id is a queue you are standing in, with no status page and no one to call.
2. Answer "what writes my backups" from a document, not from memory. Two rclone configs is a normal amount of drift for a system nobody has drawn in a year.
3. Read the destination, not the exit code. The purge "failed" and freed a terabyte; a successful-looking rename can land nothing. Same lesson as It Reported Success. Nothing Had Changed. — this estate keeps teaching it.
4. Never start a multi-day upload while the last few percent of your attention is elsewhere. Checking for RATE_LIMIT_EXCEEDED at 0.27% costs minutes; discovering it after two days costs the window.

The result, honestly stated

The plumbing works: the plans live on Google Drive, the repo identity is
unchanged, the seed is now spending my quota instead of a shared one,
and the old side gave back about a terabyte. What it is not, yet, is a
backup: at the time I wrote this, the first seed had uploaded 2.88 GB of
~1075 GB and the new repo had zero snapshots — nothing is restorable
until the first snapshot lands. That is the number I watch, because "the
transfer is running" and "you can restore" are different claims.

---

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Comments

[Jisung Yoon]

The zero-snapshot status deserves its own visible alert, separate from upload progress. Once the seed finishes, a small restore check with a Korean filename and matching file hash would test something the progress bar cannot prove.
Will you include a restore result in the follow-up?

[Paul Irolla]

Tokenade is mine, and agent tools hit the same trap: a shared key pools the quota of everyone who uses it. Give each backup job its own key, so a stall stays with one job instead of the whole pool.