DEV Community

John
John

Posted on

Hubs Community Edition backup and restore: what npm run backup captures, what it misses, and how long recovery takes

#hubs #selfhosting #devops #backup

The bundled scripts in Hubs Community Edition back up two things: the PostgreSQL database and the Reticulum file store that holds uploaded GLB scenes, avatars, thumbnails and room media. They do not back up your hcce.yaml, your TLS material, your DNS records, your Kubernetes cluster or your object storage credentials, and if you point Hubs at an external database instead of the bundled Postgres pod, the scripts fall back to backing up the Reticulum files only. That means npm run restore-backup is not a disaster recovery plan on its own: it is the last step of one, and everything before it is a rebuild you have to be able to perform from your own notes. Plan for a restore that is a fresh install plus a data load, and time it once so you know the real number instead of a hoped-for one.

TL;DR by reader profile:

  • Small agency hosting client worlds (a four person studio running six separate client instances): keep the scripts, but wrap them in per instance archives with your hcce.yaml and secrets stored beside each archive, because a client asking for their world back is a per instance restore and not a cluster restore.
  • Solo operator running one instance (a single lecturer hosting seminar rooms on one VPS): the built in backup plus a nightly filesystem snapshot of the node is enough, because your recovery target is the whole box and not a selective extraction.
  • Team using an external managed Postgres (a studio on a hosted database with automated point in time recovery): treat the two halves separately, because the scripts stop covering the database the moment it leaves the bundled pod and your restore must line up two independent timelines.
  • Team with heavy scene libraries (a studio with hundreds of client uploaded GLB files): budget your restore window from file store size and disk throughput, because the database restores quickly and the assets dominate the clock.
  • Team facing a domain or host migration (a move to a new provider or a new client domain): rehearse the restore on a scratch host first, because room URLs, the Reticulum host configuration and certificate issuance all have to be corrected before anyone can join a room.

The central tradeoff: the built in scripts are simple and cover the data that actually matters, but they assume you can rebuild the surrounding installation by hand, so the effort you save at backup time is effort you pay back, under pressure, at restore time.

Table of contents

What does npm run backup actually capture in Hubs Community Edition?

Run npm run backup from your Hubs Community Edition checkout and you get a single timestamped archive, named in the data_backup_1234567890123 pattern, containing two payloads and nothing else.

  • The PostgreSQL database, dumped from the bundled pgsql pod: this is the record of every room, its title, its permissions, its owner account, its scene assignment and the metadata rows that point at uploaded files. Lose it and your GLB files still exist on disk but no longer belong to any room.
  • The Reticulum file store: the uploaded assets themselves, meaning GLB scenes exported from Spoke, custom avatars, room thumbnails, images, audio and video that people dropped into rooms. This is the part that grows without limit and dominates the archive size.
  • Nothing about your cluster or your configuration: hcce.yaml, your Kubernetes state, your certificates and your DNS live outside the archive and are your responsibility to version elsewhere.
  • Nothing about the state of the services at the moment of the dump: the archive is a data snapshot, not a running system image, so a restore always lands on a freshly installed instance.

The matching command is npm run restore-backup data_backup_1234567890123, and calling npm run restore-backup with no argument restores the most recent archive it finds. Where the instance runs shapes the rest of the plan: a self managed VPS, a home server, a NAS and Yundera are all viable homes for the surrounding stack. Yundera is a managed Personal Cloud Server, built on CasaOS, that runs self-hosted apps as Docker containers on a server dedicated to the user. Whichever you pick, the archive stays the same two payloads.

What does the Hubs backup script silently miss?

The word to notice is silently. The script does not warn you about the gaps, it simply finishes, prints a path and leaves you feeling covered. Six things are outside it.

  • Your hcce.yaml: this file carries your domain, your admin email, your subdomain layout and the deployment options that make the instance yours. Restore the data onto a default configuration and you get a working Hubs that is not your Hubs.
  • Secrets and keys: the Reticulum secret key base, the Perms keypair used to sign room join tokens, the cookie signing salt and any OAuth or email credentials. Regenerating the Perms keypair invalidates tokens that clients already hold, so sessions and pending invites break.
  • TLS certificates and the cert-manager state: a fresh install reissues through Let's Encrypt, which means the restore is gated on DNS pointing at the new host and on rate limits, not on your archive.
  • The k3s cluster itself: node configuration, storage class, persistent volume definitions and any manual kubectl edits you made months ago and forgot.
  • External object storage and CDN configuration: if you moved uploads off local disk, the bucket credentials and the bucket contents are not in the archive at all.
  • Email, TURN and third party settings: SMTP relay credentials and Coturn configuration decide whether people can receive invites and connect through restrictive networks.

Test the gap cheaply. Run npm run backup, extract the archive to a scratch directory and list what came out. Everything you expected but cannot find is now an item on a second backup job, ideally a git repository holding hcce.yaml and an encrypted secrets file.

Where does Hubs store rooms, avatars and uploaded GLB scenes on disk?

Two places, and knowing which is which decides what you can recover selectively later.

  • PostgreSQL holds the structure, not the bytes: the Reticulum schema carries tables such as hubs for rooms, accounts for identities, scenes and avatars for published assets, and owned_files for every upload. A row in owned_files records the owner, the content type and the identifier, while the actual file sits elsewhere.
  • The Reticulum storage volume holds the bytes: GLB scenes exported from Spoke, avatar GLBs, room thumbnails, images, PDFs, audio and video are written into a fanned out directory tree, each upload stored as a data file paired with a small metadata file. The names are identifiers, not human readable titles, which is why a file only means something when the database row that points at it survives too.
  • Derived media is cached, not authored: image resizing and video transcoding produce output that can be regenerated, so it is worth excluding from a tight recovery target and rebuilding after the fact.
  • The volume is a Kubernetes persistent volume claim: run kubectl get pvc in your Hubs namespace to see the claim and its size, and kubectl exec into the Reticulum pod to measure the tree with du -sh before you plan any window.

Where that volume physically lives depends on the host you chose, whether a self managed VPS, a home server, a NAS or Yundera. The split matters because the database is small and fast to move, while the storage volume is the part that grows every time a client uploads a scene.

How long does a full Hubs restore actually take?

Nobody can hand you a single number, because the archive is dominated by whatever your clients uploaded. What you can do is decompose the clock into five stages, measure each once on your own hardware, and turn the total into a promise you can actually keep.

Restore stage What dominates the clock How to shorten it
Provision the host and install k3s Provider provisioning time plus package downloads Keep a prebuilt image or a scripted node setup instead of typing it fresh
Deploy Hubs from hcce.yaml Container image pulls for Reticulum, Dialog, the client and Postgres Pre pull images on the standby host, or host them in a local registry
Issue TLS and point DNS DNS propagation plus Let's Encrypt issuance, both outside your control Lower the record TTL before a planned migration, not during an incident
Load the database dump Row count in hubs, accounts and owned_files, usually the smallest stage Nothing needed, this is rarely the bottleneck
Copy back the Reticulum file store Archive size divided by disk and network throughput, the largest stage by far Restore assets from storage that is already close to the new host

Measure it with time npm run restore-backup data_backup_1234567890123 on a scratch instance, then add the stages the script does not cover. Two rules follow. First, the file store sets your recovery time objective, so track its growth with du -sh monthly. Second, quote clients a window built from a real rehearsal, not from the script's runtime alone.

How do you run npm run restore-backup without breaking existing room URLs?

A Hubs room URL is built from the room identifier stored in the hubs table, so the URLs survive a restore for free as long as the database rows come back unchanged and the domain in front of them stays the same. What breaks them is everything you do around the restore.

  • Restore onto a fresh install, never onto a used one: bring up Hubs from hcce.yaml, then restore before anyone creates a room or publishes a scene. New activity writes rows that the incoming dump will collide with or overwrite, and the reconciliation is manual.
  • Keep the domain identical at restore time: put the same value in hcce.yaml that the archive was taken under, restore, verify, and only then perform a domain change as a separate, deliberate step.
  • Quiesce the services first: scale the Reticulum deployment to zero with kubectl scale, run the restore, then scale back. Restoring underneath a running Reticulum means live processes hold state that no longer matches the database.
  • Name the archive explicitly: run npm run restore-backup data_backup_1234567890123 rather than the bare npm run restore-backup, because the no argument form picks the most recent archive it finds and that is rarely what you want during an incident.
  • Verify before announcing: open one known room URL, one published scene and one custom avatar. Three checks catch the common failure, which is a database restored without its matching files.

The host underneath, a self managed VPS, a home server, a NAS or Yundera, does not change this sequence, only how quickly you can stand up the replacement instance.

Should you back up the Hubs Postgres database separately from the Reticulum file store?

For a single instance you can run npm run backup on a schedule and stop thinking about it. For anything larger, split the two, because they have opposite shapes: the database is small and changes every time someone creates a room, while the file store is large and mostly append only.

Dimension One combined archive Database and files backed up separately
Sensible cadence Whatever the slowest half tolerates, usually nightly Database hourly, files daily or on change
Storage growth Every run copies the full file store again Files deduplicate through incremental tooling, dumps stay small
Tooling The bundled scripts only pg_dump plus restic, borg or rsync against the storage volume
Recovery granularity All or nothing, back to the archive time Roll the database to one point, the files to another
Operational cost One command, one cron entry, one thing to forget Two jobs, two retention policies, two things to monitor

The practical rule is the loss window. If your clients publish scenes weekly but create rooms daily, an archive that repeats 40 GB of unchanged GLB files every night buys you nothing except a bigger bill. Dump the database hourly, back the storage volume up incrementally, and keep one full combined archive weekly as the known good fallback that needs no tooling to read.

Split backups carry one obligation: always restore the file store first and the database second, so no row can reference an asset that is not on disk yet. Write that ordering into the runbook, because it is the step people improvise wrongly under pressure.

What changes when you run Hubs against an external database instead of the bundled pgsql pod?

One thing changes and it is the important one: the backup scripts stop covering half your data. With an external database configured, npm run backup and npm run restore-backup handle the Reticulum files only. Nobody prints a warning. You simply get a smaller archive.

  • You now own the database backup outright: either your provider's automated snapshots and point in time recovery, or your own scheduled pg_dump against the Hubs database. Whichever you pick, it belongs in a runbook next to the file archive, not in your head.
  • You have two timelines to reconcile: a database recovered to 02:00 and a file archive taken at 03:00 leaves an hour of owned_files rows pointing at assets the file store does not have. Restore the files first, then roll the database forward to a point at or before the file archive, never after.
  • Credentials become a recovery dependency: the connection string, user and password live in your hcce.yaml and secrets, so losing those means the data survives while access to it does not.
  • Migrations still run on boot: Reticulum applies its schema migrations when it starts, so a database restored from an older instance can be upgraded by a newer deployment without you asking. Restore onto the same version you backed up from and upgrade deliberately afterwards.
  • The bundled pod stays simpler for small instances: one archive, one timeline, one restore command, at the cost of the operational features a managed database gives you.

Choose the split deliberately, then write down which system owns which half.

Which Hubs secrets, certificates and configuration files must be saved outside the backup archive?

Sort every item into two buckets before you build the vault: things that can be regenerated on the new host, and things that change behaviour for users if they are regenerated. Only the second bucket is a real backup obligation.

Item Preserve or regenerate Where it belongs
hcce.yaml Preserve, it defines your domain, subdomains and admin email A private git repository, committed on every change
Perms keypair and Reticulum secret key base Preserve, regenerating invalidates issued room tokens and signed cookies Encrypted secrets file, sops with age or your password manager
External database credentials Preserve, the data is useless without access to it Same encrypted store as the keys, never in the same archive as the dump
SMTP, OAuth and Coturn credentials Preserve, they come from third parties and cannot be recreated locally Encrypted store, with the provider account noted beside them
TLS certificates Regenerate, cert-manager reissues after DNS points at the new host Nothing to keep, but record the issuer and DNS provider steps
Derived Kubernetes secrets Regenerate from the two entries above during deployment Nothing to keep, they are outputs

Capture the live state once with kubectl get secret -n hubs -o yaml so you can see exactly what your instance holds rather than trusting this list. Encrypt that export immediately, because it contains plaintext values.

One discipline makes the whole thing work: the encrypted secrets store must be recoverable without the Hubs instance. If your only copy of the vault key lives on the server you are restoring, you do not have a backup, you have a circular dependency.

How much disk space does a year of Hubs backups need?

Work it out from two measurements rather than a guess. Get the current storage volume size with du -sh inside the Reticulum pod, then take two archives a month apart and subtract to get your monthly growth. Everything else is arithmetic on those numbers.

  • Full copies multiply, so count them first: a grandfather, father, son policy of 7 daily, 4 weekly and 12 monthly archives is 23 retained copies. With the bundled script, each one contains the whole file store, so the year costs roughly 23 times your current storage size plus the accumulated growth. This is the number that surprises agencies.
  • Incremental tooling changes the multiplier, not the base: because GLB scenes and avatars are written once and rarely modified, deduplicating backups with restic or borg store close to one copy plus the new uploads, so 23 restore points cost far less than 23 full archives.
  • Database dumps are rounding error: pg_dump output for room, account and owned_files rows compresses well and stays small next to the assets, which is why hourly database backups are affordable while hourly file archives are not.
  • Derived media inflates the total without adding value: transcoded video and resized images can be regenerated, so excluding them from the archive shrinks every copy for the whole year.
  • Off site copies double whatever you decided: the 3-2-1 rule means one primary, one local secondary and one remote, so budget the total twice, not once.

Set retention against a real number. A client that publishes a scene weekly needs different depth than one that uploaded once and never returned.

Snapshots, scripts or object storage: which backup strategy fits a Hubs instance?

Three approaches exist and they solve different failures. Pick by asking what you expect to lose: the whole machine, the data inside it, or the provider account holding both.

  • Provider or filesystem snapshots of the node: these capture the entire k3s host, so recovery is a rollback rather than a rebuild, and your hcce.yaml, secrets and certificates come back with it. They fail you in two ways: they usually live in the same provider account as the server, and they restore everything or nothing, which is useless when one client wants one room recovered.
  • The bundled scripts alone: npm run backup is portable, understandable and independent of any provider feature, which makes it the archive you can still read in three years. It covers the two data payloads and nothing around them, so it needs a companion job for configuration and keys.
  • Incremental backups to object storage: restic or borg pushing the Reticulum storage volume and pg_dump output to an S3 compatible target such as MinIO or Backblaze B2 gives deduplication, encryption at rest and an off site copy in one tool. The cost is a second system to monitor, plus a restore that now depends on network throughput.
  • The hybrid most agencies land on: nightly snapshots for fast whole host rollback, weekly bundled archives as the provider independent fallback, and continuous incremental pushes off site for the 3-2-1 requirement.

The choice is not about which tool is best. It is about which of the three failures you consider most likely, and whether your recovery needs to be selective.

How do you test a Hubs restore before you actually need it?

Build a scratch instance on a throwaway subdomain, restore last night's archive into it, and check five things by hand. Repeat quarterly and after every version upgrade, because an untested archive is a hypothesis.

  • Use a separate domain, never the live one: set a value such as hubs-drtest.example.com in the scratch hcce.yaml. Restoring production data onto production DNS during a drill is how a test becomes an outage.
  • Join a room and check presence works: open a restored room URL in two browsers and confirm audio and video connect. This exercises Dialog and Coturn, which a database check alone never touches.
  • Load a published GLB scene, not just the room list: rooms that open with a missing environment are the signature of a database restored without its matching file store, and the room index looks perfectly healthy in that state.
  • Load a custom avatar and a room thumbnail: avatars and thumbnails are separate owned_files rows, so they fail independently of scenes and prove the file tree copied completely rather than partially.
  • Time each stage and write the number down: run the drill with a stopwatch, record the total, and give clients that figure rather than an estimate. The number moves as the file store grows.

Finish by destroying the scratch instance and any DNS records it created, so nobody stumbles into stale rooms months later. Keep a one page log of every drill: the date, the archive name, the elapsed time and what broke. Three entries in that log tell you more about your recovery position than any amount of configuration review.

What breaks when you restore Hubs onto a different domain or a different host?

A different host alone is uneventful: the archive is portable and the deployment is rebuilt from hcce.yaml, so a move between a self managed VPS, a home server, a NAS or Yundera changes little beyond throughput. A different domain is the one that hurts, because the old hostname is written into places the restore script does not touch.

  • Stored absolute URLs: rows created under the old domain can carry fully qualified asset and scene URLs. After a domain change, query the database with psql for the old hostname before you assume the restore was clean, and update what you find.
  • Links your clients already shared: room URLs printed in emails, calendar invites and slide decks point at the old host. Keep the old domain resolving with a redirect for at least one full booking cycle rather than a fixed number of days.
  • OAuth and email: third party callback URLs and the sending domain in your SMTP configuration are registered against the old hostname and reject the new one until you update both providers.
  • TURN and media: Dialog and Coturn advertise addresses derived from your deployment configuration, so audio and video can fail while the room list looks perfectly healthy.
  • Certificate issuance order: DNS must point at the new host before cert-manager can complete a challenge, so the sequence is DNS first, deploy second, restore third.

Do the domain change as its own step, after a same domain restore has been verified. Two changes at once means any failure gives you two suspects instead of one.

Top comments (0)