DEV Community

chovy
chovy

Posted on

Our P2P inference network had 4 of 48 nodes online. Here is why, and the fix

#networking #opensource #softwaredevelopment

Infernet is a peer-to-peer GPU and CPU inference network: you run one command, your machine serves open models, and clients pay per job. This morning our status page said 4 nodes online out of 48 registered, and all 4 were ours.

The other 40-odd were real people: Mac minis, Windows desktops, cloud VMs, one H20 box. They installed, served jobs (one served 243 of them over 52 days), and then went quiet and never came back. So we went looking for why.

What was killing nodes

  • Self-update ended the process it was updating. Under systemd, the daemon re-launched itself as a detached child and exited 0. Exit 0 ends the unit, and systemd's control-group kill takes the detached child with it. The node was gone until someone noticed.
  • Nothing restarted a crashed node. infernet start detached one process with no supervisor. A crash, an OOM kill or a reboot was permanent.
  • Installs from git thought they were always out of date. The repo's package.json said 0.1.45 while npm was at 0.1.62, so those nodes reinstalled every 5 minutes, forever.
  • Nobody was told. No email, no status, nothing.

What changed (v0.1.62 to v0.1.64)

  • infernet setup now installs a systemd user unit on Linux (with linger, so it survives logout and reboot) or a launchd agent on macOS.
  • Without a service manager, infernet start runs a small built-in supervisor that restarts the daemon after a crash, with backoff.
  • One exit-code contract everywhere: 0 is a deliberate stop, 75 means "restart me into the new version", anything else is a crash.
  • Self-update checks that something newer actually landed before it restarts, and pauses for 6 hours if not.
  • If your node is linked to an account and goes silent for 24 hours, you get one email with the two commands that bring it back.

Aggregators now do something

The aggregator role used to just send heartbeats. Now an aggregator dials every advertised provider each minute, checks it answers as the node it claims to be, and reports reachability and round-trip time. The status page shows, per provider, how many aggregators reached it in the last hour. We run three (Buffalo, Buffalo, Vienna) and anyone can run one on a small box:

infernet init --role aggregator --name my-aggregator
infernet register
infernet setup --yes
Enter fullscreen mode Exit fullscreen mode

On its first round it already caught a node advertising a port that belongs to something else.

See who is on the network

Every provider on infernetprotocol.com/status now links to a details page: coarse location (city, country, network; never the IP), hardware, live load, measured tokens per second, models served, job counts and success rate, recent jobs (never prompt content), and aggregator verification. Operators can hide a node from Settings.

Run a node 

curl -fsSL https://infernetprotocol.com/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

It now stays up on its own. If you ran one before and it died, re-running that line brings it back on the new code.

Code: github.com/InfernetProtocol/infernet-protocol (PRs #51, #53, #54).

Top comments (0)