From Tor Bootstrap to a Working Onion Health Probe: Debugging MyZubster End to End

Building an Onion Service is one thing. Proving that it actually works end to end is another.

While integrating the distributed Onion layer for MyZubster, we hit a sequence of issues that looked unrelated at first: missing candidate files, invalid .onion placeholders, Tor timeouts, Docker-to-host connectivity failures, and finally a health probe that was technically reaching the application but still reporting failures.

The useful part was not any single fix. It was the debugging path.

The architecture

The setup has three main components:

• a MyZubster application running on the host on port 5003;
• a Tor Onion Service running in Docker and forwarding Onion traffic to the host;
• a separate probe container running its own Tor client and periodically checking known .onion nodes.

Conceptually:

probe-agent
    |
    | SOCKS5
    v
Tor client
    |
    | Tor network
    v
MyZubster Onion Service
    |
    | HiddenServicePort
    v
Docker host :5003
    |
    v
Express application

The goal was simple: produce periodic JSON health observations for deployed Onion nodes.

First problem: the candidates path was a directory

The probe expected:

/config/candidates.txt

to be a normal file.

Instead, the host-side path had accidentally become a directory.

The probe logic was explicitly checking:

if [ -f "$CANDIDATES_FILE" ]; then
    while IFS= read -r node || [ -n "$node" ]; do
        ...
    done < "$CANDIDATES_FILE"
fi

So the probe container itself could start normally, Tor could bootstrap normally, and yet no candidates would ever be processed.

The fix was straightforward: remove the directory and recreate candidates.txt from the example file.

Tor was actually healthy

After restarting the probe, Tor completed its bootstrap sequence:

Bootstrapped 100% (done): Done
Starting MyZubster Onion probe agent...

That was an important checkpoint.

When debugging Onion connectivity, it is easy to blame Tor too early. In our case the Tor client was doing exactly what it was supposed to do.

Then we made a classic placeholder mistake

During testing, a placeholder was temporarily written into the candidates file:

HOSTNAME_REALE.onion

Tor correctly rejected it:

Invalid hostname [scrubbed]; rejecting

A v3 Onion hostname contains 56 base32 characters before .onion.

So we added a simple validation step before accepting candidates:

if [[ "$ONION" =~ ^[a-z2-7]{56}\.onion$ ]]; then
    echo "valid"
else
    echo "invalid"
fi

Small validation checks like this prevent a surprising amount of wasted debugging time.

Finding the real Onion address

The actual Onion hostname was generated by Tor inside the Onion Service container:

/var/lib/tor/myzubster/hostname

Once we inspected that file, we had a real deployed v3 Onion hostname and could feed it to the probe.

At that point the system became much more interesting.

The probe started producing observations, but they looked like this:

{
  "result": "onion_connect",
  "latency_ms": 15013,
  "error_class": "onion_connect"
}

The approximately 15-second latency matched the configured probe timeout almost exactly.

That told us the candidate was valid, but the Onion connection was not completing.

The hidden service configuration looked correct

The generated Tor configuration contained:

HiddenServiceDir /var/lib/tor/myzubster
HiddenServicePort 80 172.17.0.1:5003

The application itself was also healthy:

HTTP/1.1 200 OK
X-Powered-By: Express

and listening on:

*:5003

So the application was alive and Tor was configured to forward to the correct host port.

But from inside the Onion container:

TCP_FAIL

The container could resolve:

host.docker.internal -> 172.17.0.1

but it could not connect to port 5003.

That narrowed the issue down dramatically.

The real blocker: host firewall policy

The server was running UFW with:

Default: deny (incoming), allow (outgoing), deny (routed)

The Onion container lived on its own Docker network:

network: onion_onion_net
subnet: 172.20.0.0/16
container: 172.20.0.2

The application was reachable locally on the host, but traffic from that Docker subnet to host port 5003 was being dropped.

The minimal rule was:

ufw allow from 172.20.0.0/16 to any port 5003 proto tcp

After that:

TCP_OK

That was the turning point.

The next probe observations no longer timed out at the Onion layer.

Now the probe reached the application — and still failed

The new observations changed from:

onion_connect

to:

application_error

with much lower latency.

That meant the network path was working.

The probe was now successfully reaching the application, but the application was returning a status the probe considered unsuccessful.

A direct request through the probe's Tor SOCKS proxy showed:

HTTP/1.1 200 OK

for /.

So why did the automated probe still report an application error?

Because the script was not requesting /.

It defaulted to:

PROBE_PATH="${PROBE_PATH:-/health}"

and the application returned:

Cannot GET /health

The distributed probe was working perfectly. It was simply checking an endpoint that did not exist.

Making the probe match the deployed application

For the current deployment, we configured:

PROBE_PATH=/

Then rebuilt the probe container.

The next observation was:

{
  "node_id": "<onion-hostname>",
  "observer_id": "observer-local",
  "observed_at": "2026-08-22T01:56:29Z",
  "result": "success",
  "latency_ms": 3541,
  "error_class": null
}

That was the first complete end-to-end success.

One more bug: the observation schema

During the process we also found a semantic bug in the probe output.

The JSON schema defined:

"result": {
  "type": "string",
  "enum": ["success", "failure"]
}

but the script was emitting values such as:

"result": "application_error"

and:

"result": "onion_connect"

Those values belonged in error_class, not result.

The corrected behavior became:

{
  "result": "failure",
  "error_class": "application_error"
}

or:

{
  "result": "failure",
  "error_class": "onion_connect"
}

while successful probes remain:

{
  "result": "success",
  "error_class": null
}

This matters because health data is only useful if producers and consumers agree on the schema.

What the final working path looks like

After the fixes, the complete flow is:

probe-agent
    ↓
local Tor SOCKS client
    ↓
Tor network
    ↓
MyZubster v3 Onion Service
    ↓
HiddenServicePort 80
    ↓
Docker host 172.17.0.1:5003
    ↓
MyZubster Express Gateway
    ↓
HTTP 200
    ↓
schema-compliant health observation

The probe now produces real distributed health observations rather than simply checking whether a container is alive.

Lessons from the debugging process

A few principles turned out to be especially useful.

First, test every boundary separately. We independently tested Tor bootstrap, hostname validity, SOCKS connectivity, Onion resolution, Docker-to-host TCP connectivity, HTTP responses, and finally the observation writer.

Second, latency is diagnostic information. A repeated failure at almost exactly the configured 15-second timeout immediately suggested a connection-layer problem rather than an application response.

Third, Docker networking and host firewalling are separate concerns. A service can listen on 0.0.0.0, Docker DNS can resolve the host correctly, and the connection can still fail because the host firewall rejects traffic from the container subnet.

Fourth, health checks must match the application that actually exists. /health is a convention, not a guarantee.

Finally, validate observability data itself. A monitoring system that emits records violating its own schema is another failure mode waiting to happen.

Current status

The distributed Onion health path is now operational on the deployment we tested.

The implementation was committed in:

996876b
fix: make distributed onion probe operational

and is currently being reviewed as part of the distributed Onion integration work.

There is still one operational detail worth making reproducible: the scoped firewall rule allowing the Onion Docker network to reach the MyZubster backend port. That belongs in deployment documentation or provisioning rather than tribal knowledge.

But the important milestone is complete:

a probe running through its own Tor client can reach the deployed MyZubster Onion Service, receive the application response, and record a valid successful health observation.