How SigNoz Helped Me Build a Remote GPU Sharing Platform — And Actually See What's Happening

#backend #cloud #monitoring #tools

The Problem: GPUs Are Everywhere, But Never Where You N

I have a MacBook. It's great for writing code, browsing thl commands. But the moment I need to do anything with AI —run a model, train something, or even multiply large matrices — I'm stuck. No CUDA. No GPU. Just a fan spinning faster out of frustration.

Meanwhile, my friend has a Windows laptop with an NVIDIA GTX 1650 sitting idle 90% of the time. He plays games occasionally. The rest of the
time, that GPU is doing nothing.

This is the reality for most developers. GPUs are expensivnd often idle. You either have one and don't fully use it,or you don't have one and desperately need it.

So I built GPU Pod — a lightweight system that lets anyone with a GPU offer it over the local network, and anyone who needs compute use it. Just Python, PyTorch, and HTTP.

But here's the thing: when you have three separate program a client GUI) all talking to each other, things go wrong.Jobs fail silently. Workers disconnect. Results get lost. And you have no idea why.

That's where SigNoz came in.

The Architecture: Three Moving Parts

Before I explain how SigNoz helped, here's what I built:

Mac (Client) ──submit job──▶ FastAPI Server ──assign──▶ Windows Worker
◀──poll result──────────────◀───────return result────

Three components:

GPU Provider (Windows) — runs a Python worker that executes PyTorch matrix multiplications on CUDA
GPU User (Mac) — a desktop GUI where you pick a matrix size and submit compute jobs
Coordinator Server (Mac) — FastAPI + SQLite that qu and routes results

It works. But when something breaks, all you see is:

[WARN] Heartbeat failed

That's it. No context. No trail. No way to know why.

Before SigNoz: Debugging in the Dark

Here's what debugging looked like before observability:

A job gets submitted
It stays "pending" forever
I check the server terminal — nothing useful
I SSH into the Windows machine and check its terminal — nothing useful
I restart everything and hope it works this time

Sound familiar?

The problem is that three separate programs running on twote logs in three different places. There's no single source of truth. When something fails, you have to manually correlate timestamps across terminals that you can't even see at the same time.

I needed:

A unified view of all logs
A way to trace a job from submission to completion
Metrics to know if the system was healthy
A way to correlate errors with what happened before

How SigNoz Solved It

1. Distributed Tracing — Follow a Job End to End

Every operation in GPU Pod now creates an OpenTelemetry span with rich attributes:

Service: gpu-pod-server
Span: POST /submit-job
├── job.id = "a1b2c3d4"
├── job.type = "compute"
├── matrix.size = 16384

Service: gpu-pod-worker
Span: worker.execute_job
├── job.id = "a1b2c3d4" ├── compute.tflops = 2.67
├── worker.gpu = "NVIDIA GeForce GTX 1650"

When a job fails now, I can open the trace in SigNoz, see exactly which span failed, check the attributes, and understand why — all without digging through terminal output.
The Service Map in SigNoz visualises the entire flow: server → gpu-pod-worker`. At a glance I can see if allservices are healthy and communicating.

2. Centralised Logs — No More Terminal Hopping

Before SigNoz, logs were scattered across: - The server's terminal window

The worker's terminal window (on a different machine) - The GUI's text box After SigNoz, everything — server logs, worker status reams into one place: 2026-07-26 22:00:28 [gpu-pod-client] job submitted from 2026-07-26 22:00:21 [gpu-pod-client] connected to server 2026-07-26 21:58:54 [gpu-pod-worker] [OK] Registered | I 2026-07-26 22:00:31 [gpu-pod-worker] [JOB] 55482209 | Type: compute 2026-07-26 22:00:38 [gpu-pod-worker] [DONE] 2.67 TFLOPS 2026-07-26 22:00:38 [gpu-pod-server] [RESULT] Job 55482209 → COMPLETED I can filter by service, search by job ID, or just watch the live stream. No more SSHing into Windows to check logs. ### 3. Custom Metrics — Know Your System Health I added OTel counters and histograms for everything that matters: | Metric | What it tells me | |--------|-----------------| | jobs.total | How many jobs were submitted | | jobs.completed / jobs.failed | Success rate at a gla | workers.active | How many GPUs are available right now | | jobs.queue_depth | Is the system backed up? | | worker.job.duration | How long does a 16384×16384 multiply take? | These metrics update in real time in SigNoz. I can set up alerts for when jobs.failed spikes or workers.active drops to zero. ### 4. Error Correlation — Debug in Minutes, Not Hours Here's my favourite workflow now: A user reports "my job failed."
I open SigNoz → Services → gpu-pod-server → Traces 2. Filter by job.id = "a1b2c3d4"
I see the full trace: submit → assign → execute → result 4. If it failed, the error span has the exception message
The associated log entry shows the exact error timestamp 6. The metric chart shows if this was an isolated failure Before SigNoz: 30 minutes of SSHing, scrolling, guessing. After SigNoz: 30 seconds of clicking. --- ## The Technical Implementation Adding SigNoz was surprisingly simple. I created one shared module (gpu_pod_otel.py) that all three components import: `python from opentelemetry import trace, metrics from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.metrics import MeterProvider from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from opentelemetry.exporter.otlp.proto.grpc.metric_exporter from opentelemetry.sdk.resources import Resource, SERVICE_NAME init_otel("gpu-pod-server", env="production") Then I instrument each component: - Server: FastAPIInstrumentor.instrument_app(app) — automa
Worker: Manual spans around register(), heartbeat(), and execute_job() - Client: Spans around connect() and submit_job() All print() calls were replaced with logger.info() which r and SigNoz. The entire OTel setup is about 120 lines of code. The valu --- What I Learned
Observability isn't a feature — it's a necessity. GPU Pod worked fine when everything was perfect. The moment something went wrong, I was blind. SigNoz gave me sight.
Tracing matters even for simple architectures. I only have three services, but they communicate asynchronously via HTTP. Tracing across those boundaries caught issues I would never have found ot
Metrics tell you what's happening; logs tell you why. Having both in one tool means you don't switch contexts. You see the metric spike, click into the logs, find the error, trace it to the span.
OpenTelemetry is surprisingly easy to add. I was worried about complexity, but the OTel Python SDK is well-documented. Auto-instrumentation for FastAPI worked out of the box. --- Try It Yourself GPU Pod is open source. You can run it with just uv sync and uv run python main.py. SigNoz runs in Docker with a single command (castingcast — yes, that's the actual command). The full setup takes about 10 minutes:
Start SigNoz: castingcast 2. Start the server
Start the worker on your GPU machine 4. Open the GUI and submit jobs Everything you do — every trace, log, and metric — shows ulhost:8080. --- The Bottom Line Building GPU Pod solved a real problem: making idle GPUs accessible to people who need them. But without SigNoz, it was a black box. I'd submita job, wait, and hope. SigNoz turned that black box into a glass house. Now I canlog, every metric. When something breaks — and it will — Iknow exactly where to look. If you're building a distributed system, no matter how small, instrument it with OpenTelemetry and use SigNoz. You'll thank yourself the first time something goes wrong.