I wanted a Telegram bot. Something simple. A weather bot where you send a city name, it calls OpenWeatherMap, sends back the temperature. Took about 80 lines of Python with aiogram. I thought it was done.
Then I added SigNoz to see what was actually happening inside. And I found stuff I never would have caught without it. A UTF-8 encoding bug that crashed one command silently. A 5-second latency spike that only showed up in the flamegraph. Six spans per command that I didn't even know existed.
What the Bot Does
The bot handles 7 commands — /weather, /forecast, /error, /slow, /stats, /health, /start. When you send /weather London, it checks Redis cache first. If the city isn't cached, it calls OpenWeatherMap, caches the result, and sends back the temperature.
The point isn't the bot. The point is what SigNoz showed me about the bot.
Wiring Up OpenTelemetry
So, I started with metrics.py. Three providers — tracer, meter, logger — all in one file:
tracer_provider = TracerProvider(resource=resource)
span_exporter = OTLPSpanExporter(endpoint=OTEL_EXPORTER_ENDPOINT, insecure=True)
tracer_provider.add_span_processor(BatchSpanProcessor(span_exporter))
metric_exporter = OTLPMetricExporter(endpoint=OTEL_EXPORTER_ENDPOINT, insecure=True)
reader = PeriodicExportingMetricReader(metric_exporter, export_interval_millis=10000)
meter_provider = MeterProvider(resource=resource, metric_readers=[reader])
log_exporter = OTLPLogExporter(endpoint=OTEL_EXPORTER_ENDPOINT, insecure=True)
logger_provider = LoggerProvider(resource=resource)
logger_provider.add_log_record_processor(BatchLogRecordProcessor(log_exporter))
set_logger_provider(logger_provider)
Then in main.py, I registered them globally and added auto-instrumentation:
trace.set_tracer_provider(tracer_provider)
otel_metrics.set_meter_provider(meter_provider)
LoggingInstrumentor().instrument()
HTTPXClientInstrumentor().instrument()
RedisInstrumentor().instrument()
Every httpx.get() to OpenWeatherMap and every Redis command became a child span automatically. I didn't have to touch the bot code for HTTP and Redis — the instrumentors patched them.
But I noticed something weird. I had the LoggingInstrumentor but logs weren't showing up in SigNoz. I spent 2 hours debugging this. Turns out I forgot to set up the LoggerProvider and call set_logger_provider(). The instrumentor only patches the logging module — without the provider, nothing gets exported. I had to add logging.getLogger().addHandler(LoggingHandler()) too. Three things needed, I only had one.
What the Trace Showed
So, I sent /weather London to the bot. Opened SigNoz. Clicked on the signoz-telegram-bot service. This is what the flamegraph showed:
bot.command.weather (1416ms)
├── GET — Redis cache lookup (5ms)
├── GET — OpenWeatherMap API (362ms)
├── SET — Cache the result (1ms)
├── INFO — Redis stats (5ms)
└── DBSIZE — Redis info (2ms)
Six spans. The API call took 362ms. Redis operations were under 5ms each. But the total was 1416ms. Where did the other 1000ms go? That's the BatchSpanProcessor flushing and Telegram API latency. Without the trace I'd have no idea.
I didn't instrument any of this manually — the auto-instrumentors did it. Before SigNoz, I just saw "send message, get reply." Now I see the whole pipeline.
The Bug I Found
So, I tested /forecast Mumbai. The API returned HTTP 200. But the bot sent back "Error fetching forecast." I had no idea why.
I opened SigNoz and looked at the traces. There was a trace for bot.command.forecast with hasError=true. I clicked on it and saw the error span. I checked the logs — they were correlated with the trace via trace_id:
Forecast error: 'utf-8' codec can't encode characters
in position 32-33: surrogates not allowed
The weather emoji characters had surrogate pairs that Python's UTF-8 encoder couldn't handle. When the bot tried to format the forecast response with emojis like ☀️ and 🌧️, Python threw an encoding error. The API returned 200 fine, but the response formatting broke.
Without SigNoz I'd be guessing. With the trace plus the error span, the problem was right there. The span recorded the exception, the logs showed the exact error message, and the trace_id linked them together.
The fix was two lines — replace emoji characters with text labels:
WEATHER_EMOJIS = {
"Clear": "Sunny", "Clouds": "Cloudy", "Rain": "Rainy",
"Drizzle": "Drizzle", "Thunderstorm": "Storm",
"Snow": "Snowy", "Mist": "Misty", "Fog": "Foggy",
}
But the point is I found it because SigNoz showed me the exact span with the exception recorded.
The /slow Command — Seeing Latency
So, the /slow command does asyncio.sleep(5) to simulate a real-world slow dependency. In SigNoz, the flamegraph shows a span that's 6749ms wide — the 5-second sleep plus overhead. I could see exactly where the time went.
This is exactly the kind of latency a real production bug would cause. A database query that takes 5 seconds instead of 50ms. A third-party API that hangs. SigNoz makes it instantly visible.
Five Custom Metrics and a Dashboard
The bot emits 5 custom metrics via OTLP — bot.requests.total, bot.errors.total, weather.api.latency.ms, cache.hits.total, cache.misses.total — plus a subscriptions.active up-down counter. These show up in SigNoz alongside system metrics from the OTel Collector.
I built a dashboard with 6 panels to visualize them. The thing I liked is that custom metrics show up in the same place as CPU usage, memory, and network. One dashboard for everything.
Logs Correlated with Traces
Python logs bridge to SigNoz via LoggingInstrumentor plus a LoggerProvider. The key line I kept forgetting:
logging.getLogger().addHandler(LoggingHandler())
Every logger.info(...), logger.warning(...), and logger.error(...) call goes to ClickHouse. In SigNoz, logs are correlated with traces via trace_id and span_id — click a trace, see the logs that happened during it.
I verified this by checking ClickHouse directly. Ran SELECT count() FROM signoz_logs.logs_v2. It returned 13 log records from the bot. All of them had trace_id fields linking them to the parent traces.
Circuit Breaker and Retry Logic
The resilient_request wrapper does both retry and circuit breaking. The CircuitBreaker class tracks failures — after 5 consecutive failures it opens, after 30 seconds it goes to half-open. The @retry decorator from tenacity handles up to 3 attempts with exponential backoff.
In SigNoz, the retry attempts show up as child spans. The circuit state changes show up in span attributes. I set user.id, weather.city, bot.command, and circuit_breaker.state on spans. Then in SigNoz, I can filter by any of those. "Show me all traces where weather.city is Mumbai" — done.
Deploying with Foundry
SigNoz runs via Foundry — foundryctl cast -f casting.yaml. This gives you 6 containers: ZooKeeper, ClickHouse, PostgreSQL, SigNoz UI, OTel Collector, and a schema migrator.
One thing that bit me — the OTel Collector uses OpAMP for config, but the default config crashes trying to connect to ClickHouse before OpAMP pushes the real config. The error was Connection refused on [::1]:9000 — IPv6 localhost instead of IPv4. I fixed this by mounting otel-collector-config.yaml directly and removing the OpAMP config entirely.
Alert Rules
I set up 4 alert rules in SigNoz — high error rate, slow commands, API timeout, and zero traffic. Each one uses the custom metrics the bot emits.
What I'd Do Differently
Add log correlation from the start. I only got logs flowing on the third attempt because I kept forgetting to set up the LoggerProvider. This cost me 2 hours.
Use http.url from the start instead of peer.service for filtering HTTP calls. I spent time trying peer.service contains api.openweathermap.org and it didn't work. http.url is the right attribute for httpx requests.
Traces use signoz_traces.signoz_index_v3 not v2. Logs use signoz_logs.logs_v2 not distributed_logs. These table names matter if you're querying ClickHouse directly.
The Whole Point
This bot is about 300 lines of Python. The SigNoz integration added about 50 lines across metrics.py, main.py, and span attributes in handlers.py. But those 50 lines turned a black box into something I can actually debug.
When the bot is slow, I don't guess — I open the flamegraph. When the API fails, I don't check logs separately — I see the error span with the exception recorded and the logs correlated to the same trace. That's the difference between "it works" and "I know why it works."
Built for the Agents of SigNoz Hackathon 2026 — Track 03: Build Your Own. The bot, Foundry config, and all traces are in the repo.
Top comments (0)