Every prompt you send to an LLM costs money—but do you actually know where those tokens are going? I instrumented an existing Go AI service with OpenTelemetry to track every token, then used MCP to dig through the telemetry and uncover what was really happening behind the scenes.
Why monitor your AI Service for Token Usage?
Unmonitored AI can instantly break your budget. A tech firm recently faced a massive $500 million Claude AI bill in just one month because nobody was tracking internal token usage.
Read more :
Company racks up $500M Claude AI bill in one month after putting no usage limits on employee licenses
A company faced a shocking $500 million bill for AI usage in just one month due to lack of spending limits on employee licenses. Explore the implications of unlimited access to AI tools and the need for governance in enterprise AI spending.
economictimes.indiatimes.com
Complex automated workflows and heavy daily prompts will quietly drain your budget long before anyone in finance notices the spike.
Let's Use Datadog to solve this (Oh No Wait...)
Datadog can become expensive for AI apps because every model, user, and prompt adds more data to track. AI also creates lots of text, which costs more to store. It also uses Datadog-specific tools, so switching to another platform later can be hard and take a lot of work.
Here are few reasons why we choose OpenTelemetry :
- Datadog "Custom Metrics Tax": Tracking GenAI apps creates too many unique tags, increasing Datadog custom metric costs ("Datadog Tax").
- Vendor Lock-In: Datadog's LLM features rely on proprietary dd-trace-go instrumentation, making migration to another platform costly and code-intensive.
- Self-Hosted Option: Self-hosting gives you full control over AI data ingestion, preventing expensive log indexing costs from large LLM payloads.
Adding OpenTelemetry to Your Existing Go AI Service
This blog focuses primarily on monitoring the data that's already being generated, rather than on instrumentation itself. If you haven't set up instrumentation yet, check out these resources first—they'll help you get started. (Or, honestly, you can just ask an AI assistant; it's pretty straightforward.)
Follow this : https://signoz.io/blog/golang-monitoring/
Why I Chose SigNoz for Observability
For this guide, we'll use SigNoz for observability. It's open source, supports self-hosting, and is a great choice if you're just getting started. Self-hosting also gives you more control over your data while keeping costs low.
BIG Mistake (ASKING AI FOR HELP WITH SIGNOZ)
SigNoz is a fast-growing open source project, which means the docs, installation steps, User interface, and best practices are constantly evolving. SigNoz's team is always finding better ways to make things easier, so when you're working with SigNoz, always start with the official documentation instead of blindly following what an AI says.
Otherwise, you might end up going down a rabbit hole, spend hours on things that have already changed, and eventually come back to write a blog post telling people "why you should've just read the docs."
...At least that's what happened to my friend. 😉
Installing Signoz on Linux
Follows this : https://signoz.io/docs/install/docker/
Play Around
Verify Metrics & Logs Ingestion
Create Dashboard
AI Token Usage Monitoring
Create Alert
Before you begin, make sure you've created a notification channel. I chose Slack—this time, setting it up was much simpler.
Follow this : https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/
Now, let's create the alert. I used llm_token_total, which measures the total number of tokens generated over time.
For this example, I configured the alert to trigger when llm_token_total > 1000 within a 1-minute window. This is just an example—it's better to monitor your token usage for a few days first, understand your typical usage patterns, and then set an alert threshold that aligns with your budget.
Result: Slack Alert After a Token Usage Spike
When the alert condition is met—for example, when your token budget is exceeded—a custom message is sent to Slack. This makes it easier to track usage and plan your budget more effectively.
Use MCP to explore further
I used the VS Code + Cline + free DeepSeek model setup because, well, it was free. But feel free to use whatever stack you prefer—pun absolutely intended. With the latest updates, connecting to an MCP server has become surprisingly straightforward, regardless of the tools you choose.
Bonus Challenge
If you actually made till here, then go further. Add some custom metrics, then write a blog on how you used OpenTelemetry and SigNoz to observe your AI stack. Be sure to add this blog as a reference to that!
Share what worked, what didn't, and any observability tricks you picked up along the way. Let's learn from each other.
Conclusion
Setting all of this up took barely 3–4 hours, which honestly isn't much considering the visibility and insights you gain in return. If you've made it this far, you've already done the hard part—so you might as well add OpenTelemetry to your project and start your observability journey.
And finally, a huge shoutout to SigNoz for continuing to invest in open source. If this guide helped you or you appreciate what they're building, consider giving their GitHub repository a star. It's a small gesture that goes a long way in supporting projects like these.
SigNoz
/
signoz
SigNoz is an open-source, OpenTelemetry-native observability platform for your team and their AI agents. Get logs, metrics, and traces in one tool with features like APM, distributed tracing, log management, infra monitoring, etc. Combined with SigNoz MCP and a native AI teammate (in SigNoz Cloud) it helps you build more resilient apps.
SigNoz is an open-source observability platform built on OpenTelemetry. We’re building an enterprise-grade alternative to fragmented monitoring stacks, with logs, metrics, traces, alerts, and dashboards in one place.
Choose how to run SigNoz
SigNoz Cloud (Recommended)
Fully managed SigNoz with a 30-day free trial, no credit card required, usage-based pricing that starts at $49, and regional data hosting.
Enterprise
Enterprise Cloud, BYOC, or Enterprise Self-Hosted with compliance, support, custom retention, RBAC, ingestion controls, data residency, and region selection.
Community
Free open-source SigNoz that runs in your own infrastructure. Deploy with Docker, Kubernetes, or Linux and keep full control of your data plane.
What can you monitor?
SigNoz helps teams debug production issues faster by connecting logs, metrics, traces, alerts, dashboards, exceptions, and agent-native workflows in one place.
APM Overview
Monitor service latency, error rate, throughput, Apdex…
Collaboration
This blog was made with collaboration
Trushna from Trillion Defense &
Me (Bhavesh) from Trillion Defense
References:
https://signoz.io/docs/install/docker/
https://github.com/SigNoz/foundry/blob/main/docs/concepts/mcp-server.md
https://signoz.io/blog/golang-monitoring/
Let's Connect
Bhavesh
LinkedIn: https://www.linkedin.com/in/bhavesh-sp/
Github: https://github.com/BhaveshSP
Trushna
LinkedIn: https://www.linkedin.com/in/trushna-purohit/
Github: https://github.com/TrushnaCodes
Top comments (0)