Erik Dietrich for Sawmills

Posted on Oct 29 • Originally published at sawmills.ai

The Gotchas of Building an In-House Observability Platform Using Prometheus

#observability #pometheus

Editorial note: this post was written by the Sawmills team originally posted on the Sawmills blog .

In the era of cloud-native applications and microservices, observability has become a cornerstone of reliable software systems. Prometheus, an open-source monitoring and alerting toolkit, is often the go-to choice for organizations looking to build their own observability platforms. While Prometheus offers a rich feature set and the allure of customization, building an in-house observability platform is fraught with hidden challenges and costs that are not immediately apparent.

In this blog post, we'll delve into the intricacies of building an in-house observability platform using Prometheus. We'll explore when it makes sense to take on this endeavor, when it might be wiser to opt for alternative solutions, the known and unknown challenges you'll face, the hidden costs involved—including the risks associated with high cardinality metrics—and strategies to mitigate the associated risks.

Introduction to Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit originally developed at SoundCloud. It has become a cornerstone in the cloud-native ecosystem, especially when used in conjunction with Kubernetes. Prometheus excels at collecting time-series data, offering a powerful query language (PromQL), and integrating with various third-party tools.

Key Features of Prometheus

Time-Series Data Storage: Efficiently stores metrics data with labels.
PromQL A flexible query language for slicing and dicing collected time series data.
Service Discovery: Automatically discovers services through various mechanisms like Kubernetes APIs.
Alerting: Supports alert definitions and integrates with Alertmanager for notifications

When to Build In-House

Building an in-house observability platform using Prometheus can be advantageous under certain conditions:

1. Customization Needs

Specific Requirements: If your organization has unique monitoring needs that off-the-shelf solutions can't meet.
Integration Capabilities: Seamless integration with proprietary systems and workflows.

2. Cost Considerations

Avoiding Licensing Fees: Prometheus is free to use, which can be appealing for budget-conscious organizations.
Long-Term Savings: Potential for reduced costs over time compared to recurring fees for SaaS solutions.

3. Data Control and Compliance

Security Concerns: Keeping sensitive metrics data within your own infrastructure.
Regulatory Requirements: Meeting stringent compliance standards that require on-premises data storage.

4. Developing Internal Expertise

Skill Building: Fostering in-house talent and expertise in observability.
Innovation Potential: Encouraging a culture of innovation by solving complex problems internally.

When Not to Build In-House

However, building your own observability platform isn't always the best route:

1. Resource Limitations

Time Investment: Developing and maintaining the platform can divert focus from core business activities.
Talent Shortage: Lack of experienced personnel to manage and operate the platform effectively.

2. Total Cost of Ownership

Hidden Expenses: Infrastructure, maintenance, and operational costs can surpass initial expectations.
Opportunity Cost: Resources allocated here might yield better returns if invested elsewhere.

3. Complexity and Maintenance

High Complexity: The intricacies of setting up a scalable, reliable monitoring system can be overwhelming.
Continuous Updates: Ongoing need to update, patch, and secure the system adds to the workload.

4. Scalability Concerns

Performance Bottlenecks: Scaling Prometheus for large environments requires significant effort.
Risk of Downtime: Greater potential for outages if the system isn't managed properly.

Known Challenges

1. Storage Limitations

Local Storage Constraints: Prometheus stores data locally, which can lead to data loss if a node fails.
Scaling Storage Limited: by the storage capacity and performance of individual nodes.

2. Achieving High Availability

Complex Architectures Required: Setting up redundancy involves multiple Prometheus instances and data replication strategies.
Increased Maintenance: More components to manage and monitor.

3. Alerting Complexities

Managing Alert Noise: Without careful tuning, you may experience alert fatigue due to excessive notifications.
Configuration Overhead: Requires meticulous setup to ensure critical alerts aren't missed.

4. Limited Long-Term Data Retention

Short Default Retention: Prometheus defaults to a 15-day data retention period.
Need for External Storage Solutions: Requires integration with systems like Thanos or Cortex for extended retention.

5. Security Limitations

Lack of Built-In Security: Prometheus doesn't offer advanced authentication and authorization out-of-the-box.
Manual Security Implementations: Additional effort needed to secure data in transit and at rest.

Unknown Challenges

1. Evolving Requirements

Changing Business Needs: As your organization grows, your observability requirements may change, necessitating redesigns.
Technology Shifts: New technologies may introduce unforeseen integration challenges.

2. Community Support Variability

Inconsistent Support: Reliance on community support can be risky when dealing with critical issues.
Documentation Gaps: May lack comprehensive guides for complex setups or troubleshooting.

3. Integration Complexities

Custom Development: Integrating with proprietary or less-common systems may require custom solutions.
Data Consistency Issues: Ensuring consistent data across multiple systems can be challenging.

4. Performance Tuning

Non-Optimal Defaults: Default configurations may not suit your specific workload, requiring extensive tuning.
Continuous Optimization: Ongoing effort needed to maintain optimal performance as the system scales.

The Hidden Costs of Prometheus

While Prometheus itself is open-source and free to use, deploying it as part of an in-house observability platform involves several hidden costs that can accumulate over time.

1. Infrastructure Costs

Hardware and Storage Requirements

Scalability Demands Increased: monitoring needs require more computational resources, storage, and memory.
High Availability (HA): Implementing HA doubles or triples infrastructure requirements due to redundant instances.
Network Bandwidth: Collecting metrics from numerous endpoints consumes network resources, potentially necessitating upgrades.

Cloud Expenses

Resource Usage: Cloud deployments incur costs for compute instances, storage, and data transfer.
Data Transfer Fees: High volumes of monitoring data can lead to substantial egress charges.

2. Operational Overhead

Maintenance and Upgrades

Continuous Updates: Regularly updating Prometheus and associated tools to the latest versions demands ongoing effort.
System Monitoring: The observability platform itself requires monitoring to ensure reliability.

Configuration Complexity

Advanced Configurations: Tailoring Prometheus for complex environments requires deep expertise.
Alerting Rules Management: Crafting effective alerting rules is time-consuming and needs regular updates.

3. Scaling Challenges

Performance Tuning

Resource Optimization: Achieving optimal performance involves extensive tuning and testing.
Sharding and Federation: Scaling horizontally introduces additional layers of complexity.

External Tools Integration

Additional Components: Tools like Thanos or Cortex are necessary for scalability but add complexity and costs.
Learning Curve: Each new tool requires time to learn and integrate effectively.

4. Data Retention and Storage Costs

Limited Retention Period

Default Limitations: The default 15-day retention may not meet business or compliance needs.
Extended Storage Needs: Long-term retention requires additional storage solutions.

External Storage Solutions

Implementation Effort: Setting up external storage systems is complex and resource-intensive.
Cost of Storage: Long-term storage incurs additional hardware or cloud storage fees.

5. Security Expenditures

Lack of Native Security Features

Authentication and Authorization: Additional tools or custom solutions are needed to secure Prometheus.
Encryption: Implementing encryption for data at rest and in transit adds complexity.

Compliance Measures

Regulatory Compliance: Ensuring the observability platform meets compliance standards can be costly.

6. Staffing and Expertise

Talent Acquisition

Hiring Specialists: Experienced Prometheus engineers command high salaries due to demand.
Consultancy Fees: External consultants are expensive but may be necessary for complex implementations.

Training Existing Staff

Learning Curve: Training staff diverts time from other productive activities.
Ongoing Education: Continuous learning is required to keep up with updates and best practices.

7. Integration and Custom Development

Third-Party Integrations

Custom Solutions:
API Management:

Dashboarding and Visualization

Additional Tools Needed:

8. Alert Management Overhead

Alert Fatigue

Signal-to-Noise Ratio: Excessive alerts can overwhelm teams, leading to important alerts being missed.
Fine-Tuning Required: Significant effort is needed to configure alerts properly.

On-Call Burden

Increased Stress: Poorly managed alerts increase stress and can affect team morale.
Operational Costs: More alerts can lead to higher operational expenses due to increased incident responses.

9. Opportunity Costs

Focus Diversion

Core Business Neglect: Time spent on the observability platform could be used for core product development.
Time-to-Market Delays: Diverting resources may delay the launch of revenue-generating features.

Innovation Lag

Technology Adoption: Focusing internally may cause delays in adopting new, more efficient technologies.

10. Community and Support Limitations

Limited Official Support

No Vendor Support: Reliance on community support can be risky for critical systems.
Documentation Gaps: May lack detailed documentation for advanced configurations.

Reliance on Community Contributions

Unpredictable Roadmap: Dependence on the community for updates can be uncertain.
Compatibility Issues: Updates may not align with your custom setup, requiring additional work.
Unpredictable Roadmap: Dependence on the community for updates can be uncertain.
Compatibility Issues: Updates may not align with your custom setup, requiring additional work.

11. Legal and Compliance Risks

License Compliance

Understanding Licenses: Integrations may involve various open-source licenses requiring careful management.
Intellectual Property Concerns: Mismanagement can lead to legal issues.

12. Hidden Bugs and Reliability Issues

Testing Requirements

Quality Assurance: Ensuring reliability demands rigorous testing efforts.
Bug Fixes: May require diving into the codebase, which is time-consuming.

Downtime Cost

System Outages: Failures in the observability platform can lead to undetected issues elsewhere.
Recovery Efforts: Restoring services can be resource-intensive.

13. High Cardinality Risks

High cardinality refers to metrics that have a large number of unique label combinations. While labels in Prometheus are powerful for slicing and dicing data, they can introduce significant challenges when not managed properly.

Impact on Availability

Performance Degradation: High cardinality can lead to increased memory usage and CPU load, causing Prometheus servers to become unresponsive or crash.
Slow Queries: PromQL queries over high-cardinality metrics can be slow, impacting dashboard performance and alerting delays.

Increased Costs

Resource Consumption: More unique metrics require more storage space and processing power, leading to higher infrastructure costs.
Scaling Requirements: To handle high cardinality, you may need to scale your Prometheus setup horizontally, adding complexity and expense.

Management Challenges

Difficult to Monitor: Keeping track of which metrics are contributing to high cardinality can be challenging.
Troubleshooting Overhead: Identifying and resolving issues caused by high cardinality consumes valuable time and resources.

Risk Mitigation Strategies

To navigate these challenges and hidden costs, consider the following strategies:

1. Start Small and Scale Gradually

Pilot Projects: Begin with a limited scope to understand the complexities involved.
Incremental Scaling: Expand the platform as your team's expertise grows.

2. Leverage Managed Services

Hybrid Approach: Use managed Prometheus services to offload some operational burdens.
Cloud-Native Solutions: Consider services like Amazon Managed Service for Prometheus.

3. Invest in Training and Talent

Skill Development: Provide comprehensive training for your team.
Hire Experts: Bring in experienced personnel to guide the implementation.

4. Utilize Complementary Tools

Long-Term Storage: Integrate with Thanos or Cortex for scalable storage solutions.
Visualization Tools: Use Grafana for advanced dashboards and analytics.

5. Automate and Standardize

Infrastructure as Code: Employ tools like Terraform or Ansible for consistent deployments.
Continuous Integration/Continuous Deployment (CI/CD): Automate testing and deployments.

6. Implement Security Best Practices

Secure Configurations: Use reverse proxies, service meshes, or OAuth proxies for authentication.
Compliance Audits: Regularly audit the system for compliance with security standards.

7. Manage High Cardinality

Label Best Practices: Limit the use of labels that have high cardinality, such as user IDs or timestamps.
Metric Aggregation: Pre-aggregate metrics where possible to reduce the number of unique series.
Monitoring Tools: Use tools to analyze and report on metric cardinality to identify and address issues early.

8. Engage with the Community

Active Participation: Stay involved with the Prometheus community for updates and support.
Contribute Back: Sharing your solutions can improve the ecosystem and build goodwill.

Conclusion

Building an in-house observability platform using Prometheus offers the potential for customization and control, but it's essential to be aware of the hidden costs and challenges involved. From infrastructure expenses to operational overhead and security considerations, the total cost of ownership can be substantial. High cardinality metrics, in particular, pose significant risks to both the availability and cost-effectiveness of your Prometheus deployment.

Before embarking on this journey, conduct a thorough cost-benefit analysis that accounts for these hidden costs. Consider whether your organization has the resources and expertise to manage these challenges effectively. By starting small, leveraging managed services, investing in talent, and adopting best practices—including careful management of metric cardinality—you can mitigate risks and build a robust observability platform that aligns with your organization's needs.

Remember, the goal of observability is to enhance your ability to understand and improve your systems—not to become a drain on resources. Choose the path that best supports your organization's strategic objectives while providing the insights you need to succeed.