TL;DR
This article lists 6 top projects for simplifying the deployment and building scalable cloud-based applications for developers. 🎉
Feel free to explore these projects, contribute to your favorites, and support them by starring their repositories.🌟
Odigos - Open-source Distributed Tracing
Monitor all your apps simultaneously without writing a single line of code!
Simplify OpenTelemetry complexity with the only platform that can generate distributed tracing across all your applications.
We are really just starting out.
Can you help us with a star? Plz? 😽
https://github.com/keyval-dev/odigos
1. OpenTelemetry
💡 Collect telemetry data and forward it to analysis tools.
OpenTelemetry is an observability toolkit that helps us collect data like traces, metrics, and logs. It is vendor and tool-neutral 🛠️ meaning it can work with different Observability Backends.
What OpenTelemetry is not? 🫣
OpenTelemetry is not a monitoring backend. Many people confuse themselves between OpenTelemetry and Observability backends like Jaegar, Datadog, etc. OpenTelemetry only generates, collects, and exports telemetry data. Storage and visualization are left to other tools. 📊
OpenTelemetry sits between the code and the observability tools we use. This provides us with flexibility and makes it easier to switch tools. 🔄
To better understand how OTel (short for OpenTelemetry) works, take a look at the below image. 👇🏻
2. Kubernetes
💡 Container orchestration platform for managing applications.
Kubernetes is the most important repo on the list when we are talking about cloud-native applications. How, can we skip Kubernetes right? 😜 It is the most famous orchestration platform for container-based applications and a foundation for many cloud-native tools to work.
What is Container Orchestration, and why do we even need it? 🤔
A container engine is sufficient for managing a few containers, but for hundreds or thousands of containers and hundreds of services running, the complexity grows huge 📈. Trust me, it is something I have had experience with. 🥲
This is where the orchestration engine like Kubernetes comes into play.
The best way I think of Kubernetes is as a puppet master 🎭 for containers, manipulating their actions, coordinating their tasks, and ensuring they work in harmony across a cluster of servers.
Refer to the image below to get a better understanding of what I meant by puppet master.
3. Backstage
💡 Unifies tools for a streamlined development environment.
Backstage is an open-source project that serves as a developer portal, offering a unified view of infrastructure resources, CI/CD pipelines, and observability metrics. 📊 It gives us all the building blocks to create a platform ourselves that manages our complex software development ecosystem.
It provides us with easy access to essential information without requiring in-depth tool knowledge 🧰, acting as a central hub 🏛️ for company-wide development needs.
The core functionalities of Backstage are:
- Service Catalog 📋: This is the core Backstage foundation. It allows us to track and manage all our services, apps, and pipelines in one place.
- Software Templates 📄: With the Backstage UI interface, we can easily create new services with just a few clicks or use a YAML template to add necessary resources.
- Tech Documentations 📃: It adds seamless technical documentation management, ensuring teams can effortlessly keep content up-to-date, all from their code repositories.
In short, this abstracts away a lot of CI/CD and provides us with information on the company's services. 🚀 I would say this is a must-have tool for any company that is planning to upscale and streamline their development operations.
4. Prometheus
💡 System monitoring tool with time-series data.
Prometheus is a free open-source software application used for event monitoring and alerting. 🧐 It records real-time metrics in a time series database (allows querying the data using its own query language called PromQL).
It also provides an alerting system that allows us to define thresholds and conditions for alerts.
When certain metrics exceed or fall below defined thresholds, Prometheus can trigger notifications, helping in successfully addressing potential issues before they impact the system's health.
In short, Prometheus helps keep a close watch 👀 on our infrastructure's health and performance.
🔴 GOOD TO KNOW: Notice that I've mentioned two similar projects, OpenTelemetry and Prometheus, which serve similar purposes. 🔄
Prometheus is primarily a system-level monitoring and alerting tool, while OpenTelemetry is focused on collecting and forwarding application-level telemetry data.
Prometheus didn't initially support OpenTelemetry metrics, but now they can work together.
For more details, you can refer to this article.
5. Cilium
💡 Efficient eBPF-based networking for containers.
Cilium is a free and open-source software solution that provides networking and security 🛡️ for containerized applications.
Cilium leverages eBPF to provide high-performance networking and security and has been designed to integrate with container orchestration platforms like Kubernetes. ☸️
One important feature of Cilium is its ability to apply security rules at the network level, also allows us to encrypt and authenticate network traffic, making sure that data is transmitted securely between containers. 🐋
In short, Cilium secures container workloads in Kubernetes with eBPF, offering network security, encryption, and authentication.
To better understand how Cilium works at its architectural level, take a look at the below image. 👇🏻
6. Rook
💡 Simplifies cloud-native persistent storage management.
This tool really is a lifesaver for anyone who is working with the Ceph storage. 😍
To understand and implement Rook, you must have a basic understanding of Ceph software-defined storage. If not, let me briefly introduce Ceph.
What exactly is Ceph? 🧐
Ceph is an open-source software storage platform that provides object storage, block storage, and file system storage in a single unified storage cluster. 📦
Rook simplifies the deployment and management of Ceph storage systems on Kubernetes.
By deploying Rook on Kubernetes, we can make Ceph storage highly available, automated 🤖, and simple to operate at scale.
In short, Rook turns Ceph into a fully managed service for Kubernetes and abstracts away the complexity of Ceph, making it easy to set up and manage cloud-native storage for our containers.
Introducing Odigos
💡Distributed tracing without code changes.
If you are working with OpenTelemetry then this is the tool you need. ✨
Odigos automatically generates telemetry data in OpenTelemetry format to any Observability backend, without any code change. 😲
It achieves this all by the following:
- Automated Instrumentation 🤖: Odigos enables automated instrumentation of applications with OpenTelemetry and eBPF, eliminating the need for code modifications.
- Universal Observability Tool Compatibility 🤝: It smoothly integrates with various observability tools, providing comprehensive support and efficient collector management.
- Ease of installation ✨: Odigos excels in user-friendliness, seamlessly integrating OpenTelemetry, ensuring easy installation and adaptability to specific requirements with minimal hurdles.
We've recently published an article on implementing Odigos. Read here.
If you think of any other helpful projects that I haven't covered in this article, please share them in the comments section below. 👇🏻
So, that is it for this article.
Thank you so much for reading! 🎉🫡
Top comments (9)
Love this article! 💓💓
Its a great article but it actually it contains 6 tools instead of 5
Great Article.
Fantastic cat graphic, great list!
Just Amazing
Great Article !
Great read. Took notes and ready to try at least one from the list.
Great piece!
amazing article