TL;DR
In this easy-to-follow tutorial, you'll discover how to monitor your Python application using distributed tracing.
What you will learn: β¨
- How to build microservices in Python π.
- Setting up Docker containers π¦ for microservices.
- Configuring Kubernetes for managing microservices.
- Integrating a tracing backend for visualizing the traces π΅οΈββοΈ.
Ready to light up your Python application monitoring skills? π₯
Setup Instructions
π¨ In this section of the blog, we'll be building a dummy Python microservices application. If you already have one and are following along, feel free to skip this part.
Create the initial folder structure for your application as shown below. π
mkdir python-microservices && cd python-microservices
mkdir src && cd src
mkdir microservice1 microservice2
Setting Up the Server π₯οΈ
For demonstration purposes, I will create two microservices that will communicate with each other, and eventually, we can use that to visualize distributed tracing.
Building and Dockerizing Microservice 1
Inside the /microservice1
directory, create a new Python virtual environment, install the necessary dependencies, and initialize a Flask application.
π¨ I assume you are following along on a Unix-based machine. If you are on a Windows machine, some commands will differ.
cd microservice1
python -m venv .venv
source .venv/bin/activate
π‘ If you are on a fish shell, run the below command to activate the virtual environment.
source .venv/bin/activate.fish
Install the required dependencies:
pip install Flask requests
Get the list of installed dependencies in the requirements.txt
so that we can later use it in our container to install the dependencies.
pip freeze > requirements.txt
Create a new file called app.py
and add the following lines of code:
# π src/microservice1/app.py
import socket
import requests
from flask import Flask, jsonify, render_template
app = Flask(__name__)
def user_os_details():
hostname = socket.gethostname()
hostip = socket.gethostbyname(hostname)
return hostname, hostip
@app.route("/")
def index():
return "<p>Welcome to Flask microservice 1</p>"
@app.route("/health")
def health():
return jsonify(status="Microservice 1 Running...")
@app.route("/get-users")
def get_users():
response = requests.get("http://microservice2:5001/get-gh-users")
return render_template("index.html", users=response.json())
@app.route("/os-details")
def details():
host_name, host_ip = user_os_details()
return jsonify(hostname=host_name, hostip=host_ip)
if __name__ == "__main__":
app.run("0.0.0.0", 5000)
π‘ If you've noticed, we're requesting data from
http://microservice2:5001/get-gh-users
. You might be wondering, what is this microservice2? Well, we can use service names as host names within the same network in docker. We will build this service later once we finish writing and dockerizing this microservice.
As you can see, this is a very simple Flask application with a few endpoints. The user_os_details()
function gets the Hostname and IP address of the machine.
The @app.route("/")
and @app.route("/health")
decorators define the root and "/health" endpoints of the Flask app. The "/health" endpoint will be used later to check the container health β€οΈβπ©Ή in the Dockerfile
.
The @app.route("/get-users")
and @app.route("/os-details")
decorators define the "/get-users" and "/os-details" endpoints. The "/get-users" endpoint fetches GitHub users from microservice2 and passes them as props to the index.html
file for rendering. Meanwhile, the "/os-details" endpoint displays system details. Finally, the application is run on port 5000.
Now, let's create the index.html
file where we are rendering the received users from microservice2.
Create a new folder /templates
and add an index.html
file with the following contents:
<!-- π src/microservice1/templates/index.html -->
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Microservice 1</title>
</head>
<body>
<h1>Microservice 1</h1>
<h2>This is the data received from microservice2:</h2>
<p>{{ users }}</p>
</body>
</html>
Our first microservice is all ready. Let's dockerize it. Create a Dockerfile
inside the /microservice1
directory and add the following lines of code:
π¨ Make sure to name it exactly
Dockerfile
with no extensions.
# π src/microservice1/Dockerfile
# Use Python alpine as the base image.
FROM python:3.12.1-alpine3.18
# Optional: Upgrade pip to the latest version.
RUN pip install --upgrade pip
# Create a new user with fewer permissions.
RUN adduser -D lowkey
# Switch to user lowkey
USER lowkey
# Change the working directory to ~/app
WORKDIR /home/lowkey/app
# Copy the requirements.txt file required to install the dependencies.
COPY --chown=lowkey:lowkey requirements.txt requirements.txt
# Install the dependencies
RUN pip install -r requirements.txt
# Copy the rest of the files to the current directory in the docker container.
COPY --chown=lowkey:lowkey . .
# Expose port 5000
EXPOSE 5000
# Switch to the root user just for installing curl. It is required.
USER root
# Install curl. Alpine uses apk as its package manager.
RUN apk --no-cache add curl
# Switch back to user lowkey
USER lowkey
# Check the health of the container. The "/health" endpoint is used
# to verify the container is up and running.
HEALTHCHECK --interval=30s --timeout=30s --start-period=30s --retries=5 \
CMD curl -f http://localhost:5000/health || exit 1
# Finally, start the application.
ENTRYPOINT [ "python", "app.py" ]
Create a .dockerignore
file with the names of the files we don't want to push to the container.
__pycache__
.venv
README.md
Dockerfile
.dockerignore
Now, that is the entire setup for our first microservice. β¨
Building and Dockerizing Microservice 2
We will have a setup similar to microservice1, with just a few changes here and there.
Do the exact initial setup as we did for the microservice1. After that, in the /microservice2
folder, create an app.py
file and add the following lines of code:
# π src/microservice2/app.py
import random
import requests
from flask import Flask, jsonify, render_template
app = Flask(__name__)
def get_gh_users():
url = "https://api.github.com/users?per_page=5"
# Choose a random timeout between 1 and 5 seconds
timeout = random.randint(3, 6)
try:
response = requests.get(url, timeout=timeout)
return response.json()
except requests.exceptions.Timeout:
return {"error": "Request timed out after {} seconds".format(timeout)}
except requests.exceptions.RequestException as e:
return {"error": "Request failed: {}".format(e)}
@app.route("/")
def index():
return "<p>Welcome to Flask microservice 2</p>"
@app.route("/get-gh-users")
def get_users():
results = []
# Loop through the number of requests and append the results to the list
for _ in range(3):
result = get_gh_users()
results.append(result)
# Return the list of results as a JSON response
return jsonify(results)
@app.route("/health")
def health():
return jsonify(status="Microservice 2 Running...")
@app.route("/os-details")
def details():
try:
response = requests.get("http://microservice1:5000/os-details").json()
host_name = response["hostname"]
host_ip = response["hostip"]
return render_template("index.html", hostname=host_name, hostip=host_ip)
except requests.exceptions.Timeout as errt:
return {"error": "Request timed out after {} seconds".format(errt)}
except requests.exceptions.RequestException as e:
return {"error": "Request failed: {}".format(e)}
if __name__ == "__main__":
app.run("0.0.0.0", 5001)
The @app.route("/")
decorator defines the root endpoint of the Flask app, returning a welcome message. The @app.route("/health")
decorator defines the "/health" endpoint, which can be used to check the health status of the container.
The @app.route("/get-gh-users")
decorator defines the "/get-gh-users" endpoint, which uses the get_gh_users()
function to fetch GitHub users and return them as a JSON response. Lastly, the @app.route("/os-details")
decorator defines the "/os-details" endpoint, which retrieves operating system details from microservice1 and renders them in the index.html
file. Finally, the application runs on port 5001.
Now, let's create the index.html
file where we are rendering the received users from microservice2.
Create a new folder /templates
and add an index.html
file with the following contents:
<!-- π src/microservice2/templates/index.html -->
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Microservice 2</title>
</head>
<body>
<h1>Microservice 2</h1>
<h2>This is the hostname and IP address received from the microservice1:</h2>
<p>{{ hostname }} - {{ hostip }}</p>
</body>
</html>
Now, it's time to dockerize this microservice as well. Copy and paste the entire Dockerfile
content for microservice1 and just change the port from 5000 to 5001.
Also, add a .dockerignore
file and include the same files that we added when creating microservice1.
Now, that is the entire setup for our second microservice as well. β¨
Building Dockerfiles with Docker Compose
We will follow the best practices of image building by using Docker Compose instead of building each image manually. Here we have only two images, but imagine if we had hundreds or thousands of Dockerfiles. Building each manually would be a tedious task. π΄
In the root of the project, create a new file named docker-compose.yaml
and add the following code:
services:
microservice1:
build:
context: ./src/microservice1
dockerfile: Dockerfile
image: microservice1-image:1.0
ports:
- "5000:5000"
restart: always
microservice2:
build:
context: ./src/microservice2
dockerfile: Dockerfile
image: microservice2-image:1.0
ports:
- "5001:5001"
restart: always
This Docker Compose file defines two services, microservice1 and microservice2. Each service is built using its respective Dockerfile
located in the /src
directory, with microservice1 mapped to port 5000 and microservice2 to port 5001.
The resulting images are tagged microservice1-image:1.0
and microservice2-image:1.0
, respectively. Both services are set to restart always, making sure if the container fails it restarts.
Now, build the images using the following command:
docker compose build
Deployment on Kubernetes π§βπ
Make sure Minikube is installed, or follow this link for installation instructions. π
Create a new local Kubernetes cluster, by running the following command. We will need it when setting up Odigos and Jaeger.
Start Minikube: π
minikube start
Since we are running on a local Kubernetes environment, we need to point our shell to use the minikube's docker-daemon.
To point your shell to minikube's docker-daemon, run:
minikube -p minikube docker-env | source
And now, when running any Docker commands such as docker images
or docker ps
, you will see what is inside Minikube rather than what you have locally on your system.
Now that we have both of our microservices ready and dockerized, it's time to set up Kubernetes for managing these services.
At the root of the project, create a new folder /k8s/manifests
. Inside this folder, we will add deployment and service configurations for both of our microservices.
- Deployment Configuration π: For actually deploying the containers on the Kubernetes Cluster.
- Service Configuration π: To expose the pods to both within the cluster and outside the cluster.
First, let's create the manifest for the microservice1
. Create a new file microservice1-deployment-service.yaml
and add the following content:
// π k8s/manifests/microservice1-deployment-service.yaml
version: apps/v1
kind: Deployment
metadata:
name: microservice1
spec:
selector:
matchLabels:
app: microservice1
template:
metadata:
labels:
app: microservice1
spec:
containers:
- name: microservice1
image: microservice1-image:1.0
imagePullPolicy: Never
resources:
limits:
memory: "200Mi"
cpu: "500m"
ports:
- containerPort: 5000
---
apiVersion: v1
kind: Service
metadata:
name: microservice1
labels:
app: microservice1
spec:
type: NodePort
selector:
app: microservice1
ports:
- port: 8080
targetPort: 5000
nodePort: 30001
This configuration deploys a microservice named microservice1 with resource limits of 200MB memory ποΈ and 0.5 CPU cores. It exposes the microservice internally on port 5000 through a Deployment and externally on NodePort 30001 through a Service.
π€ Remember the
docker-compose build
command we used when building our Dockerfiles, especially the image names? We are using the same images to create the containers.
It is accessible on port 8080 within the cluster. We assume microservice1-image:1.0
is locally available with imagePullPolicy: Never
. If this is not in place, it would attempt to pull the image from the Docker Hub π and fail.
Now, let's create the manifest for microservice2. Create a new file named microservice2-deployment-service.yaml
and add the following content:
// π k8s/manifests/microservice2-deployment-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: microservice2
spec:
selector:
matchLabels:
app: microservice2
template:
metadata:
labels:
app: microservice2
spec:
containers:
- name: microservice2
image: microservice2-image:1.0
imagePullPolicy: Never
resources:
limits:
memory: "200Mi"
cpu: "500m"
ports:
- containerPort: 5001
---
apiVersion: v1
kind: Service
metadata:
name: microservice2
labels:
app: microservice2
spec:
type: NodePort
selector:
app: microservice2
ports:
- port: 8081
targetPort: 5001
nodePort: 30002
It is similar to the manifest for microservice1
, with just a few changes.
This configuration deploys a microservice named microservice2 and exposes it internally on port 5001 through a Deployment and externally on NodePort 30002 through a Service.
Accessible on port 8081 within the cluster, assuming the microservice2-image:1.0
is locally available with imagePullPolicy: Never
.
Once, this is all done, make sure to apply these configurations and start the Kubernetes cluster with these services. Change the directory to /manifests
and execute the following commands:
kubectl apply -f microservice1-deployment-service.yaml
kubectl apply -f microservice2-deployment-service.yaml
Check that both our deployments are Running by executing the following command:
kubectl get pods
Finally, our application is ready and deployed on Kubernetes with the necessary deployment configurations. π
Installing Odigos π§βπ»
π‘ Odigos is an open-source observability control plane that enables organizations to create and maintain their observability pipeline. In short, we will use Odigos to auto-instrument our Python application.
βΉοΈ If you are running on a Mac run the following command to install Odigos locally.
brew install keyval-dev/homebrew-odigos-cli/odigos
βΉοΈ If you are on a Linux machine, consider installing it from GitHub releases by executing the following commands. Make sure to change the file according to your Linux distribution.
βΉοΈ If the Odigos binary is not executable, run this command
chmod +x odigos
to make it executable before running the install command.
curl -LJO https://github.com/keyval-dev/odigos/releases/download/v1.0.15/cli_1.0.15_linux_amd64.tar.gz
tar -xvzf cli_1.0.15_linux_amd64.tar.gz
./odigos install
If you need more brief instructions on its installation, follow this link.
Now, Odigos is ready to run π. We can execute its UI, configure the tracing backend, and send traces accordingly.
Connecting Odigos to a Tracing Backend β¨
π‘ Jaeger is an open source, end-to-end distributed tracing system.
Setting up Jaeger β¨
For this tutorial, we will use Jaeger π΅οΈββοΈ, a popular open-source platform for viewing distributed traces in a microservices application. We will use it to view the traces generated by Odigos.
For Jaeger installation instructions, follow this link. π
To deploy Jaeger on a Kubernetes cluster, run the following commands:
kubectl create ns tracing
kubectl apply -f https://raw.githubusercontent.com/keyval-dev/opentelemetry-go-instrumentation/master/docs/getting-started/jaeger.yaml -n tracing
Here, we are creating a tracing
namespace and applying the deployment configuration π for Jaeger in that namespace.
This command sets up the self-hosted Jaeger instance and its service.
Run the below command to get the status of the running pods:
kubectl get pods -A -w
Wait for all three pods to be Running before proceeding further.
Now, to view the Jaeger Interface locally, we need to port forward. Forward traffic from port 16686 on the local machine to port 16686 on the selected pod within the Kubernetes cluster.
kubectl port-forward -n tracing svc/jaeger 16686:16686
This command creates a tunnel between the local machine and the Jaeger pod, exposing the Jaeger UI so you can interact with it.
Now, on http://localhost:16686
, you should be able to see the Jaeger instance running.
Configuring Odigos to Work with Jaeger π
βΉοΈ For Linux users, go to the folder where you downloaded the Odigos binaries from GitHub releases and run the following command to launch the Odigos UI.
./odigos ui
βΉοΈ For Mac users, just run:
odigos ui
Visit http://localhost:3000
and you will be presented with the Odigos interface where you will see both deployments in the default
namespace.
Select both of these and click Next. On the next page, choose Jaeger as the backend, and add the following details when prompted:
- Destination Name: Give any name you want, let's say python-tracing.
-
Endpoint π―: Add
jaeger.tracing:4317
for the endpoint.
And that's it β Odigos is all set to send traces to our Jaeger backend. It's that simple.
Viewing Distributed Tracing π§
Odigos has already begun sending traces of our application to Jaeger as soon as we set up Odigos to work with Jaeger as our tracing backend.
Visit http://localhost:16686
and select both our microservices in the dropdown.
Make a few requests to our endpoints, and eventually, Jaeger will begin to populate with traces.
Click on any of the requests and explore the traces.
This was all done without changing a single line of code. π₯
Wrap-Up! β‘
So far, you've learned to closely monitor π your Python application with distributed tracing, using Odigos as the middleware between your application and the tracing backend Jaeger.
The source code for this tutorial is available here:
https://github.com/shricodev/blogs/tree/main/odgs-monitor-PY-like-a-pro
Thank you so much for reading! π π«‘
Drop down your thoughts in the comment section below. π
Top comments (15)
This is the first time Iβm using such a cover for my blog post. Let me know if you like it or not! π
This is a great in-depth article, broo. I love it. And this cover looks more professional than the previous ones.
Appreciate it, brother. π€ I plan to use this for my future blogs. The previous ones took quite some time to get ready, so it was quite a hassle.
It looks wonderful! What did you use to make it?
@best_codes I'm glad you loved it. I actually used Canva.
You must be good at Canva. I've tried it before and didn't do anywhere near as good! :D
Keep up the good work.
nice one buddy β€οΈπ
Glad you liked it, Lokesh! π
Python talk of the town developer or what? π Two posts on active discussions.
@bh4skar π
Please share your thoughts on this article in the comments section! ;)
This is a real good article. Thank You brother. I keep it bookmarked. Never knew about Odigos though .
Thank you, @shekharrr :)
Thank you for sharing.
Thanks. Really good article on the use of Odigos and Jaeger.