I already wrote about my flow when I deploy one container per service and use it for all of my projects. So it's easier to get a development process up to speed.
This time I'll tell you about connecting containerized services so they can talk to each other. And we'll start with somewhat easy things like Prometheus and Grafana. And after that it will be a no-brainer for us to spin up an ELK stack. Let's go! 🚀
All code examples and configs are available on my GitHub repo.
If you didn't know what Prometheus is used for - it collects various metrics from a service. And a little side thing we need to do before setting up our God of Fire is to create a service we'll be monitoring. I chose it to be a simple Flask app because it requires almost zero efforts.
flask_app directory and put next two files there:
# flask_app/app.py from flask import Flask from prometheus_flask_exporter import PrometheusMetrics app = Flask(__name__) metrics = PrometheusMetrics(app)
# flask_app/wsgi.py from app import app
requirements.txt right next to previous two:
click==7.1.2 Flask==1.1.2 itsdangerous==1.1.0 Jinja2==2.11.3 MarkupSafe==1.1.1 prometheus-client==0.9.0 prometheus-flask-exporter==0.18.1 Werkzeug==1.0.1
Dockerfile, same place:
FROM python:3.9 ENV PYTHONUNBUFFERED 1 ENV PYTHONDONTWRITEBYTECODE 1 EXPOSE 8000 RUN apt-get update \ && apt-get install -y build-essential \ && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \ && apt-get clean -y && rm -rf /var/lib/apt/lists/* \ && pip install --upgrade pip WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt COPY . . ENTRYPOINT ["gunicorn", "wsgi:app", "--bind", "0.0.0.0:8000"]
That's quite mouthful but all it does is:
- creating an image from python 3.9;
- opens up port 8000;
- updates everything;
- and installs our great app.
Now we build!
docker build . -t flask_app
And now we run!!
docker run -d -p 8000:8000 flask_app --name flask_app
And on http://0.0.0.0:8000/metrics we can see a bunch of stuff!!1 💪
It's time to start metering. Open
my our favorite tool to manage containers - Portainer. And first, create a volume. Actually, two volumes. If you've read my previous article you know how it goes. But in case you didn't (why didn't you tho?) - go to "Volumes" tab and hit that blue "Add volume" button. First volume will be used for Prometheus data. So let's name it
prometheus_data. Second will be used for Prometheus config, therefore the name
"Why data for config?" you might ask. Well, because apparently you can't access the Prometheus container via the Portainer console button. Or I'm just stupid 🤷♂️ But we need a way to edit config files without rebuilding container every time. Thus volume for config files.
Creating container, finally. Go to "Containers" and smash the "Add container" button. Then look at the screenshot below and fill values for Name, Image, Ports and Volumes. Maybe change Restart Policy to Unless stopped.
Then hit the "Deploy the container" button and wait a little.
don't know how to can't connect to a Prometheus container we will edit it's config file via terminal session. If you're using nano like I do then just execute next line and make it look the same as below:
sudo nano /var/lib/docker/volumes/prometheus_config_data/_data/prometheus.yml
# my global config global: scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. # scrape_timeout is set to the global default (10s). # Alertmanager configuration alerting: alertmanagers: - static_configs: - targets: # - alertmanager:9093 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. rule_files: # - "first_rules.yml" # - "second_rules.yml" # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. - job_name: 'prometheus' # metrics_path defaults to '/metrics' # scheme defaults to 'http'. static_configs: - targets: ['localhost:9090'] - job_name: 'flask_app' metrics_path: '/metrics' static_configs: - targets: ['172.17.0.1:8000']
It's the same default config but with additional
job_name at the end. I decided to be explicit with
metrics_path here, showing that we can change that if we need to. But the most important part is
targets! You can just believe me that on Linux your host machine's IP address will be
172.17.0.1 for your docker containers. Or you can visit a "Network" tab in Portainer and take a look at "IPV4 IPAM Gateway" for a
Now restart Prometheus container and visit http://0.0.0.0:9090/targets. You should see "State"
UP for both targets.
Cool, eh? Let's make it beautiful! 🐹
Again, create a volume, for example
grafana_data. Then create container as on screenshot below
Hit deploy, wait and go to http://0.0.0.0:3000/. Default login/password is admin/admin. You can change that after login.
Configure data source on http://0.0.0.0:3000/datasources. Or locate "Data Sources" tab hovering on ⚙️ sign on the left. Then click the blue "Add data source" button, select "Prometheus". Very important, as before, "URL" parameter inside the "HTTP" section should be
172.17.0.1:9090. Because Grafana can't access Prometheus container by host name. So we tell it (her?) to connect to our host machine IP address on the port where Prometheus is running. Then smash "Save & Test". As always, you can try to set "URL" to
prometheus_container_name:port and after saving observe "HTTP Error Bad Gateway".
Now we can hover over ➕ sign and click on "Dashboard" under "Create". Or just visit http://0.0.0.0:3000/dashboard/new. Then smash "Add new panel". On the right side change "Title" to whatever your heart desires. I chose simple "Flask Total Requests". Below graph set "Metrics" to
flask_http_request_total. Here is a handy screenshot for you:
Then click "Apply" in the top right corner. And now you have Grafana which makes beautiful graphs from Prometheus data which monitors your application!
In the next article we will set up ELK. But I guess you already know how to do that 😉