DEV Community: Mirco

Valet Key Design Pattern in 30 Seconds

Mirco — Thu, 18 Nov 2021 14:12:02 +0000

Do not waste your applications resources on big uploads.

Photo by Old Money on Unsplash

Routing large up- and downloads through your application absorbs valuable resources like bandwidth and CPU time. After the service authenticated the upload request, it uploads data to a storage system like Amazon S3 or Google Cloud Storage. Authentication is fast. The service wastes most of its time while it waits for the file to be saved by the storage system.

The service wastes most of its time while it waits for the file to be saved by the storage system.Uploading via the service locks resources.

Why waste so much valuable time? The Valet Key Pattern creates a remedy for this problem.

Clients do not call your service anymore to upload things. They only call your service to get a specialized key with limited validity. Then, they use this key to upload their data directly to the storage system. The storage system must be able to verify that the key is valid. In this way, your application will not have to wait for the upload.

Use the Valet Key Pattern to reduce load on your system.

On the downside, you lose some control over your storage system. You have to ensure that the key has the lowest possible scope. Limit the maximum upload size, the reusability of the key and its validity. If possible, make the client notify you after the upload is finished, so that you can invalidate the key right away.

Do not forget to validate and sanitize all uploaded data before using it. You can achieve this with an own service which regularly scans all new uploads.

Use this pattern only if clients must upload extensive files often. It will save you resources. And this will save you money.

Want to learn more about Cloud Design Patterns? Check out these articles:

If this article was helpful for your, please consider to buy me a coffee :-)

Circuit Breaker Pattern in 30 seconds

Mirco — Thu, 11 Nov 2021 13:47:00 +0000

In a microservice architecture, remote services may be unavailable for a long time. Retrying a request won’t help in this case.

The Circuit Breaker pattern offers a solution to this problem.

The CB has three states: closed, half-open, and open.

It starts in closed state. Your service executes all calls to the remote service.
CB switches to open if many requests fail. Your service starts a timer and does not execute any calls to the service. It returns a default answer or throws an exception instead.
CB switches to half-open when the timer is up. Your service executes a fraction of the calls. If all pass, it switches to “closed”. If one fails, it switches to “open” again and restarts the timer.

To summarize, the Circuit Breaker prevents the execution of calls which are doomed to fail. Look here/circuit-breaker) for more information.

Learn four more patterns here!

React faster: Forward Prometheus Alerts to Teams

Mirco — Fri, 05 Nov 2021 08:54:00 +0000

Many companies use Microsoft Teams. Receiving alerts in Teams allows you to react faster. It makes working on an incident visible, as you can answer the message. Let's add it to our monitoring stack!

Head over to GitHub to get the full code.

Teams configuration

Alertmanager does not support Teams out of the box. You must use webhooks to achieve this. The webhook is used by prom2teams to write messages.

Add a channel to the team which should receive the notifications. Click on the three dots beside the team name and choose Add channel.

To add the webhook, click on the dots beside the channel and click on Connectors.

Configure an Incoming Webhook. Teams shows the webhook URL once you click Create. Copy this URL in the next step.

Add prom2teams to the stack

prom2teams will use the webhook to send messages to Teams.

Add prom2teams to your docker-compose.yml:



  prom2teams:
    image: idealista/prom2teams:2.7.0
    restart: unless-stopped
    environment:
      PROM2TEAMS_CONNECTOR: "url from the webhook"
    ports:
      - 8089:8089

Update your alerts

The Alertmanager uses labels to decide which alert goes to which notification channel. Change the prometheus/alerts.yml to contain the following:




groups:
  - name: DemoAlerts
    rules:
      - alert: InstanceDown 
        expr: up{job="services"} < 1 
        for: 1m
        labels: # labels and annotations are new
          severity: low
        annotations:
          summary: 'Alert with low severity.'

      - alert: InstanceDownCritical
        expr: up{job="services"} < 1 
        for: 1m 
        labels:
          severity: high
        annotations:
          summary: 'Alert with high severity.'

Now you have two alerts with different labels.

Configure the Alertmanager

Change the alertmanager/alertmanager.yml so that the Alertmanager is aware of prom2teams.



route:
  group_by: [ alertname ]
  receiver: 'mail' # default receiver
  repeat_interval: 24h
  routes:
    - receiver: 'teams'
      repeat_interval: 12h
      matchers:
        - severity="medium"

    - receiver: 'teams'
      repeat_interval: 4h
      matchers:
        - severity="high"


receivers:
  - name: 'mail'
    email_configs:
      - smarthost: 'yourmailhost.com:465'
        auth_username: 'yourmail@yourmailhost.com'
        auth_password: "your mail password"
        from: 'yourmail@yourmailhost.com'
        to: 'someonesmail@yourmailhost.com'
        require_tls: false

  - name: 'teams'
    webhook_configs:
      - url: "http://prom2teams:8089"
        send_resolved: true

Now, the Alertmanager can publish to two channels.
matchers tell the Alertmanager which channel to use.

The Alertmanager sends alerts with medium severity to Teams. This is repeated every four hours - until you fix it!

The Alertmanager uses email if no route matcher matches.

You can use the routing tree editor to check if the configuration is what you need. Add labels and values of your alert to see which route they will take.

Check the result

Start the stack with docker-compose up and wait until the alerts fire.

You will get a notification from Teams for the InstanceDownCritical alert:

And an email for the InstanceDown alert:

Which notification channels do you want to see next? Leave me a message!
Head over to GitHub to get the full code.

If this article was helpful for your, please consider to buy me a coffee :-)

Monitoring setup with docker-compose - Part 3: Alertmanager

Mirco — Wed, 27 Oct 2021 11:47:00 +0000

In short: What is Alertmanager?

The Alertmanager sends alerts to various channels like Slack or E-Mail.
Recall from part one that Prometheus creates an alert if something violates a rule.

You can use the Alertmanager to silence and group alerts as well.

Configuration

You can get all source code from GitHub. Check out the tag part-2-grafana if you want to follow along.

docker-compose.yml

First of all, add the Alertmanager and a volume to docker-compose.yml:




  alertmanager:
    image: prom/alertmanager:v0.23.0
    restart: unless-stopped
    ports:
      - "9093:9093"
    volumes:
      - "./alertmanager:/config"
      - alertmanager-data:/data
    command: --config.file=/config/alertmanager.yml --log.level=debug


volumes:

  alertmanager-data:

Alertmanager will persist silence configurations to the volume.

alertmanager/alertmanager.yml

This configuration contains information about which channels to send to. For simplicity, we use E-Mail. Refer to the Alertmanager docs to learn about other channels.

Create a folder alertmanager and add a file alertmanager.yml to it:



route:
  receiver: 'mail'
  repeat_interval: 4h
  group_by: [ alertname ]


receivers:
  - name: 'mail'
    email_configs:
      - smarthost: 'smtp.gmail.com:465'
        auth_username: 'your_mail@gmail.com'
        auth_password: ""
        from: 'your_mail@gmail.com'
        to: 'some_mail@gmail.com'

The route section configures which alerts will be sent. In our case, we sent all alerts. You could add more routes and filter, for example, on alert tags (see the example in the docs.).

receivers configures our target channels. Note how route refers to the receiver mail on line two.

Connect to prometheus

Finally, we need to tell Prometheus about the Alertmanager.

Open prometheus/prometheus.yml and add the following:



alerting:

  alertmanagers:

    - scheme: http

      static_configs:

        - targets: [ 'alertmanager:9093' ]

Testing

Run docker-compose up. Open http://localhost:9093 in your browser to see the Alertmanager UI.

After a couple of minutes, the test alert fires. You can check this in your Prometheus instance.

Now, you can see the alert in the http://localhost:9093 as well:

Check your inbox for the notification:

Final note

This series shows how to set up a basic Prometheus stack. Keep in mind that this is not production-ready! But it is enough to get your hands dirty.

If this article was helpful for your, please consider to buy me a coffee :-)

Local Testing With Spring Data and GCP Datastore

Mirco — Wed, 27 Oct 2021 08:57:00 +0000

Developing in the cloud is quite different from traditional development. A recurring challenge is to write tests that work locally or on a CI system without access to the cloud.

This article presents a way to test services that use Spring Boot and Google’s Datastore.
Prerequisites

Java 11 ( which can easily be installed with sdkman.)
Docker and docker-compose
Basic knowledge Maven, Spring Boot and Google Cloud Platform.

Before we start

To see that everything runs locally, log out of gcloud and unset the project.

gcloud auth revoke
gcloud config unset project

The complete code is available on GitHub.

Project Setup

We start with a Spring Boot project which has all the necessary dependencies to use Datastore. You can find all code on GitHub.

The project contains a simple entity ( Article) and a spring-data repository with a custom method ( ArticleRepository ).
Let the test fail.

Checkout the tag 1-let-the-tests-fail if you want to follow this step by step.

If you run mvn clean install in this state, the test will fail since there is no active project:

[...]
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.google.cloud.datastore.Datastore]: Factory method ‘datastore’ threw exception; nested exception is java.lang.IllegalArgumentException: A project ID is required for this service but could not be determined from the builder or the environment. Please set a project ID using the builder.
Caused by: java.lang.IllegalArgumentException: A project ID is required for this service but could not be determined from the builder or the environment. Please set a project ID using the builder

This is easily fixed by adding a mock project id to the application.properties (See the git tag 2-add-project-id-property) :

spring.cloud.gcp.project-id=test-project

Maven (mvn clean install) now fails with the following error:

com.google.cloud.datastore.DatastoreException: Cloud Datastore API has not been used in project … before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/datastore.googleapis.com/overview?project=… then retry.

Spring now tries to connect to Datastore in test-project , which does not exist.
Make the tests green.

Here comes the Datastore emulator into play. You can install it via gcloud, but for portability, we will use a dockerized version.

We add the following docker-compose.yml to the root of the project:

version: '3'

services:

  datastore:
    image: knarz/datastore-emulator
    ports:
      - 8081:8432
    environment:
      CONSISTENCY: 1 # If you omit this, the emulator will emulate eventual consistency.

To ensure that Spring uses the emulator, we must add a property to application.properties that points to it:

spring.cloud.gcp.datastore.host=localhost:8081

(See the git tag 3-datastore-emulator for all code in this step)

Start the emulator with docker-compose up and run the test again with mvn clean install .

Now, the tests are green! You created a test with datastore which runs 100% locally!
Make it more comfortable.

Starting and stopping the container by hand is tedious and error-prone. Luckily, we can configure maven so that this happens automatically.

First, we add maven-failsafe-plugin to our build section in the pom.xml :

...
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <version>2.22.2</version>
</plugin>

The main task of the plugin is to run integration tests. You can discuss if our test qualifies as an integration test or not. However, there are no appropriate maven lifecycle phases for unit tests as there are for integration tests (which we’ll need in a second). Feel free to alter this step so that it suits your needs.

In order for maven-failsafe-plugin to execute our test, we have to rename it from ArticleRepositoryTest to ArticleRepositoryIT .

Then, we add exec-maven-plugin which will start the emulator before the integration tests and stop them afterward.

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>exec-maven-plugin</artifactId>
  <version>3.0.0</version>
  <executions>
    <execution>
      <id>start-datastore-emulator</id>
      <phase>pre-integration-test</phase>
      <goals>
        <goal>exec</goal>
      </goals>
      <configuration>
        <executable>docker-compose</executable>
        <arguments>
          <argument>up</argument>
          <argument>-d</argument>
        </arguments>
      </configuration>
    </execution>
    <execution>
      <id>stop-datastore-emulator</id>
      <phase>post-integration-test</phase>
      <goals>
        <goal>exec</goal>
      </goals>
      <configuration>
        <executable>docker-compose</executable>
        <arguments>
          <argument>down</argument>
          <argument>-v</argument>
        </arguments>
      </configuration>
    </execution>
  </executions>
</plugin>

Make sure to stop the docker container now with docker-compose down . Then, run mvn clean install to see that maven starts the emulator, runs the test and stops the emulator afterward:

[INFO] Scanning for projects…
[INFO] 
[INFO] — — — — — — — < com.example:data-datastore-local-testing > — — — — — — — 
[INFO] Building data-datastore-local-testing 0.0.1-SNAPSHOT...[INFO] 
[INFO] — — exec-maven-plugin:3.0.0:exec (start-datastore-emulator) @ data-datastore-local-testing — -
Creating network “data-datastore-local-testing_default” with driver “bridge”
Creating data-datastore-local-testing_datastore_1 … done
[INFO] 
[INFO] — — maven-failsafe-plugin:2.22.2:integration-test (default) @ data-datastore-local-testing — -
[INFO] 
[INFO] — — — — — — — — — — — — — — — — — — — — — — — — — — — -
[INFO] T E S T S
[INFO] — — — — — — — — — — — — — — — — — — — — — — — — — — — -
[INFO] Running com.example.localtesting.repository.ArticleRepositoryIT...[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.849 s — in com.example.localtesting.repository.ArticleRepositoryIT
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] — — exec-maven-plugin:3.0.0:exec (stop-datastore-emulator) @ data-datastore-local-testing — -
Stopping data-datastore-local-testing_datastore_1 … done
Removing data-datastore-local-testing_datastore_1 … done
Removing network data-datastore-local-testing_default...[INFO] — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 
[INFO] BUILD SUCCESS
[INFO] — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 
[INFO] Total time: 18.264 s
[INFO] Finished at: 2021–10–19T16:10:59+02:00
[INFO] — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Done! Thanks for reading this article. Head over to GitHub to get the complete code.

If this article was helpful for your, please consider to buy me a coffee :-)

Monitoring setup with docker-compose - Part 2: Grafana

Mirco — Fri, 21 May 2021 11:03:00 +0000

I assume that you have Prometheus running as described in part 1 of the series.

In short: what is Grafana?

Grafana is a tool to create rich dashboards from your metrics.
Grafana can ingest from many different data sources, including Prometheus.

Configuration

Grafana can work without any configuration files. However, we want to configure Prometheus as a data source, so we create grafana/provisioning/datasources/prometheus_ds.yml.

grafana/provisioning/datasources/prometheus_ds.yml

This configuration file will tell Grafana about Prometheus. You could omit this and add the configuration via the Grafana UI.

Add the following to grafana/provisioning/datasources/prometheus_ds.yml:

datasources:
- name: Prometheus
  access: proxy
  type: prometheus
  url: http://prometheus:9090
  isDefault: true

docker-compose.yml

Next, add the following to docker-compose.yml to add Grafana:

  grafana:
    image: grafana/grafana:7.5.7
    ports:
      - 3000:3000
    restart: unless-stopped
    volumes:
      - ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources
      - grafana-data:/var/lib/grafana

The first volume points to our data source configuration. The second volume is used to save dashboards etc. We need to add the second volume to the volumes section in docker-compose.yml:

volumes:
  prometheus-data:

  grafana-data:

Start

Start Prometheus and Grafana:

docker-compose up -d

and open http://localhost:3000 in your browser. Use user admin with password admin for the first login.

You should see the Grafana Landing Page after login:

Add the first dashboard

Go to http://localhost:3000/dashboard/new to add a new dashboard (or use the plus sign in the navigation bar).

Click on Add an empty panel.

In the dropdown which says -- Grafana -- select Prometheus. You can select Prometheus here because we added the configuration earlier.

Now we need Grafana to tell what to graph. Add the following PromQL statement to the input field metric:

increase(prometheus_http_requests_total[1m])

This will show you how often Prometheus endpoints were called. To learn more about PromQL, see the official documentation.

Click on Apply to save and go back to the dashboard. Finally, click on the dashboard save button in the upper right corner.

Your final dashboard

Great! Next time I will show you how to add AlertManager to the stack.

See the full source code on GitHub

If this article was helpful for your, please consider to buy me a coffee :-)

Monitoring setup with docker-compose - Part 1: Prometheus

Mirco — Fri, 02 Oct 2020 14:11:00 +0000

In this post, you'll learn how to set up Prometheus in a Docker container.

In short: what is Prometheus?

Prometheus is an open-source monitoring application. It scrapes HTTP endpoints to collect metrics exposed in a simple text format.

For example, your web app might expose a metric like



http_server_requests_seconds_count{exception="None", method="GET",outcome="SUCCESS",status="200",uri="/actuator/health"} 435

which means that the endpoint /actuator/health was successfully queried 435 times via a GET request.

Prometheus can also create alerts if a metric exceeds a threshold, e.g. if your endpoint returned more than one-hundred times the status code 500 in the last 5 minutes.

Configuration

To set up Prometheus, we create three files:

prometheus/prometheus.yml - the actual Prometheus configuration
prometheus/alert.yml - alerts you want Prometheus to check
docker-compose.yml

prometheus/prometheus.yml

Add the following to prometheus/prometheus.yml



global:
  scrape_interval: 30s
  scrape_timeout: 10s

rule_files:
  - alert.yml

scrape_configs:
  - job_name: services
    metrics_path: /metrics
    static_configs:
      - targets:
          - 'prometheus:9090'
          - 'idonotexists:564'

scrape_configs tell Prometheus where your applications are. Here we use static_configs hard-code some endpoints.
The first one is Prometheus (this is the service name in the docker-compose.yml) itself, the second one is for demonstration purposes. It is an endpoint that is always down.

rule_files tells Prometheus where to search for the alert rules. We come to this in a moment.

scrape_interval defines how often to check for new metric values.

If a scrape takes longer than scrape_timeout (e.g. slow network), Prometheus will cancel the scrape.

prometheus/alert.yml

This file contains rules which Prometheus evaluates periodically. Insert this into the file:



groups:
  - name: DemoAlerts
    rules:
      - alert: InstanceDown 
        expr: up{job="services"} < 1 
        for: 5m

up is a built-in metric from Prometheus. It returns zero if the services were not reachable in the last scrape.

{job="services"} filters the results of up to contain only metrics with the tag service. This tag is added to our metrics because we defined this as the job name in prometheus.yml

docker-compose.yml

Finally, we want to launch Prometheus. Put this into your docker-compose.yml:



version: '3'

services:
  prometheus:
    image: prom/prometheus:v2.21.0
    ports:
      - 9000:9090
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus-data:/prometheus
    command: --web.enable-lifecycle  --config.file=/etc/prometheus/prometheus.yml


volumes:
  prometheus-data:

The volume ./prometheus:/etc/prometheus mounts our prometheus folder in the right place for the image to pick up our configuration.

prometheus-data:/prometheus is used to store the scraped data so that they are available after a restart.

command: --web.enable-lifecycle --config.file=/etc/prometheus/prometheus.yml is optional. If you use --web.enable-lifecycle you can reload configuration files (e.g. rules) without restarting Prometheus:



curl -X POST http://localhost:9000/-/reload

If you change the command, you override the default of the image and must include --config.file=/etc/prometheus/prometheus.yml.

Start Prometheus.

Finally, start Prometheus with:



docker-compose up -d

and open http://localhost:9000 in your browser.

You'll see Prometheus UI where you can enter some ad-hoc queries on your metrics, like up:

As expected, this tells you that your Prometheus is up, and the other service is not.

If you go to Alerts you'll see that our alert is pending (or already firing):

See the full source code on GitHub

If this article was helpful for your, please consider to buy me a coffee :-)