DEV Community: Stefan Hudelmaier

Using nmap for Continuous Vulnerability Monitoring

Stefan Hudelmaier — Thu, 04 Apr 2024 16:16:37 +0000

I don't have to tell you how important it is to make sure that your systems are not vulnerable to attacks. There are a number of complementary measures that you should implement. Here is an (incomplete) list:

Keeping your software supply chain secure
Protecting your systems from outside access through Firewalls and VPNs
Making sure that the operation system is always up-to-date in terms of security patches
Continuous scanning for vulnerabilities on the systems
Continuous scanning for vulnerabilities from the outside

In this post, we want to focus on the last item on this list. Using a vulnerability scanning tools to monitor your systems externally.

It is important to not only perform the scan now and then but continuously: New CVEs are discovered all the time. Also, your system might change over time for any number of reasons (updates being applied, new software is installed, etc.)

A very powerful tool for vulnerability scanning is the venerable nmap. As an example, we will use nmap to monitor the SSH daemon on our servers for vulnerabilities.

Take the following command:

nmap -p 22 -sV -oX /tmp/nmap-output.xml --script vulners example.com

-p 22: Only scan port 22 (SSH)
-sV: Detect the version of services based on probing them
-oX /tmp/nmap-output.xml: Output the result of the scan as an XML file
-script vulners: Use the vulners script ((more info)[https://nmap.org/nsedoc/scripts/vulners.html]). This will use the detected version and check the API of https://vulners.com for CVEs.
example.com: The server to check

An example output could look like this:

22/tcp  open  ssh      OpenSSH 8.9p1 Ubuntu 3ubuntu0.6 (Ubuntu Linux; protocol 2.0)
| vulners: 
|   cpe:/a:openbsd:openssh:8.9p1: 
|_      CVE-2023-51767  3.5 https://vulners.com/cve/CVE-2023-51767

nmap has found out that the version of the OpenSSH server is 8.9p1. For this version there is a CVE for this version.

You can of course do this as well for other services that are running, e.g. for web servers running on ports 80 and 443.

Our goal is to not only perform this scan once, but do it continuously. For this we will use Checkson. We will write a script that performs the scan using nmap and report a failure to Checkson if any CVEs are found. As an added bonus, we will create an attachment to the check run with an HTML report of the findings. First, the script. We will use Python for this and invoke nmap via the subprocess module:

from subprocess import check_output, call
import xml.etree.ElementTree as ET
import os
import sys


def main():
    cmd = "nmap -p 22 -sV -oX /tmp/nmap-output.xml --script vulners example.com"
    nmap_output = check_output(cmd, shell=True)
    print("nmap output was: ", nmap_output)


if __name__ == '__main__':
    main()

As you can see it performs the scan using nmap. It saves the output to an XML file: /tmp/nmap-output.xml

The output XML file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE nmaprun>
<?xml-stylesheet href="file:///usr/bin/../share/nmap/nmap.xsl" type="text/xsl"?>
<nmaprun scanner="nmap" args="nmap -p 22 -sV -oX /tmp/nmap-output.xml -&#45;script vulners example.com" start="1712122535" startstr="Wed Apr  3 07:35:35 2024" version="7.80" xmloutputversion="1.04">
  <scaninfo type="connect" protocol="tcp" numservices="1" services="22"/>
  <verbose level="0"/>
  <debugging level="0"/>
  <host starttime="1712122535" endtime="1712122536">
    <status state="up" reason="syn-ack" reason_ttl="0"/>
    <address addr="100.100.100.100" addrtype="ipv4"/>
    <hostnames>
      <hostname name="example.com" type="user"/>
    </hostnames>
    <ports>
      <port protocol="tcp" portid="22">
        <state state="open" reason="syn-ack" reason_ttl="0"/>
        <service name="ssh" product="OpenSSH" version="8.9p1 Ubuntu 3ubuntu0.6" extrainfo="Ubuntu Linux; protocol 2.0" ostype="Linux" method="probed" conf="10">
          <cpe>cpe:/a:openbsd:openssh:8.9p1</cpe>
          <cpe>cpe:/o:linux:linux_kernel</cpe>
        </service>
        <script id="vulners" output="&#xa;  cpe:/a:openbsd:openssh:8.9p1: &#xa;    &#x9;CVE-2023-51767&#x9;3.5&#x9;https://vulners.com/cve/CVE-2023-51767">
          <table key="cpe:/a:openbsd:openssh:8.9p1">
            <table>
              <elem key="id">CVE-2023-51767</elem>
              <elem key="is_exploit">false</elem>
              <elem key="cvss">3.5</elem>
              <elem key="type">cve</elem>
            </table>
          </table>
        </script>
      </port>
    </ports>
    <times srtt="33172" rttvar="25279" to="134288"/>
  </host>
  <runstats>
    <finished time="1712122536" timestr="Wed Apr  3 07:35:36 2024" elapsed="0.95" summary="Nmap done at Wed Apr  3 07:35:36 2024; 1 IP address (1 host up) scanned in 0.95 seconds" exit="success"/>
    <hosts up="1" down="0" total="1"/>
  </runstats>
</nmaprun>

Let's extend the script to parse the CVEs. While we are at it, we will add the possibility to ignore CVEs that we have analyzed, but that we consider harmless, e.g. because there is no way to practically exploit them. We will use an env variable called $IGNORED_CVES.

from subprocess import check_output, call
import xml.etree.ElementTree as ET
import os
import sys


def main():
    cves = []
    ignored_cves = os.environ.get('IGNORED_CVES', '').split(',')

    cmd = "nmap -p 22 -sV -oX /tmp/nmap-output.xml --script vulners example.com"
    nmap_output = check_output(cmd, shell=True)
    print("nmap output was: ", nmap_output)

    xml = ET.parse('/tmp/nmap-output.xml')
    root = xml.getroot()
    xpath = './/host/ports/port'
    for port in root.findall(xpath):
        table = port.find('script').find('table')
        for vulnurability in table.findall('table'):
            vulnurability_id = vulnurability.find('.//elem[@key="id"]').text
            cve_score = vulnurability.find('.//elem[@key="cvss"]').text
            print(vulnurability_id, cve_score)
            if vulnurability_id not in ignored_cves:
                cves.append(vulnurability_id)

    print("Non ignored CVEs: ", cves)
    if len(cves) > 0:
        print("At last one relevant CVE found, exiting with status 1")
        sys.exit(1)

    print("No relevant CVEs found, exiting with status 0")
    sys.exit(0)


if __name__ == '__main__':
    main()

This will exit with exit code 1 if there are non-ignored CVEs. An exit code that is not 0 will tell Checkson that the check should be CRITICAL and notifications should be sent out.

We can execute the script and ignore certain CVEs with this:

export IGNORED_CVES="CVE-2023-51385,CVE-2023-51384"
python3 check.py

Let's add one final thing: A report of the findings in HTML format. For nmap XML output this can be achieved with XSLT for turning XML into HTML. There is a command line tool for that: xsltproc. We will simply invoke that from Python. The file is placed in the $CHECKSON_DIR/attachments directory, so it will be made available by Checkson as an attachment. This attachment will be accessible via the Checkson web app. The complete listing of the Python script is this:

from subprocess import check_output, call
import xml.etree.ElementTree as ET
import os
import sys


def main():
    cves = []
    ignored_cves = os.environ.get('IGNORED_CVES', '').split(',')

    cmd = "nmap -p 22 -sV -oX /tmp/nmap-output.xml --script vulners example.com"
    nmap_output = check_output(cmd, shell=True)
    print("nmap output was: ", nmap_output)

    create_html_report()

    xml = ET.parse('/tmp/nmap-output.xml')
    root = xml.getroot()
    xpath = './/host/ports/port'
    for port in root.findall(xpath):
        table = port.find('script').find('table')
        for vulnurability in table.findall('table'):
            vulnurability_id = vulnurability.find('.//elem[@key="id"]').text
            cve_score = vulnurability.find('.//elem[@key="cvss"]').text
            print(vulnurability_id, cve_score)
            if vulnurability_id not in ignored_cves:
                cves.append(vulnurability_id)

    print("Non ignored CVEs: ", cves)
    if len(cves) > 0:
        print("At last one relevant CVE found, exiting with status 1")
        sys.exit(1)

    print("No relevant CVEs found, exiting with status 0")
    sys.exit(0)


def create_html_report():
    checkson_dir = os.environ.get('CHECKSON_DIR', '/tmp')
    if not os.path.exists(checkson_dir):
        os.makedirs(checkson_dir)
    call(f'xsltproc /tmp/nmap-output.xml -o {checkson_dir}/attachments/nmap-output.html', shell=True)


if __name__ == '__main__':
    main()

In order to be able to deploy this check to Checkson, the only step left to do is to wrap it in a Docker container. The following Dockerfile installs the required pieces of software nmap and xsltproc:

FROM alpine:3
RUN apk add --update nmap nmap-scripts libxslt bash python3
ADD main.py /check.py
CMD [ "python", "/check.py" ]

You can build this Docker image locally and test it:

docker build . -t myuser/sshd-scan:1.0
docker run --rm -it -e IGNORED_CVES="CVE-2023-51385" myuser/sshd-scan:1.0
echo "Exit code was: $?"

Everything is ready for the check to be deployed to Checkson. Please refer to the guide on how to to it using the CLI or to the guide on how to do it via the web UI whichever you prefer. It only takes 5 minutes.

After the check is deployed, you will receive an E-Mail or Slack message when new CVEs for the SSH daemon on your server of servers are discovered.

This was an example of using nmap for vulnurability monitoring. Of course, we only touched the surface. There is a whole category of scripts for nmap for all kinds of vulnerabilities.

On The Importance of End-to-End Monitoring for IoT

Stefan Hudelmaier — Sun, 24 Mar 2024 11:46:05 +0000

The Importance of End-to-End Monitoring for IoT

IoT systems are made up of many building blocks these days. A common setup looks like this: You have an edge gateway communicating via the public Internet or via VPNs to an MQTT broker (the "southbound" interface). The gateways send messages with telemetry data from attached devices and machines. This data gets processed by normalization and transformation services before it is written to a time series database. REST or GraphQL APIs make the data available to a user interface or third-party systems (the "northbound" interface). The fundamental business function of this IoT system is that it can ingest data, process it, store it and make it available. All other business functions (KPI calculation, alert detection, etc) are usually built upon this basic one. It spans the southbound and northbound interfaces and all the steps in between.

You must, at all times, know the status of this functioniality. If it breaks, you must be the first to know, so you can start fixing.

End-to-end monitoring will give you this knowledge and the confidence in your IoT solution that follows from it.

How does an IoT End-to-End Check look like?

An end-to-end check could look like this: Each time it runs, it will send data to your MQTT broker. Often, sending just one datapoint is enough - more can be necessary if you want to also monitor the data processing logic. Your check will then start quering the REST of GraphQL API, until it can read the exact value that it sent previously. The datapoint's value is often a UUID or a timestamp so it can be uniquely identified. If you received the value before your check times out, your check is green, if you don't, or if it took too long, it's red.

This kind of check also gives you a useful metric: The end-to-end latency of data ingestion, processing and storage. You can use this to detect gradual decreases in performance (maybe the ingestion performance of your time-series database gets worse, the more data it stores) or of temporary spikes (if devices send more data during certain times of the day, causing congestions and delays in the data processing). The end-to-end latency is also a good SLA metric for reporting the perfomance to stakeholders.

Taking Action

If the end-to-end check breaks, your on-call DevOps staff must be informed right away, so that they can investigate. It will not tell them what is wrong, but it will tell them that something is wrong. They can now use more fined grained monitoring metrics to pinpoint the cause of the issue. To be clear: End-to-end monitoring isn't a replacement for individual component checks, it’s a complementary tool.

If your time for setting up monitoring checks is limited (as it often is, because features are deemed more important than reliability): Make sure you have at least this end-to-end monitoring check in place. It's a lot of value for a small amount of effort.

Monitoring your Website End to End with Playwright

Stefan Hudelmaier — Fri, 15 Mar 2024 12:30:41 +0000

It is hugely important to your business that the login, signup and payment flows of your website are working as expected. If any of these are broken, it can have a huge impact on your business. You could lose customers, revenue, and reputation. That's why it's important to continuously monitor these flows to make sure they are working as expected. And it's not enough to test the functionalities when you deploy a new version of your website. You need to monitor them all the time to guard against operational problems. Especially today, when systems are made up of a
multitude of services, PaaS and SaaS offerings. End-to-end monitoring checks can serve as smoke tests that detect when any required systems is having problems that break the end to end flow.

In this article, we'll show you how to monitor the login to your website end to end with Playwright, Javascript and Checkson.

What is Playwright?

Playwright is a Node.js library for automating browsers. It allows you to write scripts to interact with web pages, just like a real user would. You can use it to fill out forms, click buttons, and navigate between pages. It is most commonly used for end-to-end testing of the code-base, but it's also a great tool for monitoring continuously.

There are similar tools like Puppeteer, Selenium, and Cypress, but Playwright is probably the most popular one at the moment.

Monitoring the Login Flow

Let's use a dummy signin flow as an example.

First, we need to install Playwright and a browser for Playwright to use. We can do this with the following commands:

mkdir playwright-check
cd playwright-check
npm install playwright
npx playwright install chromium

Then, we can write a script called index.js to automate the login:

const {chromium} = require('playwright');
const {expect} = require('playwright/test');

(async () => {
  const browser = await chromium.launch();

  const page = await browser.newPage();

  try {

    await page.setViewportSize({ width: 1280, height: 1200 });

    await page.goto('https://practicetestautomation.com/practice-test-login/');
    await page.getByLabel('Username').fill('student');
    await page.getByLabel('Password').fill('Password123');
    await page.click('#submit.btn');

    await expect(page.getByRole('heading')).toContainText('Logged In Successfully');

    console.log("Logged In Successfully")
  } catch (error) {
    console.log(error);
  } finally {
    await browser.close();
  }
})();

This opens up the login website and fills out the username and password fields. It fills the username and password fields by finding their label.

It then selects the submit button (using the CSS selector #submit.btn) and clicks it.

Then, it is evaluated if text appears that confirms that the login was successful.

You can run this script with Node JS:

node .

You should see the message "Logged In Successfully" in the console.

Creating a Screenshot

It would be useful to generate a screenshot, if there was an error to help with analysis of the error. We can do this by adding the following line to the catch block:

await page.screenshot({ path: './error-screenshot.jpg' });

In case of an error, a screenshot will automatically be created and stored as a file.

We now want to run this regularly on checkson.io. For this we have to created a Docker image out of our script. This is pretty easy since there are base images available for Playwright that we can use.

Let's create the following Dockerfile:

FROM mcr.microsoft.com/playwright:v1.42.1-jammy

WORKDIR /usr/src/app
COPY package*.json /usr/src/app
RUN npm clean-install

COPY index.js /usr/src/app

CMD [ "node", "index.js" ]

This uses a base image, copies our index.js, package-lock.json and package.json to the image and runs npm clean-install. It executes the script as the CMD of the Docker image.

We can build the Docker image:

docker build -t myuser/end-to-end-playwright-check .

You can run the Docker image with the following command:

docker run --rm -it myuser/end-to-end-playwright-check

Note: It could happen that you see an error message like this:

browserType.launch: Executable doesn't exist at /ms-playwright/chromium-1105/chrome-linux/chrome
╔══════════════════════════════════════════════════════════════════════╗
║ Looks like Playwright Test or Playwright was just updated to 1.42.1. ║
║ Please update docker image as well.                                  ║
║ -  current: mcr.microsoft.com/playwright:v1.40.0-jammy               ║
║ - required: mcr.microsoft.com/playwright:v1.42.1-jammy               ║
║                                                                      ║
║ <3 Playwright Team                                                   ║
╚══════════════════════════════════════════════════════════════════════╝

In that case, please update the line starting with FROM in the Dockerfile to the version indicated in the error message.

When running the Docker image, you should also see the output: "Logged In Successfully".

Creating a Checkson Attachment

We want to make the screenshot available on checkson.io. For this, we have to provide it as a so called attachment. Attachments are arbitrary files that can be added to a run of a Checkson check. More information can be found in the docs.

Creating an attachment is easy: They only have to be written to a special directory: $CHECKSON_DIR/attachments. All files from that directory will be picked up by the Checkson cloud runner when it executes the check.

Let's extend our Playwright check so that the screenshot is provided as an attachment.

const {chromium} = require('playwright');
const {expect} = require('playwright/test');
const {existsSync, mkdirSync} = require('fs');

const { CHECKSON_DIR } = process.env;

(async () => {
  const browser = await chromium.launch();

  // Make sure that the attachments directory exists
  const attachmentsDir = `${CHECKSON_DIR}/attachments`;
  if (!existsSync(attachmentsDir)){
   mkdirSync(attachmentsDir);
  }
  const screenshotFilePath = `${attachmentsDir}/screenshot.jpg`;

  const page = await browser.newPage();

  try {

    await page.setViewportSize({ width: 1280, height: 1200 });

    await page.goto('https://practicetestautomation.com/practice-test-login/');
    await page.getByLabel('Username').fill('student');
    await page.getByLabel('Password').fill('Password123');
    await page.click('#submit.btn');

    await expect(page.getByRole('heading')).toContainText('Logged In Successfully');

    console.log("Logged In Successfully")
  } catch (error) {
    console.log(error);
    await page.screenshot({ path: screenshotFilePath });
  } finally {
    await browser.close();
  }
})();

Deploying the Check to checkson.io

Now, everything is ready for the check to be deployed to Checkson and to configure notifications. It only takes 5 minutes. Here is a guide on how to to it using the CLI and here is a guide on how to do it via the web UI.

After you have done this, you will receive an E-Mail or Slack message once the login to your website fails for any reason.

Using PageSpeed Insights to Improve and Monitor Your Website's SEO

Stefan Hudelmaier — Fri, 11 Aug 2023 18:04:54 +0000

Search engine optimization (SEO) is the process of making your website more visible and relevant to search engines and users. SEO can help you attract more organic traffic, increase conversions, and improve your users' experience. However, SEO is not a one-time task that you can set and forget. It requires constant monitoring and improvement to keep up with
the changing algorithms and user expectations.

One of the most important aspects of SEO is page speed, which is how fast your website loads and responds to user interactions. A slow website will result in lower rankings, higher bounce rates, and lost revenue.

Fortunately, Google provides a tool that can help you measure and optimize your website's page speed (along with other SEO-related best practices): PageSpeed Insights. You can use this web app to analyze your homepage. This is great for optimizing your page, because it gives you actionable suggestion on how to improve your score. When you are satisfied with the results, you should think about monitoring your score to make sure it does not drop again over time due to changes that are made to our website or changing best practices and expectations.

You will be pleased to know that Google also provides an API for PageSpeed Insights, which you can use to automate the monitoring part. In this article, we will show you how to use the PageSpeed Insights API to continuously monitor your website's page speed scores and get notified when it drops below a certain threshold.

What is the PageSpeed Insights API?

The PageSpeed Insights API is a RESTful web service that allows you to analyze the performance of any web page with a simple HTTP request. The API returns data in JSON format containing all the Lighthouse metrics and more. Lighthouse is an open-source tool that audits web pages for performance, accessibility, SEO, best practices, and other criteria. It is the foundation of PageSpeed Insights.

The PageSpeed Insights API can provide you with two types of data: lab data and field data. Lab data is generated by running a simulated page load on a controlled environment using a predefined device and network settings. Lab data can help you identify potential performance issues and
opportunities for improvement. Field data is collected from real users who have visited the page through the Chrome User Experience Report (CrUX). Field data can help you understand how your website performs in the real world and how it affects user satisfaction.

How to Use the PageSpeed Insights API?

To use the PageSpeed Insights API, you need to make an HTTP GET request to the following URL:

https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=YOUR_URL

You need to replace YOUR_URL with the URL of the web page that you want to analyze. For example, if you want to analyze example.com, you can use this URL:

https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=https://example.com

You can also append other parameters to customize your request, such as strategy, category, locale, utm_campaign, utm_source, and key. For a full list of parameters and their descriptions, please refer to the official documentation.

You can make the request using any tool or language that supports HTTP requests, such as curl, Postman, Python, JavaScript, etc. Let's use Python:

import requests
import json

url = "https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=https://example.com"
response = requests.get(url)
data = response.json()
print(json.dumps(data, indent=2))

The PageSpeed Insights API response is a JSON object that contains various properties and values. The response structure may vary depending on the parameters that you use in your request. We will focus on the score property and use that to continuously monitor our website.

How to monitor your website's page speed score?

Let's expand our Python script, read the score property from the response and compare it with a threshold. Let's also use environment variables for the URL and threshold, so we can easily change them without modifying the code.

import requests
import os
import sys

url_to_check = os.environ['URL']
score_threshold = float(os.environ['SCORE_THRESHOLD'])

category = 'performance'
strategy = 'desktop'

if score_threshold < 0 or score_threshold > 1:
    raise ValueError('SCORE_THRESHOLD must be between 0 and 1')

base_url = 'https://www.googleapis.com/pagespeedonline/v5/runPagespeed'
url = f'{base_url}?url={url_to_check}&strategy={strategy}&category={category}'

response = requests.get(url)
response.raise_for_status()
data = response.json()
score = float(data['lighthouseResult']['categories'][category]['score'])

if score < score_threshold:
    print(f'{category} score of {score} is below threshold of {score_threshold}')
    sys.exit(1)

print(f'{category} score of {score} is above threshold of {score_threshold}, everything is fine')
sys.exit(0)

Following UNIX conventions, the script will exit with status code 0 if everything is good and status code 1 if it is not. As you can see, we are using the performance category both for the request and for interpreting the response. There are other useful categories like accessibility, best-practices, and seo. Please refer to the official documentation for a full list and description.

You can run the script locally like this:

export URL=https://example.com
export SCORE_THRESHOLD=0.9
python main.py

To have it run periodically and to get an email (or a Slack message) when the score drops below the threshold, you can use Checkson. Checkson is a service that allows you to run your scripts periodically and get
notified when they notice an issue. It is free for up to 2 checks, so you can use it to monitor your website's page
speed score for free.

First we need to wrap the script in a Docker image and push it to Docker Hub. The Dockerfile should look like this:

FROM python:3.10-slim

COPY . /app
WORKDIR /app
RUN pip install requests

ENTRYPOINT ["python", "app.py"]

Now, build the image and push it to Docker Hub:

docker build -t USERNAME/checkson-pagespeed .
docker push USERNAME/checkson-pagespeed

First, click on the "Create new check" button:

Then give the check a name, e.g. "pagespeed" and configure how often it should run.

Configure the Docker image you pushed to Docker Hub:

Now, set the environment variables (as we did in the local example earlier):

Click on Create to create the check. It will now be run every 15 minutes.

To get notified when the score drops below the threshold, you can add a notification channel of type Email:

Configure the check to use that channel:

You are all done. As soon as the performance score drops below 90%, you will get an email.

Conclusion

The PageSpeed Insights API is a powerful tool that can help you measure and optimize your website's page speed and adherence to best practices. By increasing these metrics, our homepage will be more enjoyable for users and will rank higher in search results. To make sure that your website's score stays high, it makes sense to monitor it continuously,
so you will know when you have to take action again.

Monitor the Existence of Docker Images in a Registry

Stefan Hudelmaier — Sat, 05 Aug 2023 08:02:23 +0000

Docker is everywhere. It's used for deploying cloud services (e.g. Kubernetes, Docker Compose), doing CI/CD (e.g. GitHub, GitLab) or monitoring systems (e.g. Checkson). Sometimes it's a good idea to monitor if a certain Docker image is still available in a registry, especially if the registry has a retention policy that deletes unused or untagged images after a period of time.

In this blog post, we will show you how to verify the existence of Docker images in a registry without pulling them, using the Docker CLI and a simple bash script. We will also show you how to use Checkson, a cloud-based service that runs your script and alerts you when images are missing.

Why do you need to verify the existence of Docker images?

There are several reasons why you may want to verify the existence of Docker images in a registry, such as:

Some Docker registries have a retention policy that limits the storage of Docker images, so they may get deleted after a certain period of time or when they reach a certain quota.
Some Docker images are pulled only infrequently, but are nevertheless important for your applications or workflows. For example, you may have some base images that you use to build other images, or some backup images that you use for disaster recovery.

How to verify the existence of Docker images using the Docker CLI?

One way to verify the existence of Docker images in a registry is to use the Docker CLI. The Docker CLI has a command called docker manifest that allows you to inspect the manifest of an image. The manifest is a JSON document that describes the image layers, configuration, and metadata.

In newer versions of Docker, the docker manifest command is available by default. To use the docker manifest command in older versions, you need to enable experimental features. You can do this by setting the environment variable DOCKER_CLI_EXPERIMENTAL=enabled or by editing the ~/.Docker/config.json file and adding "experimental": "enabled".

You can now use the docker manifest inspect command to check if an image exists in a registry. For example, to check if the image python:3.10 exists in the default Docker Hub registry, you can run:

docker manifest inspect python:3.10

If the image exists, you will see the manifest information printed on the screen. If the image does not exist, you will see an error message like:

no such manifest: docker.io/library/python:3.10

You can also specify a different registry by using the full image name. For example, to check if the
image quay.io/prometheus/node-exporter:v1.6.1 exists on quay.io, you can run:

docker manifest inspect quay.io/prometheus/node-exporter:v1.6.1

Note that if you use a private registry, you may need to authenticate with the registry before using the docker manifest inspect command. You can do this by using the docker login command with your registry credentials.

How to verify the existence of Docker images using a bash script?

Let's put this command in a bash script and check multiple images at once.

You loop over a list of image names, and use the docker manifest inspect command with each image name. Then, you can use the exit code of the command to determine if the image exists or not. The exit code is 0 if the image exists, and non-zero if it does not.

Here is an example of a script that checks if a list of images exists in a registry:

#!/bin/bash

# Define an array of image names
images=(
  ubuntu:latest
  alpine:latest
  busybox:latest
  myimage:latest
)

# Loop over the array
for image in "${images[@]}"; do
  # Check if the image exists using Docker manifest inspect
  docker manifest inspect "$image" > /dev/null 2>&1

  # Get the exit code of the command
  exit_code=$?

  # Print the result based on the exit code
  if [ $exit_code -eq 0 ]; then
    echo "$image exists"
  else
    echo "$image does not exist"
  fi
done

If you run this script, you will see something like:

ubuntu:latest exists
alpine:latest exists
busybox:latest exists
myimage:latest does not exist

How to use Checkson to monitor your Docker images?

If you want to monitor your Docker images and get notified when they are missing, you can use Checkson.

Let's modify the script from before somewhat so that it exits with a non-zero exit code if any of the images does not
exist. Let's also make it possible to specify the images as environment variables, so that the check is more reusable.

#!/bin/bash

# Check if images are configured
if [[ -z "$IMAGES" ]]; then
  echo "Please set the IMAGES environment variable"
  exit 1
fi

# Split images by comma
IFS=',' read -ra IMAGES <<< "$IMAGES"

for image in "${IMAGES[@]}"; do
  docker manifest inspect "$image" > /dev/null 2>&1

  # Get the exit code of the command
  exit_code=$?

  if [ $exit_code -eq 0 ]; then
    echo "$image exists"
  else 
    echo "$image does not exist"
    exit 1
  fi
done

echo "All images are present in the registry"
exit 0

You can now wrap this script in a Docker image using the following Dockerfile:

FROM docker:24-cli

COPY . /app
WORKDIR /app

RUN apk add --no-cache bash

ENTRYPOINT ["bash", "check.sh"]

The image is based on the docker:24-cli image which includes the Docker command line tool. It does not, however, include bash, so you have to install this package using apk add --no-chache bash.

After you have built the Docker image and pushed it to a registry (let's assume you have used the tag someuser/docker-images-present-check:1.0), you can create an automated check using the Checkson CLI:

checkson create image-check \
  --image someuser/docker-images-present-check:1.0 \
  --email me@example.com \
  --env IMAGES=ubuntu:latest,alpine:latest,busybox:latest,myimage:latest \
  --check-interval 120

You will now receive an email if any of the configured images is missing.

Conclusion

In this blog post, we have shown you how to verify the existence of Docker images in a registry without pulling them, using the Docker CLI and a bash script. We have also shown you how to use Checkson to monitor the images continuously.

Note: There is a working example of this check on GitHub.

Ensuring Smooth Redirects with Dockerized Monitoring

Stefan Hudelmaier — Tue, 18 Jul 2023 07:34:26 +0000

Sometimes it becomes necessary to transition to a new domain, e.g. because you rebranded your product. This comes with the crucial task of ensuring a seamless user experience during the migration process. One vital aspect is to set up redirects from the old domain to the new one, directing all visitors to the new website. To ensure the redirect works flawlessly, we can leverage Python, Docker, and a helpful tool called Checkson. In this blog post, we'll walk you through the process of monitoring the redirect using a Dockerized environment.

Python Code for Redirect Monitoring

First, let's take a look at the Python code snippet responsible for monitoring the redirect. This code will be used to check if the redirect itself works correctly and to ensure other important details, for example that the TLS certificate is not expired.

import requests
import sys

EXPECTED_TARGET_URL = 'https://www.newcompanyname.com'

response = requests.get('https://www.oldcompanyname.com', allow_redirects=False)

if response.status_code != 301:
    print('Not redirected')
    sys.exit(1)

target_url = response.headers['Location']
if target_url != EXPECTED_TARGET_URL:
    print(f'Incorrect redirect location: {target_url}')
    sys.exit(1)

print('Redirected successfully')
sys.exit(0)

You can test this Python code locally to ensure it functions as expected:

python3 main.py

You can check the exit code of the last command like this:

echo $?

If the redirect works, this will return a 0 (success!) or 1 (failure).

Dockerizing the Redirect Monitoring

To package our monitoring process into a Docker image and push it to a Docker registry (e.g., Docker Hub), we need to create a Dockerfile to define the necessary configuration and dependencies for our Docker image.

FROM python:3.10-slim

COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt

ENTRYPOINT ["python", "app.py"]

Now, let's build the Docker image using the following bash command:

docker build . -t myuser/redirect-check:1.0.0

For a final check before deployment, run the Docker image locally with the following command:

docker run --rm -ti myuser/redirect-check:1.0.0

Setting up Monitoring with Checkson

Now that we have our Dockerized monitoring image, it's time to set up the redirect check using Checkson. You can install the Checkson command line tool by following the instructions here. As an alternative to the CLI, you can also use the web UI.

After installation of the CLI, you can configure the check via the command line using the following commands:

checkson create redirect-check \
  --docker-image myuser/redirect-check:1.0.0 \
  --email me@example.com \
  --failure-threshold 2 \
  --check-interval 10

Let's break this down:

Creating the Check: We create a new check named redirect-check.
Configuring the Docker Image: The --docker-image flag specifies the Docker image we built earlier to use for monitoring the redirect.
Notification Channel: We implicitly configure an Email notification channel and associate it with the check. This way, if the redirect check fails, notifications will be sent to the designated recipients.
Failure Threshold: The parameter --failure-threshold indicates that there must be two consecutive runs of the check with an exit code other than 0 for the check to be considered unsuccessful and trigger notifications. Using a failure threshold helps reduce false positives, ensuring you receive notifications only when necessary.
Check Interval: The parameter --check-interval instructs Checkson to execute this check every 10 minutes in its cloud environment.

By following these steps and utilizing the power of Checkson, you can seamlessly monitor your redirect during the rebranding process, ensuring a smooth transition for your users.

In conclusion, when making significant changes like rebranding and domain transitions, diligent monitoring becomes essential. Leveraging Python and Docker to create a monitoring solution, along with Checkson, gives you the ability to detect any redirect issues promptly. With this reliable monitoring in place, you can confidently proceed with your rebranding efforts, providing an smooth user experience throughout the process.

Note: A working example of this check can be found on GitHub

Introducing Checkson

Stefan Hudelmaier — Wed, 05 Jul 2023 06:10:37 +0000

Monitoring the systems you have built is a crucial part of the development process. It is important to know when something goes wrong, so you can fix it as soon as possible.

There are a lot of existing tools for monitoring servers. There is also a lot of solutions for monitoring web applications and REST APIs. Existing tools often fall short, however, when the monitored workflows are complex, especially when they involve multiple systems and protocols. Checkson focuses on being extremely flexible to handle these scenarios. The basic steps to setup a monitoring check are:

Write a script in any programming language you like, using any library you need
Run the script locally until it works as expected
Wrap your script in a Docker image and push it to a registry
Configure Checkson so that your script is run regularly in the cloud
Get notified if the script fails

It’s as simple as that.

Let’s look at an example. Imaging that you want to monitor a purchase process. Let’s write a script that

Places an order via a REST API
Validates that the order was processed successfully using another API
Validates that a confirmation email was sent, e.g. using MailTrap
Cancels the order

In Python this could look like this:

import requests
import os

def main():
  try:
    # Place order
    response = requests.post('https://example.com/api/order', json={'product': '123'})
    response.raise_for_status()

    # Validate order
    response = requests.get('https://example.com/admin-api/order/123')
    response.raise_for_status()

    if response.json()['status'] != 'confirmed':
      raise Exception('Order not confirmed')

    # Validate email
    email = get_latest_email_from_mailtrap()
    if 'Thank you for your purchase' not in email.subject or '123' not in email.body:
      raise Exception('Issue with email')

  except:
    os.exit(1)
  finally:
    # Cancel order
    requests.delete('https://example.com/admin-api/order/123')


if __name__ == '__main__':
  main()

Checkson checks often serve as smoke-tests. When they fail, you know that something is wrong and you can investigate further.

As you can see, although a lot of components are checked at the same time, the script can be kept very simple and readable. Also it is very straight-forward to develop as you can simply run it locally until it works as expected. Then you wrap it in a Docker image, push it and hand it over to Checkson so that it is run regularly.

Your Dockerfile could look like this:

FROM python:3.10-slim

COPY . /app
WORKDIR /app
RUN pip install requests

ENTRYPOINT ["python", "app.py"]

You can then build and push the image to a registry, e.g. Docker Hub:

docker build -t someuser/my-checkson-check:1.0 .
docker push someuser/my-checkson-check:1.0

You can either configure the check using the Checkson Web UI or using the Checkson CLI:

checkson create order-check \
  --image someuser/my-checkson-check:1.0 \
  --email someuser@example.com

In one line, this creates a new check that will be run every 10 minutes and sets up a notification channel. When the check fails, you will receive an email.

As you can see, we have modeled a pretty complex workflow using a simple script that is very maintainable. We have also used the Checkson CLI to create a check. In a later blob post we will explore how to build the Docker image and configure the check in a CI/CD pipeline.

You can get more information about Checkson on checkson.io.