DEV Community: Cody Meadows

Running Matomo & GlitchTip with Dokku

Cody Meadows — Tue, 07 Nov 2023 20:22:32 +0000

I've been meaning to set up analytics and error reporting for my sites for a while now, and finally got around to it today. I'd heard Dokku was a good way to deploy resources like this, and while it worked pretty well in the end, some of the examples for these apps were quite dated.

Thought I'd share my new instructions on these.

Matomo

https://github.com/rclement/dokku-matomo

I originally found this guide, but some of the instructions are dated and gave me trouble. Below should be a more accurate guide on running Matomo with the current version of Dokku.

Deploy your own instance of Matomo on Dokku.

This setup is makes use of the great ready-to-use matomo-docker image by crazy-max, which does all of the heavy-lifting to properly deploy Matomo without too much headache.

Requirements

App and database

First create a new Dokku app. We will call it matomo.

dokku apps:create matomo

Next create the MariaDB database required by Matomo and link it to the app.

dokku mariadb:create mariadb-matomo
dokku mariadb:link mariadb-matomo matomo

Configuration

Main configuration

dokku config:set --no-restart matomo TZ=America/Chicago
dokku config:set --no-restart matomo MEMORY_LIMIT=256M
dokku config:set --no-restart matomo UPLOAD_MAX_SIZE=16M
dokku config:set --no-restart matomo OPCACHE_MEM_SIZE=128
dokku config:set --no-restart matomo REAL_IP_FROM=0.0.0.0/32
dokku config:set --no-restart matomo REAL_IP_HEADER=X-Forwarded-For
dokku config:set --no-restart matomo LOG_LEVEL=WARN

Persistent storage

You need to mount a volume on your host (the machine running Dokku) to persist all settings that you set in the Matomo interface.

mkdir /var/lib/dokku/data/storage/matomo
# UID:GUID are set to 101. These are the values the nginx image uses,
# that is used by crazymax/matomo
chown 101:101 /var/lib/dokku/data/storage/matomo
dokku storage:mount matomo /var/lib/dokku/data/storage/matomo:/data

Domain setup

To get the routing working, we need to apply a few settings. First we set the domain.

dokku domains:add matomo matomo.example.com

We also need to update the ports set by Dokku.

dokku ports:set matomo http:80:8000
dokku ports:remove matomo http:80:5000

If Dokku proxy:report shows more than one port mapping, remove all port mappings except the added above.

Deploy app for the first time

Deploy Matomo from latest version of the docker image

dokku git:from-image matomo crazymax/matomo:latest

Setup Let's Encrypt

Setup an SSL certificate via Let's Encrypt.

dokku letsencrypt:set matomo email user@domain.com
dokku letsencrypt:enable matomo
dokku letsencrypt:auto-renew matomo

Grep MariaDB information for the setup

We will need to set up Matomo in the web interface and provide the database details. You should be able to access the page via https://matomo.example.com.

Run the command below to retrieve the DSN.

dokku mariadb:info mariadb-matomo

An example DSN might look like this: mysql://mariadb:ffd4fc238ba8adb3@dokku-mariadb-mariadb-matomo:3306/mariadb_matomo. Copy and paste the details as follows:

Hostname: dokku-mariadb-mariadb-matomo
Username: mariadb
Password: ffd4fc238ba8adb3
Database Name: mariadb_matomo

After going through the setup, you should be able to use Matomo.

GlitchTip

I was originally planning to use Sentry, but they've really made it a pain to self-host. They pretty much require you to use docker compose, which Dokku doesn't appear to have solid support for. Or at least, documentation.

I would probably need a dedicated machine or proper Docker environment to run it alongside other applications such as Matomo.
So I looked for lighter weight error reporting applications and came across GlitchTip, which seems to fit the bill.

Didn't see any Dokku instructions for this one, but the following worked for me:

Install Dokku plugins

Go to your dokku server and install following plugins:

Install official postgresql plugin

sudo dokku plugin:install https://github.com/dokku/dokku-postgres.git postgres

Install official redis plugin

sudo dokku plugin:install https://github.com/dokku/dokku-redis.git redis

Prepare dokku

Create dokku app

dokku apps:create glitchtip

Create postgresql db and link it to the app

dokku postgres:create glitchtip
dokku postgres:link glitchtip glitchtip

Create redis instance and link it to the app

dokku redis:create glitchtip
dokku redis:link glitchtip glitchtip

Set required variables for GlitchTip

dokku config:set --no-restart glitchtip SECRET_KEY=$(echo `openssl rand -base64 64` | tr -d ' ')
dokku config:set --no-restart glitchtip GLITCHTIP_DOMAIN=https://glitchtip.example.com
dokku config:set --no-restart glitchtip GLITCHTIP_DOMAIN="smtp://email:password@smtp_url:port"

Domain setup

dokku domains:add glitchtip glitchtip.example.com

We also need to update the ports set by Dokku.

dokku ports:set glitchtip http:80:8080
dokku ports:remove glitchtip http:80:5000

If Dokku proxy:report shows more than one port mapping, remove all port mappings except the added above.

Deploy

Deploy Matomo from their docker image

dokku git:from-image glitchtip glitchtip/glitchtip

Setup Let's Encrypt

Setup an SSL certificate via Let's Encrypt.

dokku letsencrypt:set glitchtip email user@domain.com
dokku letsencrypt:enable glitchtip
dokku letsencrypt:auto-renew glitchtip

You should then be able to access glitchtip.example.com and create the first user account.

Scraping, Visualizing r/tea

Cody Meadows — Sun, 19 Feb 2023 18:48:58 +0000

Reddit Tea Scraper

Occasionally I like to find new online tea vendors to buy tea from, and when I do I would visit r/tea to see what was recommended. However I noticed that there were a bunch of new suggestions scattered all over the place, so I figured I would spend way too much time scraping all of those vendors to see who was most recommended.

Unfortunately, scraping an entire year's worth of data ended up being more difficult than I originally thought, in part due to Reddit's API utilization limits.

Scraping Data

I started out making requests manually to the API endpoint, but because of the way it returns comments, I ended up switching to the PRAW (Python Reddit API Wrapper) package to handle much of the logic. No need to reinvent the wheel when PRAW already handles tasks such as fetching all comments and metadata in a much easier-to-parse manner.

PRAW too ended up having its own limitations though. It respected Reddit's official API limitations of 1,000 submissions and there did not seem to be a good way to filter by date to batch my queries. As a result, I turned to Pushshift - a third-party Reddit API that allows you to query its own database of Reddit submissions and comments.

Unfortunately, it too had some issues - due to a data migration dating back to late last year, its database was missing submissions prior to November. Fortunately, they had their data dumps stored in monthly chunks that you could download from their site.

In the end, I used a combination of:

Pushshift data dumps to get all submissions for 2022
PRAW to get all of their comments

with open('RS_2022-12', "r", encoding="utf8") as f:
    for line in f:
        smb = json.loads(line)
        if smb['subreddit_id'] == "t5_2qq5e" and smb['score'] > 1:
            pprint(smb['permalink'])

            comments = []

            try:
                submission = reddit.submission(smb['id'])

                submission.comments.replace_more(limit=None)
                for comment in submission.comments.list():
                    if comment.score > 0:
                        comments.append(comment.body_html)

I had to download the dumps and process them in batches by month because the uncompressed files hit up ~130GB but allowed me to achieve what I wanted in regard to reliably getting all submissions and their comments for 2022.

I did apply some conditions to filter out low-quality submissions and comments.

Storing Data

While I could have applied my logic to this data while scraping it, I wanted to break up the flow and simply store the data I needed for processing somewhere until I was ready to do something with it. This would also help me make changes to how I extract references to vendors.

I first considered storing it in a CSV or JSON file, but both seemed clunky ways to store data like this. Instead, I choose to write all of this to a Postgres database using the psycopg2 package.

try:
    pg_connect = psycopg2.connect(
        host="xxx",
        database='xxx',
        user='xxx',
        password= os.environ['POSTGRES_PW'])

    print("Database connected successfully")

...

    cur = pg_connect.cursor()
                    try:
                        cur.execute("""
                            INSERT INTO tea (postid, title, url, permalink, selftext, comments, created_utc)
                            VALUES (%s, %s, %s, %s, %s, ARRAY [%s], %s)
                            ON CONFLICT DO NOTHING
                        """,
                        (submission.id, submission.title, submission.url, submission.permalink, submission.selftext_html, comments, datetime.utcfromtimestamp(submission.created_utc).strftime('%Y-%m-%d %H:%M:%S')))
                        pg_connect.commit()
                    except (Exception, psycopg2.DatabaseError) as error:
                        print(error)

...

pg_connect.close

Processing Data

One benefit of using a database like Postgres was that I could query from it instead of having to write a separate Python script to do it for me.

This may not be the 'best' query but it returned to me what I needed. I used UNNEST to separate the array items in the "Comments" array and the REGEXP_MATCHES with a very loose pattern to match the domain.com syntax.

SELECT POSTID,
    PERMALINK,
    REGEXP_MATCHES(UNNEST(COMMENTS),
        '\w{4,}(?<!wordpress|blogspot|amazon|reddit|imgur|youtu.*|seriouseats|aliexpress|etsy|facebook|github|wikipedia|redd)\.(?!png|jpg|html|wordpress|pdf|htm|php|blogspot)[a-z]+',
        'g') AS WEBSITE_DOMAIN
FROM TEA
WHERE permalink !~ 'marketing_monday'
AND permalink !~ 'daily_discussion'
ORDER BY WEBSITE_DOMAIN

Normally I would have liked to count the results via the query as well, but the way Postgres projects the results AS WEBSITE_DOMAIN made it difficult for me to do so with my limited Postgres knowledge. Instead, I exported the results to a CSV and used pivot tables to calculate the results for me.

Visualizing Data

Now that I had what I wanted - what do I do with it? I thought it would be neat to visualize the top 20 results.

To this end, I used the ipyvizzu package in combination with a jupyter notebook to create an animated graph with my results.

I could probably write up an entire separate article on this package as I found the documentation for both it and the underlying Vizzu documentation to be lacking in some aspects. The general synopsis is that it allows you to take your data and animate it.

chart.animate(
    Config(
        {
            "channels": {
                "y": {"set": ["Vendor"],"labels": False},
                "x": {"set": ["Count"],"range": { "max": '100%' },"title": "# of References on r/tea","interlacing": False},
                "label": { "attach": ['Vendor']},
                "color": { "set": ["Vendor"] }
            },
            "reverse": True,
            "title": "Most Popular Online Vendors 2022"
        }
    ),
    Style(
            {
            "plot": {
                "backgroundColor": "#1b2e24"
                }
        }
    ),
    duration=5,
    easing="ease"
)

Was more work than I expected to animate this to my liking, but I'm pretty happy with the end result:

Find the full GitHub Repo here: https://github.com/cmeadowstech/Reddit-Tea-Scraper

Website... Unchained

Cody Meadows — Sat, 07 Jan 2023 16:18:55 +0000

Overview

Overview
- Models and Views
- settings.py
- Cosmos MongoDB
- Azure Storage backend
- Security
- Terraform
- Security
- Pipelines
- Dockerfile

The original version of my website was created for the Cloud Resume Challenge, but as I've progressed after achieving the AZ-104 certification I decided to break away from its mold a bit. I started to get more focused on Python and wanted to redo my site with Django. I also wanted to implement IAC via Terraform and CI/CD via GitHub Actions, to implement things properly.

A little overkill for a single-page site, but I thought it was a fun project and it allows me to better expand in the future.

I delved into Django a bit last summer, and IAC / CI/CD the last several months, but I want to thank the following sources for helping me get through this project:

mdn web docs - Beginner's Guide to Django
simple is better than complex - Another beginner's guide to Django
Terraform: Up and Running - Excellent read on using Terraform
Deploying Terraform at scale with GitHub Actions - A very straight-forward article by Facundo Gauna on using GitHub Actions to deploy Terraform

    ├───app
    │   ├───cmeadows_tech           # Project folder
    │   ├───home                    # Main app folder
    └───infra
        ├───global                  # Global Terraform IAC
        └───web                     # Terraform IAC for running app

Models and Views

Pretty straightforward for a single-page site. At the moment I have one model for my projects, so in the future I can more easily add, remove and modify them. You know, your basic CRUD operations.

class Project(models.Model):
    title = models.CharField(max_length=30, help_text='The title of the project')
    url = models.URLField(help_text='The URL for the project\'s repo or blog post', blank=True)
    tech = models.CharField(max_length=200, help_text='Comma separated list of technologies used by the project')
    synopsis = models.TextField(help_text='Summary of the project')

In the future I would like to add another model for a more extended project description, akin to a blog post, so visitors can read more about my projects without having to follow through to another site.

settings.py

Cosmos MongoDB

To try and save on costs and because I like NoSQL databases, I wanted to put my database up on Cosmos DB. The easiest way to do this was with its MongoDB API and the Djongo backend.

This did come with some difficulties though, as the package dependencies were a mess after updates to Python3 and Django itself.

This open issue has an ongoing conversation on the problems I faced, and is what helped me correct my dependencies.

These packages work together:

asgiref==3.6.0
Django==4.1.4
djongo==1.3.6
dnspython==2.2.1
pymongo==3.12.3
pytz==2022.7           # This  must be installed manually, it is not included in djongo
sqlparse==0.2.4
tzdata==2022.7

Note: Cosmos uses a different port than the default MongoDB port, so you have to specify it in the connection string.

Azure Storage backend

I also wanted to use Azure Storage to serve my static files, which luckily was pretty straightforward to set up with the Azure Storage backend. In the future I might look into setting up a CDN to serve these files as well.

AZURE_ACCOUNT_NAME = "<storage account name>"
AZURE_CONTAINER = 'static'
AZURE_ACCOUNT_KEY = '<storage account key>'
AZURE_OVERWRITE_FILES = 'True'

STATIC_URL = 'static/'

STATIC_DIRS = [
BASE_DIR / "static"
]

Security

Of course, this resulted in a bunch of vulnerabilities if I were to store the connection strings and keys as plaintext in settings.py, so I utilized environment variables, GitHub Secrets, and App Service Settings to secure the important bits.

DJANGO_SECRET_KEY - Used to update SECRET_KEY in production
- Secret = DJANGO_SECRET_KEY
DJANGO_DEBUG - Used to disable debug mode in production
- No Secret
DJONGO_HOST - Used to secure connection string to Cosmos MongoDB
- Secret = DJONGO_HOST        
ACCOUNT_KEY - Used to secure connection key to Azure Storage hosting static files
- Secret = ACCOUNT_KEY

These can be added as Application Settings via the azure/appservice-settings GitHub Action

- uses: azure/appservice-settings@v1
    with:
     app-name: ${{ env.AZURE_WEBAPP_NAME }}
     mask-inputs: false
     app-settings-json: '[
          {
            "name": "ACCOUNT_KEY",
            "value": "${{ secrets.ACCOUNT_KEY }}",
            "slotSetting": false
          },
          {
            "name": "DJANGO_DEBUG",
            "value": "False",
            "slotSetting": false
          },
          {
            "name": "DJANGO_SECRET_KEY",
            "value": "${{ secrets.DJANGO_SECRET_KEY }}",
            "slotSetting": false
          },
          {
            "name": "DJONGO_HOST",
            "value": "${{ secrets.DJONGO_HOST }}",
            "slotSetting": false
          }
        ]'
     general-settings-json: '{"linuxFxVersion": "PYTHON|${{ env.PYTHON_VERSION }}"}'

Terraform

I have two main Terraform configurations where I am utilizing the Azure Backend and a Service Principal for authentication, as mentioned earlier.

global - This is used to define the resource group and storage account used by both the remote state and static files. Using the same storage account for both probably isn't considered a best practice, but I thought it acceptable for a small personal project where everything would likely share the same lifecycle
web - This defines the infrastructure my Django app needs to run. The Cosmos DB, the static container, and the App Service

Security

For authentication, I am using a Service Principal and setting the below environment variables during the workflow run, with their values retrieved from repo secrets.

ARM_CLIENT_ID
ARM_CLIENT_SECRET
ARM_SUBSCRIPTION_ID 
ARM_TENANT_ID

Pipelines

Just two pipelines at the moment

deploy.yml - Not well-named, but is what I used to deploy my web Terraform configuration. I want to thank Facuno Gauna for his wonderful article on deploying Terraform with GitHub Actions
appcontent.yml - Used to deploy my app content to the Azure App Service. Using this sample provided by Microsoft, with very minor modifications.

Dockerfile

I started creating a Dockerfile with the intention of containerizing, which I still might, but haven't completed due to deciding on Azure App Service hosting instead of a container instance. It really only needs environment variables for settings.py at this point, then publishing somewhere.

Cloud Resume Challenge Part #1

Cody Meadows — Mon, 21 Mar 2022 02:00:50 +0000

In my efforts to pick up some skills with Azure, I came across the Cloud Resume Challenge and thought I'd throw myself at it while studying for the AZ-104. It consists of several sections designed to throw you into various skills used when working in the Azure environment - blob storage, CDNs, programming with JavaScript and Python, and automation with Azure Functions and GitHub Actions.

I'm already somewhat familiar with HTML and CSS so while I'm not far into my Azure studies, the first few steps were pretty easy to bang out over the weekend. So here I present to you my new resume website - https://web.cmeadows.tech/

Certification

I actually got the AZ-900 last summer. I know, a while between then in and but I was busy getting accustomed to my new L2 position. (There is no L3 at my company, next stop Microsoft) This was really an introductory certification anyways, not one that had you doing much of anything.

HTML

I've put together a few websites over the years, so a simple static page was pretty quick to get together. It's mostly just some DIVs for each section, and some headers, paragraphs and unordered lists for the contents.

I mean, look at this. Not trying to undersell my work but I know people would scoff at this being called "web development"

    <div class="section">
    <h2 class="head">Certifications & Skills</h2>
        <ul>
            <li><b>Certifications:</b> CompTIA Network+, MS-900: Microsoft 365 Fundamentals, AZ-900: Microsoft Azure Fundamentals, CompTIA A+</li>
            <li><b>Skills:</b> Office 365, Azure, Active Directory, PowerShell, Exchange, DNS, Cloud Security, Remote Access Software, Windows 10, macOS, Android, Microsoft Office</li>
        </ul>
    </div>

CSS

This is also pretty straight-forward. I used a basic reset template to normalize everything and made some minor flavor changes. A background image, centered resume, rounded corners, dashed borders. I might add some animation to it if I get bored.

Static Website

Getting into the fun stuff. I hadn't really done anything with blob storage when studying for the AZ-900. Azure Static Webpages made this real easy though - enable it and upload your files to the pre-configured $web container.

I did create a PowerShell script to upload my updated files though so I don't have to putz around in the portal. It seemed like most people used AzCopy for this, but I opted for Set-AzStorageBlobContent instead. Partly because I'm more comfortable with PowerShell, partly because I was worried how AzCopy integrates with GitHub Actions down the line.

Connect-AzAccount       # If you have multiple directories you will need to specify with -Tenant

# Set container and context below

$localfolder = "C:\your path here"
$storageAccount = Get-AzStorageAccount -ResourceGroupName "Your Resource Group" -Name "Name of storage account"
$Context = $storageAccount.context
$ContainerName = '$web'
$Storage = Get-AzStorageBlob -Context $Context -Container '$web'
$files = Get-ChildItem $localfolder

# Replace files if they exist, and upload them if they don't

foreach ($file in $files) {
    $name = $file.name
    $path = "$($localfolder)\$($name)"
    $blob = Get-AzStorageBlob -Container $ContainerName -Context $Context -Blob $name -ErrorAction:SilentlyContinue
    if ($blob -eq $null) {
        Set-AzStorageBlobContent -Container $ContainerName -Context $Context -File $path -Blob $name -Properties @{"ContentType" = [System.Web.MimeMapping]::GetMimeMapping($path)}    # If the file does not currently exist on the container
    } else {
        $blob | Set-AzStorageBlobContent -File $path -Properties @{"ContentType" = [System.Web.MimeMapping]::GetMimeMapping($path)} -Force    # If the file does currently exist on the container
    }
}

# Purge CDN Endpoint
Get-AzCdnProfile | Get-AzCdnEndpoint | Unpublish-AzCdnEndpointContent -PurgeContent "/*"

HTTPS

Configuring the CDN was easy, but setting up HTTPS was more of a pain than it should be in my opinion. I mean, enabling it for subdomains is a flip of a switch but it seems inordinately difficult to do so for a root domain. The only options are to buy an expensive certification from someone like DigiCert or a complex automated process to obtain and renew certificates from the common Let's Encrypt.

In the future I'll revisit automating Let's Encrypt certificates, but for now I'll opt to use a subdomain for my site.

DNS

Azure DNS zones are pretty easy to set up. I had to set my nameservers in NameCheap, but other than that they have an easy gui to point your records to already existing resources like the CDN endpoint I previously configured.

In summary

At this point my Resource Group contains the following:

Storage account with a Static Web Page enabled
CDN Endpoint
Front Door and CDN profiles
DNS Zone

Hopefully more to add as I start to work through the AZ-104. I still have quite a list to finish up:

Javascript
Database
API
Python
Tests
Infrastructure as Code
Source Control
CI/CD (Back end)
CI/CD (Front end)
Blog post (Well, here's part #1. I probably won't have part #2 until I finish this thing)