DEV Community: Jon Neverland

Pixelfed (beta) with Docker and Traefik

Jon Neverland — Tue, 09 Jul 2019 14:32:19 +0000

Are you sick of data mining and ads? Want to share photos in an Instagram like way with friends and keep control over your data? Pixelfed is an open source alternative that's fairly easy to self host.

This guide will show you an easy way to get an instance up in no time with the help of Traefik and Docker. If you already have a server with Traefik running then continue. Otherwise I recommend you read my previous post and you'll be back here soon.

It's also possible to skip using Traefik and put your own reverse proxy (f.e nginx) in front of Pixelfed. Using https is a must today.

Bear in mind that the current release is v0.9.4, so things are still a work in progress. With that said, it's working fine for me and my group of friends.

Prerequisites

This guide assumes some general knowledge of Linux and that you have a server available with these services installed:

docker
docker-compose
A domain to host your apps on

For cheap and good servers, checkout DigitalOcean or Hetzner. Also I always recommend this guide when setting up a new VPS.

If you want to use Digital Ocean feel free to use my referral link to get $10 for your server 😊

Setup

We start with cloning the repository to its own directory and build the docker image from there. There are pre-built ones, like this, but in this case I recommend building your own.

Let's make a directory for your data to live in, as sudo if needed.

sudo mkdir -p /opt/pixelfed_source
sudo chown $USER:$USER /opt/pixelfed_source

Fetch the repo, checkout the commit of the release then build the image (replace v0.9.4 in the tag with whatever the latest version is for easy rollback) and wait...

git clone git@github.com:pixelfed/pixelfed.git /opt/pixelfed_source
# OR 
git clone https://github.com/pixelfed/pixelfed.git /opt/pixelfed_source
cd /opt/pixelfed_source
git checkout v0.9.4
docker build -t pixelfed:v0.9.4 .

Now we can create a directory for the service.

sudo mkdir -p /opt/pixelfed
sudo chown $USER:$USER /opt/pixelfed
cd /opt/pixelfed

Configuration

Next, let's edit the configuration.

cp ../pixelfed_source/.env.example .env
nano .env

Replace all *_DOMAIN with your hostname and APP_URL=http://localhost with your full hostname including protocol.

We will have Pixelfed generate an APP_KEY for us later, so don't touch that one.

I'm using Postgres, mostly because MySQL was throwing weird errors but also because it's what I generally use, so these need to change.

DB_CONNECTION=pgsql
DB_HOST=db
DB_PORT=5432
DB_DATABASE=pixelfed
DB_USERNAME=pixelfed
DB_PASSWORD=USE_A_PROPER_PASSWORD

Since we are using docker and an internal network for the database and Redis, also change this: REDIS_HOST=redis

If you don't want federation, change the following.

ACTIVITY_PUB=false
REMOTE_FOLLOW=false

You can also edit these:

# For a personal server, I don't really feel this is necessary
ENFORCE_EMAIL_VERIFICATION=false
# I had trouble running Horizon (background jobs) embedded in Docker
HORIZON_EMBED=false

Docker

Next we need that internal network for docker.

docker network create pixelfed

And now we can add a docker-compose file.

touch docker-compose.yml
nano docker-compose.yml

Copy and paste this.

version: '3'
services:

  app:
    image: pixelfed:v0.9.4
    restart: unless-stopped
    # This is for Traefik
    labels:
       - traefik.enable=true
       - traefik.frontend.rule=Host:pixelfed.example.com
       - traefik.port=80
       - traefik.docker.network=web
    env_file:
      - ./.env
    volumes:
      - "app-storage:/var/www/storage"
      - "app-bootstrap:/var/www/bootstrap"
      # We can remove this after generating APP_KEY
      - ./.env:/var/www/.env
    networks:
      - web
      - pixelfed

  db:
    image: postgres:9.6.4
    restart: unless-stopped
    networks:
      - pixelfed
    volumes:
     - db-data:/var/lib/postgresql/data
    environment:
     - POSTGRES_PASSWORD=${DB_PASSWORD}
     - POSTGRES_USER=${DB_USERNAME}

  worker:
    image: pixelfed:v0.9.4
    restart: unless-stopped
    env_file:
      - ./.env
    volumes:
      - "app-storage:/var/www/storage"
      - "app-bootstrap:/var/www/bootstrap"
    networks:
      - web # Required for ActivityPub
      - pixelfed
    command: gosu www-data php artisan horizon

  redis:
    image: redis:5-alpine
    restart: unless-stopped
    volumes:
      - "redis-data:/data"
    networks:
      - pixelfed

# Adjust your volume data in order to store data where you wish
volumes:
  redis-data:
  db-data:
  app-storage:
  app-bootstrap:

networks:
  pixelfed:
    internal: true
  web:
    external: true

☝️ Remember to replace pixelfed.example.com in traefik.frontend.rule with your hostname. Also make sure to add/replace your Traefik network (in my case web) in networks.

If you are not using Traefik, replace the labels section for app with this to be able to reverse proxy.

  ports:
    - "127.0.0.1:8080:80"

Now we should be able to start the services and generate an APP_KEY.

docker-compose up -d
docker-compose exec app php artisan key:generate
# Make sure there's a value
cat .env | grep APP_KEY

And now we can restart, clear the cache and run the migrations on the database.

docker-compose restart app
docker-compose exec app php artisan config:cache
docker-compose exec app php artisan migrate
# Answer yes

That's it 👏 Your Pixelfed instance should be up and running. Now go register a user! You can do that either on the web or through your terminal.

docker-compose exec app php artisan user:create

Site admin

Then you probably want to make yourself an admin of the site...

docker-compose exec app php artisan user:admin [username]

The end

I hope you enjoy your social photo sharing and don't hesitate to drop me a line for help (with this guide, for Pixelfed issues check out their Help center).

Adding webhooks to Matrix

Jon Neverland — Wed, 27 Feb 2019 12:00:34 +0000

Now that you got your homeserver you want some integrations right? Webhooks seems like a great place to start - it's fairly easy to set up and the use cases are virtually unlimited.

I have a couple of scenarios in Huginn for posting tomorrows weather, top voted posts from various subreddits and alerts from Grafana, to mention a few.

In this guide we will use an application service (👈 read this if you haven't) called matrix-appservice-webhooks to be able to add webhooks to a room by just issuing a command. It has a Docker image so it fits right into my standard Traefik setup. If your Synapse isn't set up like mine you'll have to make some adjustments.

Setup

First let's get the files we need for the appservice.

cd /opt/matrix # or wherever your stuff is 
mkdir matrix-appservice-webhooks
cd matrix-appservice-webhooks
curl https://raw.githubusercontent.com/turt2live/matrix-appservice-webhooks/master/config/database.json -o database.json
curl https://raw.githubusercontent.com/turt2live/matrix-appservice-webhooks/master/config/sample.yaml -o config.yaml

Then we replace paths in database.json...

sed -i 's/db\//\/data\//' database.json

... so that it looks like this.

{
  "defaultEnv": {
    "ENV": "NODE_ENV"
  },
  "development": {
    "driver": "sqlite3",
    "filename": "/data/development.db"
  },
  "production": {
    "driver": "sqlite3",
    "filename": "/data/production.db"
  }
}

We edit config.yaml...

nano config.yaml

... and put in the (internal) URL to Synapse - accessible via our Docker network - the public URL for our webhooks and our Matrix domain name.

homeserver:
  # The name here is from docker-compose.yaml
  url: "http://synapse:8008"

  # Domain for you matrix server, with subdomain if needed
  domain: "example.com"

web:
  # This domain will be proxied via Traefik
  hookUrlBase: 'https://webhooks.example.com'
  # Because we're in a container
  bind: '0.0.0.0'

☝️ I only included the settings that needs to be changed for this guide. Edit the others according to comments in the file and remember to replace example.com.

Registration file

Then we need to generate a registration file to add to Synapse via homeserver.yaml.

docker run --rm \
  -v /opt/matrix/matrix-appservice-webhooks:/data \
  turt2live/matrix-appservice-webhooks \
  node index.js -r \
  -f /data/appservice-registration-webhooks.yaml \
  -u "http://webhooks:9000" \
  -c /data/config.yaml

Now you should have a file called appservice-registration-webhooks.yaml in your appservice directory. Copy that file to your Synapse dir (/opt/matrix/synapse in my case).

sudo cp appservice-registration-webhooks.yaml ../synapse

Add/edit the line below in your homeserver.yaml, so that Synapse is aware of the appservice.

app_service_config_files: ['/data/appservice-registration-webhooks.yaml']

docker-compose

Time to add the container to our docker-compose.yaml.

  webhooks:
    image: turt2live/matrix-appservice-webhooks
    restart: unless-stopped
    depends_on:
      - synapse
    networks:
      - web
      - default
    volumes:
      - "/opt/matrix/matrix-appservice-webhooks:/data"
    labels:
      - "traefik.enable=true"
      - "traefik.frontend.rule=Host:webhooks.example.com"
      - "traefik.port=9000"
      - "traefik.docker.network=web"

☝️ If you've followed my previous guide, this should be enough. Remember to replace example.com.

Now we're all set and ready to start the service.

docker-compose up -d webhooks
docker-compose restart synapse

Adding hook to a room

Go to a room where you want to have a webhook, invite the bot user @_webhook:example.com (adjust handle if you changed it in config.yaml), send the message !webhook in the room and you should see something like this 👇

Posting them hooks

Test your webhook with an app like Insomnia / Postman or just use curl.

curl --header "Content-Type: application/json" \
     --data '{
       "text": "Hello world!",
       "format": "plain",
       "displayName": "My Cool Webhook",
       "avatarUrl": "http://i.imgur.com/IDOBtEJ.png"
     }' \
   YOUR_URL

Updating hooks

The first time you post the appservice will invite a virtual user with the name and avatar you set. If you change displayName on succeeding posts another user will join and if you change avatarUrl but not displayName the avatar will update.

That's it! Have fun 🎉

Matrix homeserver (Synapse v1.0.0) with Traefik

Jon Neverland — Wed, 20 Feb 2019 22:21:58 +0000

🚀 Update 🚀

Synapse v1.0.0 is now released! I have upgraded my server without any problems and therefore updated the guide with the latest version.

So our good friends at matrix.org released a v0.99.1.1 of Synapse as a precursor for a much anticipated 1.0 release. A big change in the upcoming release is that federation between servers will now require a proper TLS certificate and the current self signed cert that Synapse provides won't work. Homeserver owners are advised to make that change as soon as possible, as v1 is supposed to be released in March.

I've been running my Synapse behind Traefik for quite some time now but never bothered to replace the certificate for federation, although I have proper TLS on the client side. Once I figured out the different options it was pretty straight forward to find one that worked with Traefik.

In this guide we are going to put both the client side and federation of Synapse on a non-subdomain - ie example.com - and use port 443 for both, so no 8448 for federation. If you have another setup - for example matrix.example.com - it should be possible with some adjustments.

We'll achive this by using both a SRV record and a .well-known/matrix/server file to delegate federation to example.com. Synapse itself will be available on synapse.example.com.

I'm also using postgres as the database for Synapse.

Prerequisites

If you already have a server with Traefik running then continue. Otherwise I recommend you read my previous post and you'll be back here soon.

Setup

Let's make a directory for your data to live in, as sudo if needed.

mkdir /opt/matrix
# If you had to use sudo, set permissons
sudo chown -R $USER:$GROUP /opt/matrix

Then we need an internal docker network for the services to communicate on

docker network create matrix

docker-compose.yml

We'll put our services in a docker-compose file to be able to easily manage things. We'll start with Synapse. I'm using the avhost/docker-matrix image but it should be possible to use the official one too I'm sure.

nano /opt/matrix/docker-compose.yml

version: "2"
services:
  postgres:
    image: postgres:9.6.4
    restart: unless-stopped
    networks:
      - default
    volumes:
     - /opt/matrix/pgdata:/var/lib/postgresql/data
    environment:
     - POSTGRES_PASSWORD=SECRET
     - POSTGRES_USER=synapse
    labels:
      - "traefik.enable=false"

  synapse:
    image: avhost/docker-matrix:v1.0.0
    restart: unless-stopped
    depends_on:
      - postgres
    networks:
      - web
      - default
    ports:
     # Coturn
     - "3478:3478"
     - "5349:5349"
    volumes:
     - /opt/matrix/synapse:/data
    labels:
      - "traefik.backend=synapse"
      - "traefik.enable=true"
      - "traefik.frontend=true"
      - "traefik.port=8008"
      - "traefik.frontend.rule=Host:synapse.example.com"
      - "traefik.docker.network=web"

networks:
  default:
    external:
      name: matrix
  # Traefiks network
  web:
    external: true

☝️ Remember to replace example.com in traefik.frontend.rule. That is where we do the reverse proxy for servers. Also change the POSTGRES_PASSWORD env.

Nginx

To be able to serve the .well-known path on our domain we need Nginx. We're also going to use nginx to proxy clients to Synapse. With this setup you have the options to also serve static files on your domain, like example.com/some-site.

Let's make a directory for nginx with a subdirectory for www.

mkdir -p /opt/matrix/nginx/www

Then we need a .well-known directory to put files in, for both our server and for clients.

mkdir -p /opt/matrix/nginx/www/.well-known/matrix

We start with the server file. This will take care of proxying federation to your subdomain so you'll be able to have usernames in Matrix like @me:example.com, while actually hosting Synapse at synapse.example.com.

nano /opt/matrix/nginx/www/.well-known/matrix/server

Here you place a JSON formatted file.

{
  "m.server": "synapse.example.com:443"
}

☝️ Replace example.com above and make sure to NOT include https or anything like that.

Next up is a client file, which makes it possible to sign in to your Riot with custom domain configured by only using your username and not full MXID (@username:example.com).

nano /opt/matrix/nginx/www/.well-known/matrix/client

Another JSON formatted file.

{
  "m.homeserver": {
    "base_url": "https://example.com"
  }
}

☝️ Here you include the full URL. You can also add m.identity_server if you have your own.

Now we need a config file for nginx!

nano /opt/matrix/nginx/matrix.conf

server {
  listen         80 default_server;
  server_name    example.com;

 # Traefik -> nginx -> synapse
 location /_matrix {
    proxy_pass http://synapse:8008;
    proxy_set_header X-Forwarded-For $remote_addr;
    client_max_body_size 128m;
  }

  location /.well-known/matrix/ {
    root /var/www/;
    default_type application/json;
    add_header Access-Control-Allow-Origin  *;
  }
}

Now we can add nginx to our docker-compose.yml.

version: "2"
services:
  postgres:
    # ...
  synapse:
    # ...
  # Add this
  nginx:
    image: nginx:1.12-alpine
    restart: unless-stopped
    networks:
      - web
      - default
    labels:
      - "traefik.enable=true"
      - "traefik.frontend.rule=Host:example.com"
      - "traefik.frontend.passHostHeader=true"
      - "traefik.port=80"
      - "traefik.docker.network=web"
    volumes:
      - ./nginx/matrix.conf:/etc/nginx/conf.d/matrix.conf
      - ./nginx/www:/var/www/

networks:
    # ...

Synapse config

Now we are ready to configure our Synapse so we can get it up and running.

This will create a homeserver.yaml in our synapse directory.

docker run -v /opt/matrix/synapse:/data --rm \
  -e SERVER_NAME=example.com \
  -e REPORT_STATS=yes \
  avhost/docker-matrix:v0.99.1.1 \
  generate

☝️ Replace example.com with your "short" domain, no subdomains.

We then need to edit some stuff.

sudo nano /opt/matrix/synapse/homeserver.yaml

This is not a complete configuration file, just the parts that needs editing.

## Server ##

server_name: "example.com"

☝️ Since we are using a proxy for federation we only use the "short" domain here, without the subdomain (synapse.example.com).

listeners:
  - port: 8008
    tls: false
    # Since it's running in a container we need to listen to 0.0.0.0
    # The port is only exposed on the host and put behind reverse proxy
    bind_addresses: ['0.0.0.0']
    # Previous
    # bind_addresses: ['::1', '127.0.0.1']
    type: http
    x_forwarded: true

    resources:
      - names: [client, federation]
        compress: false

We change from sqlite to postgres.

# Database configuration
database:
  name: psycopg2
  args:
    user: synapse
    # From your docker-compose.yml
    password: YOUR_PASSWORD_HERE
    database: synapse

    # This hostname is accessible through the docker network and is set 
    # by docker-compose. If you change the name of the service it will be different
    host: postgres

We'll enable registration to be able to test. You can change this later on.

# Enable registration for new users.
enable_registration: True

That should be it. I do recommend reading through the config and perhaps another guide to checkout all the options but for now let's try and start up our services...

cd /opt/matrix
docker-compose up -d
docker-compose ps

... and you should get something like this.

      Name                     Command              State                            Ports
--------------------------------------------------------------------------------------------------------------------
matrix_nginx_1      nginx -g daemon off;            Up      80/tcp
matrix_postgres_1   docker-entrypoint.sh postgres   Up      5432/tcp
matrix_synapse_1    /start.sh autostart             Up      0.0.0.0:3478->3478/tcp, 0.0.0.0:5349->5349/tcp, 8448/tcp

If the services are not up have a look at the logs with docker-compose logs (start with docker-compose logs synapse if the wall of text is too much).

Register account and login

If everything looks good, you can now use Riot to create an account and login. You need to select "Advanced" and then change Homeserver URL to your own.

Final notes

SRV Record

You should also add a SRV record to support older homeservers.

_matrix._tcp.example.com. IN SRV 10 5 443 synapse.example.com.

Troubleshooting

If it seems like people on other homeservers can't see your messages, or the other way around, your federation might be broken. Matrid Federation Tester is a good first place to check. If you want to see the actual data from the tester, use this link instead but replace the hostname.

Hosting Riot

If you want your own Riot you can use this guide I've written. It should also possible be to add Riot to the setup above with some modifications to your docker-compose.yml (add riot.example.com to traefik.frontend.rule) and your nginx.conf (add a new location and directory to nginx/www/).

Thanks!

If you have any questions or feedback please comment below or hit me up on Twitter 🙌

Element (Riot) web for Matrix with Docker and Traefik

Jon Neverland — Sat, 12 Jan 2019 18:24:10 +0000

So Element ~~Riot~~ is the "glossy" web client for Matrix that I prefer to use, at least when not using the desktop one. I've previously written a guide on how to set up your own homeserver and now I thought I'd share how I deploy my Element instance.

Prerequisites

As always this guide assumes some general knowledge of Linux and also that you have a server available with these services installed:

docker
docker-compose
Traefik
A domain to host your Element on*

⚠️ The domain needs to be different than your homeserver domain. Read more here. Using a subdomain should be fine, ie if your homeserver is at example.com you could use element.example.com.

If you don't want to use Traefik you could use any other reverse proxy to forward the traffic to the docker container. Be sure to add SSL/TLS to that proxy with for example Let's Encrypt!

If you're setting up a new VPS feel free to use my referral link at Digital Ocean to get $10 for your server 😊

Setup

We'll create a directory for Element to live in and some supporting files.

mkdir -p /opt/element 

# If you had to use sudo above, set the permissions to your user to make life easier
sudo chmod -R $USER:$USER -R /opt/element

mkdir /opt/element/config
mkdir /opt/element/versions
touch /opt/element/docker-compose.yml

docker-compose, with Traefik

Let's start with our docker-compose. Scroll down a bit for an example not using Traefik.

nano /opt/element/docker-compose.yml


version: '2'

services:
  web:
    image: nginx:1.12-alpine
    container_name: element
    restart: unless-stopped
    networks:
      - web
    labels:
      - "traefik.enable=true"
      - "traefik.frontend.rule=Host:element.example.com"
      - "traefik.port=80"
      - "traefik.docker.network=web"
    volumes:
      - ./element-web:/usr/share/nginx/html/
      - ./config/config.json:/usr/share/nginx/html/config.json

networks:
  web:
    external: true

☝️ Remember to replace element.example.com in traefik.frontend.rule and change the name of the external network if needed.

No Traefik

Use or modify this if using another reverse proxy.

version: '2'

services:
  web:
    image: nginx:1.12-alpine
    container_name: element
    restart: unless-stopped
    ports:
      - "127.0.0.1:8080:80"
    volumes:
      - ./element-web:/usr/share/nginx/html/
      - ./config/config.json:/usr/share/nginx/html/config.json

Fetch latest Element version

Now we fetch the tarball of the lastest Element release, extract it and put it in the versions folder. Then we create symlink to be able to update the version in an easy way.

wget -qO- https://github.com/vector-im/element-web/releases/download/v0.17.8/riot-v0.17.8.tar.gz | tar xvz -C /opt/element/versions
ln -s /opt/element/versions/riot-v0.17.8 /opt/element/element-web

☝️ Be sure to check the releases page and update the commands above accordingly if there is a newer release!

Config

Copy the sample config and edit it to your needs.

cp element-web/riot-v0.17.8/config.sample.json /opt/element/config/config.json
nano /opt/element/config/config.json

I only changed these values:

{
    "default_hs_url": "https://matrix.example.com",
    "brand": "Custom Brand",
    "roomDirectory": {
        "servers": [
            "matrix.example.com",
            "matrix.org"
        ]
    }
}

Run it!

And we're good to go. Run docker-compose up -d and you should be able to access your domain and login 😃

Updating

To update to a new Element version check the releases page, fetch the new version by modifying these commands. Then link the new release, stop and start your container again.

wget -qO- https://github.com/vector-im/element-web/releases/download/v0.17.8/VERSION.tar.gz | tar xvz -C /opt/element/versions
docker-compose stop
rm element-web
ln -s /opt/element/versions/VERSION /opt/element/element-web
docker-compose up -d

Web client for Gammu SMSD

Jon Neverland — Tue, 01 Jan 2019 16:55:00 +0000

Finally I have released the client I've written for Gammu SMSD, which I use on my Raspberry Pi Zero SMS gateway. It's a React app with a Node Express backend to fetch the stored messages from a Postgres database. While still a WIP (needs auth and some UI love) it works fine for receiving and sending/replying to SMS.

💻👉 The source code is here with instructions for the different ways of deploying.

I myself run it directly on my Raspberry Pi with Docker but it's also possible to have it connect to an existing Postgres instance in case you already have that set up. Or just run it on a separate machine, preferably with docker-compose and have your gammu-smsd connect to the database container.

I learnt quite a lot building this, both about some Gammu internals and React/Node. My first try was to build the backend in Crystal because I figured having a small binary to ship would be perfect for a RPi but that turned out being harder then expected. That might be content for another blog post...

It also turned out building Docker images on a RPi was messy which led me to the discovery I wrote about in my last post. I included a version of the script from my example in this repository too, to make it easy for others to build and deploy the armv6 version of the Docker image.

Anyways I hope someone finds this useful and please drop me a line with any feedback you might have!

Building Node Docker images for Raspberry Pi Zero

Jon Neverland — Sun, 02 Dec 2018 16:00:00 +0000

While developing a web app for my Raspberry Pi Zero SMS gateway I ran into trouble. I wanted to be able to run the app in Docker but building an image from a Dockerfile on the RPi0 was incredibly slow and I didn't manage to complete it because my Pi ran out of memory.

I thought about creating the image locally and pushing it to Docker Hub but it felt a bit overkill for small, personal apps. After some thinking and googling I realised that you can save docker images and then load them on another computer.

There are 2 ways to go about this:

Install npm on your computer, fetch all dependencies (node_modules) and copy them together with the code into the armv6 image.
Run npm in a container with a multistage build and then copy the files to image.

With local NPM

This is more useful and faster with an app you're developing yourself, since you probably have npm installed.

Fetch the deps as usual with npm install then add a Dockerfile (and possibly a .dockerignore).

# Check for your version: https://hub.docker.com/r/arm32v6/node/tags/
FROM arm32v6/node:8.14.0-alpine
RUN mkdir /app
WORKDIR /app
ENV PATH /app/node_modules/.bin:$PATH

# Either be specific with what to add, docker caches every step
ADD package.json /app/package.json
ADD package-lock.json /app/package-lock.json
ADD node_modules /app/node_modules
ADD app.js /app/app.js

# Or add the whole dir, I always use a .dockerignore file when doing that
ADD . /app

ENV PORT=5000
EXPOSE 5000
CMD ["npm", "start"]

Multistage build

This is useful when you want to make an image of someone elses project and/or don't want to install npm on your machine.

Since you can't run the arm32v6/node on your computer, you need to first use a working node image to fetch/build your project.

# Fetch node_modules for backend, nothing here except 
# the node_modules dir ends up in the final image
FROM node:8.14.0-alpine as builder
RUN mkdir /app
WORKDIR /app
ENV PATH /app/node_modules/.bin:$PATH
COPY package.json /app/package.json
RUN npm install

# Add the files to arm image
FROM arm32v6/node:8.14.0-alpine
RUN mkdir /app
WORKDIR /app
ENV PATH /app/node_modules/.bin:$PATH

# Same as earlier, be specific or copy everything
ADD package.json /app/package.json
ADD package-lock.json /app/package-lock.json
ADD . /app

COPY --from=builder /app/node_modules /app/node_modules

ENV PORT=5000
EXPOSE 5000
CMD ["npm", "start"]

Building

Then you can build and save your image...

docker build -t my-app -f Dockerfile .
docker save -o my-app.tar my-app
ls
# my-app.tar

... transfer it to your RPi (with scp or whatever) and load it!

# On RPi
docker load -i my-app.tar
docker run --rm -p 5000:5000 my-app

Automation

Doing this can get tedious if you make a lot of changes to your app, so I created a simple bash script that you can use: https://github.com/joenas/node-docker-raspberry-zero

Note : In my repo I've named the file Dockerfile.armv6 because I already have a Dockerfile for running the app on other architectures that doesn't require a specific docker image. I also tag the image with the suffix :armv6. The commands above would then be like this:

# Build & save
docker build -t my-app:armv6 -f Dockerfile.armv6 .
docker save -o my-app.tar my-app:armv6

# Load & run
docker load -i my-app.tar
docker run --rm -p 5000:5000 my-app:armv6

Sample .dockerignore

Here's a sample .dockerignore file you can use to not include all the files in your image.

/.git/
.gitignore

docker-compose.yml
Dockerfile*
LICENSE
README.md
*.tar

Sharing is caring

I hope you have any use for this and I'd love to see more stuff for the RPi0/1 so please comment below if you create something! 🙏

Easy InfluxDB & Grafana setup with Traefik

Jon Neverland — Wed, 15 Aug 2018 20:42:03 +0000

What is more exciting than staring at graphs? Nothing right. If you just love some system metrics or want to display the temperature in your home over time, a time series database like InfluxDB combined with Grafana is the way to go!

This guide will show you an easy way to get both of these systems up in no time with the help of Traefik and Docker. If you already have a server with Traefik running then continue. Otherwise I recommend you read my previous post and you'll be back here soon.

Setup

Let's make a directory for your data to live in, as sudo if needed

sudo su
mkdir /opt/influx

Then we need an internal docker network for the services to communicate on

docker network create influx

docker-compose.yml

We'll put our services in a docker-compose file to be able to easily manage things. We'll put the basics in first.

nano /opt/influx/docker-compose.yml

version: "2"
services:
  influxdb:
    # We'll add stuff here
  grafana:
    # and here

networks:
  default:
    external:
      name: influx # The network we created
  web:
    external: true # For traefik

InfluxDB

There are a number of environment variables available for the image of which we'll use a few.

If you want to send data to Influx from the outside world you need to have a frontend in Traefik with the port and hostname, if not then remove the labels section. You can also remove - web under networks.

We also need a user with write access and a database to write to. For some nice system metrics I use telegraf so here we'll have the InfluxDB image create that database and user initially. If you want to use something else as input just change the values accordingly.

influxdb:
    image: influxdb:1.5.4-alpine
    restart: unless-stopped
    ports:
      - "127.0.0.1:8086:8086"
    volumes:
      - ./influxdb:/var/lib/influxdb
    environment:
      - INFLUXDB_DB=telegraf 
      - INFLUXDB_USER=telegraf
      - INFLUXDB_USER_PASSWORD=secretpassword
    labels:
      - "traefik.enable=true"
      - "traefik.frontend.rule=Host:influxdb.example.com"
      - "traefik.port=8086"
      - "traefik.docker.network=web"
    networks:
      - web
      - default # The external "influx" network

☝️ Remember to replace example.com in traefik.frontend.rule.

Grafana

Now let's add the Grafana service so that we have something to display our future stats with. First let's add a volume for persistent storage.

docker volume create grafana-storage

Then add this to docker-compose.yml between influxdb and networks.

grafana:
    image: grafana/grafana:4.4.1
    restart: unless-stopped
    ports:
      - "127.0.0.1:3000:3000"
    volumes:
     - grafana-storage:/var/lib/grafana
     - ./log/grafana:/var/log/grafana
    environment:
     - GF_SECURITY_ADMIN_PASSWORD=supersecret
     # Comment out if you have users that should sign up
     - GF_USERS_ALLOW_SIGN_UP=false 
     - GF_SERVER_ROOT_URL=https://grafana.example.com
    labels:
      - "traefik.enable=true"
      - "traefik.frontend.rule=Host:grafana.example.com"
      - "traefik.port=3000"
      - "traefik.docker.network=web"
    networks:
      - web
      - default

volumes:
  grafana-storage:
    external: true

☝️ Remember to replace grafana.example.com in the 2 places above.

Grafana UI

Now you should be able to run the services!

# To start
docker-compose up -d
# Check logs
docker-compose logs
# Check processes
docker-compose ps

If everything looks alright, open up your Grafana URL and you should be greeted with a login screen. Use admin and the password you set as GF_SECURITY_ADMIN_PASSWORD. After logging in click "Add data source" and fill out the fields like this.

The hostname influx_influxdb_1 in the Url setting is from the docker network influx, since the services are in separate containers you can't use localhost here.

If you're getting error messages when saving/testing the data source and using Safari , try another browser.

Getting some data in there

If you already have a way to send data to Influx, then you're done! If not then previously mentioned telegraf is just as easy to set up.

mkdir /opt/telegraf
cd /opt/telegraf
nano docker-compose.yml

version: "2"
services:
  telegraf:
    image: telegraf:1.5.2
    restart: unless-stopped
    volumes:
      - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
      # For docker stats
      - /var/run/docker.sock:/var/run/docker.sock:ro
    labels:
      - "traefik.enable=false"
    networks:
      - default

networks:
  default:
    external:
      name: influx

nano telegraf.conf

Here's a very simple conf just to get up and running. Feel free to use the default one for more options.

# Configuration for telegraf agent
[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  debug = false
  quiet = false
  logfile = ""
  hostname = "MYHOST"

###############################################################################
# OUTPUT PLUGINS #
###############################################################################

# Configuration for influxdb server to send metrics to
[[outputs.influxdb]]
  urls = ["http://influxdb:8086"] # required
  database = "telegraf" # required
  username = "telegraf"
  password = "secretpassword"
  retention_policy = ""
  write_consistency = "any"
  timeout = "5s"

###############################################################################
# INPUT PLUGINS #
###############################################################################

# Read metrics about cpu usage
[[inputs.cpu]]
  ## Whether to report per-cpu stats or not
  percpu = true
  ## Whether to report total system cpu stats or not
  totalcpu = true
  ## If true, collect raw CPU time metrics.
  collect_cpu_time = false

# Read metrics about disk usage by mount point
[[inputs.disk]]
  ## Ignore some mountpoints by filesystem type. For example (dev)tmpfs (usually
  ## present on /run, /var/run, /dev/shm or /dev).
  ignore_fs = ["tmpfs", "devtmpfs"]

# Read metrics about disk IO by device
[[inputs.diskio]]
  ## Setting devices will restrict the stats to the specified devices.
  # devices = ["sda", "sdb"]

# Get kernel statistics from /proc/stat
[[inputs.kernel]]
  # no configuration

# Read metrics about memory usage
[[inputs.mem]]
  # no configuration

# Get the number of processes and group them by status
[[inputs.processes]]
  # no configuration

# Read metrics about swap memory usage
[[inputs.swap]]
  # no configuration

# Read metrics about system load & uptime
[[inputs.system]]
  # no configuration

# # Read metrics about docker containers
[[inputs.docker]]
  endpoint = "unix:///var/run/docker.sock"
  timeout = "5s"

# # Read metrics about network interface usage
[[inputs.net]]
  interfaces = ["eth*"]

Start the service and you'll have some data coming in.

docker-compose up -d
docker-compose logs

Now go back to your Grafana instance and create some dashboards!

☝️ Telegraf sysstats by broferek

The End

If you noticed any factual errors, spelling mistakes (yes), best practises I'm not aware of or run into any trouble please comment below or hit me up on Twitter 🙌

Traefik with Docker and Let's Encrypt

Jon Neverland — Wed, 14 Mar 2018 17:25:00 +0000

I run most of my services in Docker and previously I was using nginx as a reverse and TLS termination proxy together with Let's Encrypt. That worked great but everytime I wanted to try something new I had to copy-paste another conf and change a few values. I probably could have automated that to some extent and there are others who have but with my recent migration to VPSes I figured I'd give Traefik a try, if nothing else for their awesome logo!

Here I'll show you how to set up Traefik with GUI, http redirection and automatic Let's Encrypt certificates. We'll also add basic auth to the Traefik GUI.

Prerequisites

This guide assumes some general knowledge of Linux and that you have a server available with these services installed:

docker
docker-compose
A domain to host your apps on

For cheap and good servers, checkout DigitalOcean or Hetzner. Also I always recommend this guide when setting up a new VPS.

If you want to use Digital Ocean feel free to use my referral link to get $10 for your server 😊

Setup

This is based on the official guide but with a few additions. I will show you how to add the web dashboard and API - protected by Basic Auth - mostly because it's fun. If you have no use for it or believe it to be unsafe, you can skip that part.

First start with creating a network for your web-facing containers to connect to.

docker network create web

Then we create a directory and the necessary files, as sudo if needed.

sudo su
mkdir -p /opt/traefik
touch /opt/traefik/docker-compose.yml
touch /opt/traefik/acme.json && chmod 600 /opt/traefik/acme.json
touch /opt/traefik/traefik.toml

Now let's add our docker-compose...

nano /opt/traefik/docker-compose.yml

version: '2'

services:
  proxy:
    image: traefik:v1.7.12-alpine
    command: --configFile=/traefik.toml
    restart: unless-stopped
    networks:
      - web
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /opt/traefik/traefik.toml:/traefik.toml
      - /opt/traefik/acme.json:/acme.json
    # REMOVE this section if you don't want the dashboard/API
    labels:
      - "traefik.enable=true"
      - "traefik.frontend.rule=Host:example.com"
      - "traefik.port=8080"

networks:
  web:
    external: true

☝️ Remember to replace example.com in traefik.frontend.rule if you keep the API.

... and the config for Traefik...

nano /opt/traefik/traefik.toml

# Change this if needed
logLevel = "ERROR"
defaultEntryPoints = ["https","http"]

[entryPoints]
  [entryPoints.http]
    address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
    address = ":443"
  [entryPoints.https.tls]

# REMOVE this section if you don't want the dashboard/API
[api]
entryPoint = "api"
dashboard = true

[retry]

[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "mydomain"
watch = true
# I prefer to expose my containers explicitly
exposedbydefault = false

[acme]
email = "myemail"
storage = "acme.json"
entryPoint = "https"
OnHostRule = true
[acme.httpChallenge]
entryPoint = "http"

☝️ Add your domain and email under [docker] and [acme], respectively.

... and we should be good to go!

cd /opt/traefik/
docker-compose up -d

Check the logs (docker-compose logs) and head to your configured domain and you should see something like this (screenshot was taken a few versions back, it's been redesigned).

Basic auth

Since we have exposed the API of Traefik we'd like to have some authentication. Basic auth is supported so let's add that. Run this for the username you want - for example admin - and enter your password.

sudo apt install apache2-utils
htpasswd -n username

Here's what I got for admin/admin.

admin:$apr1$IBj9Hfsd$kf7vXLpY4/9XD365jcp/n1

Now that needs to go in the traefik.toml but to work any $ signs have to be escaped with another $.
Add this to the [entrypoints]-section...

  [entryPoints.api]
    address = ":8080"
    [entryPoints.api.auth]
     [entryPoints.api.auth.basic]
       users = [
         "admin:$$apr1$$IBj9Hfsd$$kf7vXLpY4/9XD365jcp/n1"
       ]

Now stop and rebuild your service...

docker-compose stop
docker-compose up -d

...and you should have basic auth!

Add a container

Now of course to have any use for this we need a container! Why not a blog with ghost?

Create a directory and a docker-compose.yml, remember to change the domain and add the hostname to your DNS! 👍

mkdir -p /opt/ghost
nano /opt/ghost/docker-compose.yml

version: '2'
services:
  server:
    image: ghost:alpine
    container_name: ghost
    restart: unless-stopped
    networks:
      - web
    labels:
      # This one is important since we default to not expose
      - "traefik.enable=true"
      - "traefik.frontend.rule=Host:blog.example.com"
      - "traefik.port=2368"
      - "traefik.docker.network=web"
    volumes:
      - /opt/ghost/blog:/var/lib/ghost/content
    environment:
      - NODE_ENV=production
      - url=https://blog.example.com

networks:
  web:
    external: true

Run the usual docker-compose up -d and voilà blog up with SSL/TLS and all, pure magic 😀

Updates

2019-07-15: Updated to newest version (v1.7.12) and also changed to only using traefik.toml instead of mixing with a command in docker-compose.yml

High memory usage in Influxdb

Jon Neverland — Sun, 11 Mar 2018 17:14:46 +0000

I recently moved my various services from a bare metal server at Hetzner to their new cloud and a couple of different VPSes, which saves me quite a few dollars (kronor actually but...) a month. Most of my services run in Docker and I've got telegraf running on all servers to monitor my stuff. Mostly because I enjoy looking at all them dashboards in grafana :)

After my migration one of the servers started reporting very high memory usage and it turned out to be the InfluxDB container growing more or less indefinitely. I didn't have that problem before, not that I was aware of atleast, but then again I had 32GB of memory on the old server and wasn't really paying as much attention to the resource consumption of each service.

Anywho, after some research I found this article explaining the problem. The takeaway is that Influx creates an in-memory index with all tags and that quickly grows with a lot of data.

So how to fix? In my case I only have two services I've written myself sending data to Influx and neither were using tags excessively so I started examining the telegraf plugins I had enabled. I'm not 100% certain but I'm pretty sure the culprit was the docker plugin. That plugin paired with my dokku host which creates new containers in abundance with each deploy of a service made up for quite a lot of data. Fortunally the plugin has a setting container_name_include = [] that accepts globs, so limiting the amount of containers reported made a big difference. Especially since the docker telegraf data has a lot of tags in it.

I also set a new retention policy on my database like so:

CREATE RETENTION POLICY one_month ON telegraf DURATION 4w REPLICATION 1 DEFAULT;

and deleted a lot of old data, for example from the docker plugin with a simple DROP SERIES docker (etc, there are about 4 series).

The result? Pretty good.

Raspberry Pi Zero as SMS gateway

Jon Neverland — Tue, 23 Jan 2018 16:00:00 +0000

I've previously written about setting up a SMS gateway using a 3G modem. I had that plugged into my HTPC and honestly it just didn't look good next to the TV. I figured using a Raspberry Pi and just stash it away somewhere would be better. At the same time I was preparing to create a Magic Mirror of sorts (without the actual mirror) and I accidentally (?) stumbled across the Adafruit Fona 808 when doing my regular window shopping for Pi stuff.

After some reading up - there are a lot of great tutorials I'll link to at the end of the post - I felt that there should be something a bit easier and I found a couple of HATs designed for the Pi.

I ended up buying one from Sixfab but either of these two should work as well. (If you try one of those I'd love to hear about it!)

Assembling

I own a soldering iron but prefer not to use it unless necessary so imagine my joy when I found the solderless GPIO header. While at it I also bought a nice little case. Assembling the header and case was pretty easy and the result great! Just follow this guide.

Zero mounted in the rig

Header attached

Case assembled

and finally GSM pHAT added

Setup

Now to getting the thing to work! There's a blog post at Sixfab, but it's aimed at setting up a GRPS connection. I was just interested in sending and receiving sms with gammu which I already had installed (with postgres as backend this time).

First you need to disable the serial console.

sudo systemctl stop serial-getty@ttyAMA0.service
sudo systemctl disable serial-getty@ttyAMA0.service

That one worked for me. Depending on your Pi you might need to do this instead.

sudo systemctl stop serial-getty@ttyS0.service
sudo systemctl disable serial-getty@ttyS0.service

Then remove the corresponding line in boot commandline

sudo nano /boot/cmdline.txt
# Remove "console=serial0, 115200" or "console=ttyAMA0, 115200"

Enable UART and disable Bluetooth on serial.

sudo nano /boot/config.txt
# Add these
enable_uart=1
dtoverlay=pi3-disable-bt

That last bit with disabling BT did the trick for me and took a reasonable amount of googling to find out. I don't use BT but if you do I recall seeing a solution somewhere...

Connecting

Reboot your Pi and to test the connection to the pHat do this

sudo nano /etc/gammurc
# Add 
[gammu]
device = /dev/serial0
connection = at115200

And gammu should find your device

sudo gammu --identify

You can also interact with the shield via screen

sudo screen /dev/serial0 115200

Type AT, press enter and you should get OK back. Ctrl-A, D to exit.

If everything is ok you should now be able to use gammu-smsd to send and receive SMS. First power up the shield by pressing POWER-UP for a few seconds.

sudo nano /etc/gammu-smsdrc

This is my entire config with postgres and all.

[gammu]
port = /dev/serial0
connection = at115200

# SMSD configuration, see gammu-smsdrc(5)
[smsd]
service = sql
driver = native_pgsql
host = localhost
user = smsd
password = SUPERSECRET
database = smsd
RunOnReceive = python /opt/gammu/receivesms.py

Sending SMS!

sudo service gammu-smsd restart
gammu-smsd-inject TEXT 123456 -unicode -text "Oh hai"

I've also written a small html client with an API backend for gammu that I'll (probably) publish soon.

Reading list

Matrix homeserver with Docker

Jon Neverland — Fri, 15 Sep 2017 14:12:00 +0000

There are many good reasons to switch to Matrix from whatever proprietary monolith chat system you are using today. As others have written about that I'm not gonna delve on those now.

I'm a big Docker fan so when I decided to setup my own Synapse homeserver I was glad to find an image ready to use. I'll take you through setting up Synapse together with Postgres and a Let's Encrypt certificate. With this setup adding additional services like bots or bridges is easy.

Before we start here are some general points about Matrix.

Matrix is the protocol.
There are a couple of servers available but Synapse is AFAIK the only (more or less) production ready. It will eventually be replaced by Dendrite (written in Go).
Clients are plenty so you should be able to find one to your liking. I prefer Riot both for desktop and mobile.
There are also a lot of bridges (called Application Services) to other networks like IRC, Slack and Gitter to name a few.

Prerequisites

This guide assumes some general knowledge of Linux and that you have a server available with these services installed:

docker
docker-compose
Let's Encrypt certbot
nginx
A domain to host your homeserver on

I use Ubuntu 16.04 , most of this should work just fine for other distros but you know, YMMV.

If you're gonna start up a new server - like a VPS - I recommend this guide for some basic security.

Also, I used Digital Ocean to follow my own guide and make sure everything worked. I think they are great and if you haven't tried it already feel free to use this referral link to get $10 for your Matrix server :D

Edit : For those unfamiliar with nginx Martial Lienert suggested Caddy. I haven't tried it myself but for serving a single host like Synapse it looks pretty neat.

Setup

We'll be using docker-compose to be able to easily change options for the containers or adding new services. The default database is sqlite but even with a small homeserver like mine I noticed a big difference in responsiveness with postgres. And if you plan on joining big rooms like #matrix:matrix.org it's a must since Matrix will federate data to all servers with at least one user in a room. So in a room with 10000+ users there will be a lot of writes to the database.

Clicking on the #matrix link above will display a page where you can pick a client and join directly. Although I recommend not creating an account on matrix.org if you plan on running your own homeserver, since migration isn't available at the moment.

It might be a good idea to read through the Synapse README if you haven't already. Note that the parts about which ports to use is kinda confusing. This guide will have everything setup the recommended way but if you're curious about the details you should read this issue.

Base directory

To keep the different services grouped together and for a more manageable docker-compose.yaml we'll create a base directory.

sudo mkdir /opt/matrix

Generating Synapse files

Next we will generate the required files for Synapse. This will add a self-signed certificate used for federation, a homeserver.yaml config file and a log config.

You need to decide on what hostname to use. It's possible to host Synapse on a subdomain (f.e matrix.example.com) and still have clients connect to example.com, but it requires some extra setup. I'd recommend having a dedicated domain.

# This will create /opt/matrix/synapse
docker run -v /opt/matrix/synapse:/data --rm \
    -e SERVER_NAME=example.com -e REPORT_STATS=yes silviof/docker-matrix generate

☝️ Remember to replace example.com in the command.

Edit: I've changed the REPORT_STATS above from no after discussion since I think it's important to support Matrix in any way possible. Originally I just copied the command from the image README and gave it no further thought. If you are interested in what's being shared, have a look here. (Thanks Rob!)

Creating a docker network

To have the containers talk to each other and also the ability to add other services to the same network without including them in the docker-compose.yaml (bots for example), we'll create a docker network.

docker network create matrix-network
# To see what containers are connected (none atm..)
docker network inspect matrix-network

Setting up docker-compose

Now we create a docker-compose.yaml with our two services. Some options here are rather important to the setup so I've commented them in the file. Here is a gist with the contents as well.

cd /opt/matrix
sudo nano docker-compose.yaml

version: "2"
services:
  postgres:
    image: postgres:9.6.4
    restart: always

    # I like to be able to use psql on the host to connect to the database 
    # for maintenance. If you already have a postgres running you should remove 
    # the 'ports' section and uncomment 'expose'
    # expose:
    # - 5432

    # Adding 127.0.0.1 ensures the port isn't exposed ON the host
    ports:
      - "127.0.0.1:5432:5432"
    volumes:
     - /opt/matrix/pgdata:/var/lib/postgresql/data

    # These will be used in homeserver.yaml later on
    environment:
     - POSTGRES_PASSWORD=YOUR_PASSWORD_HERE
     - POSTGRES_USER=synapse

  synapse:
    image: silviof/docker-matrix
    # Exposing 8008 (no TLS) on localhost means we can reverse proxy with nginx
    # 8448 is for federation and should be exposed on host
    # 3478 is for TURN (voip calls)
    ports:
     - "127.0.0.1:8008:8008"
     - "8448:8448"
     - "3478:3478"
    volumes:
     - /opt/matrix/synapse:/data

# Our docker network!
networks:
  default:
    external:
      name: matrix-network

Editing homeserver.yaml

There are a couple of places we need to make modifications. We want to disable the built-in webclient and make sure port 8008 is accessible from the host.

sudo nano /opt/matrix/synapse/homeserver.yaml

# I've remove default comments and added mine
listeners:
  -
    port: 8448

    bind_addresses:
      - '0.0.0.0'

    type: http
    tls: true

    resources:
      -
        names:
          - client
          #- webclient # I've disabled this
        compress: true

      - names: [federation] # Federation APIs
        compress: false

  # Unsecure HTTP listener,
  - port: 8008
    tls: false

    # Since it's running in a container we need to listen to 0.0.0.0
    # The port is only exposed on the host and put behind reverse proxy
    bind_addresses:
      - '0.0.0.0'

    type: http
    x_forwarded: true
    resources:
      # I've removed webclient here as well
      - names: [client]
        compress: true
      - names: [federation]
        compress: false

We change from default sqlite database to postgres with our credentials from docker-compose.yaml.

# Database configuration
database:
  name: psycopg2
  args:
    user: synapse
    password: YOUR_PASSWORD_HERE
    database: synapse

    # This hostname is accessible through the docker network and is set 
    # by docker-compose. If you change the name of the service it will be different
    host: postgres

We'll enable registration to be able to test. You can change this later on.

# Enable registration for new users.
enable_registration: True

☝️ Don't replace the entire homeserver.yaml with this, just make sure the corresponding sections are correct.

Editing log config (optional)

I prefer to have the logs in a separate directory so let's change that. There should be a file in your /opt/matrix/synapse called yourhostname.log.config. Edit it and change to

handlers:
  file:
    filename: /data/log/homeserver.log

# Create the directory
sudo mkdir /opt/matrix/synapse/log

Obtaining Let's Encrypt cert

You need to have certbot installed!

sudo service nginx stop
sudo letsencrypt certonly --standalone -d yourhostname.com
sudo service nginx start

Nginx configuration

Here's the gist.

sudo nano /etc/nginx/sites-available/example.com # or whatever

server {
       listen 80;
       server_name example.com www.example.com;
       return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name example.com;

    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

    # If you don't wanna serve a site, comment this out
    root /var/www/example.com;
    index index.html index.htm;

    location /_matrix {
      proxy_pass http://0.0.0.0:8008;
      proxy_set_header X-Forwarded-For $remote_addr;
    }
 }

sudo ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/example.com

Start all the things!

Now we should be ready to go so let's try.

cd /opt/matrix
docker-compose up -d
docker-compose ps

# Something like this 
matrix_postgres_1 docker-entrypoint.sh postgres Up 127.0.0.1:5432->5432/tcp
matrix_synapse_1 /start.sh start Up 0.0.0.0:3478->3478/tcp, 127.0.0.1:8008->8008/tcp, 0.0.0.0:8448->8448/tcp

If everything looks good

sudo service nginx reload

Register account and login

Here comes the fun part! Let's create an account :)

Any client would do but for this let's use Riot. Click here, fill in your info and change the "Custom server" to the hostname of your newly created one. Adding an email is optional but if you ever need to reset your password you can't without it.

If everything is ok you should be greeted by the friendly `@riot-bot`!

Happy happy joy joy

So, hopefully you've made it this far and now have your own Matrix homeserver. There are a lot of neat things to do with Matrix and I'll be posting more about that, bots and other integrations for example. But for now enjoy your awesome federated open source chat and invite some friends!

If you found any errors in this guide or just feel like sharing your appreciation, drop me an email or tweet :)

SMS gateway and push notifications

Jon Neverland — Sun, 27 Aug 2017 20:35:36 +0000

A lot of services today either require or really want you to provide a cellphone number when signing up. I prefer not sharing my private number with these services and I also prefer not having these texts clog up my inbox.

First I thought I could do this with Twilio but apparently they don't allow receiving texts from short codes like Twitter and Facebook uses. So I decided to build an SMS gateway! You don't really need much to do this

A 3G/4G modem
A sim card, preferably prepaid
A computer :)

There are a few frameworks if you want more advanced features, for example playSMS or smsgateway. I haven't tried any of these myself but they look great! Since I only really wanted to receive texts I went with gammu. I found this guide which explained most of what I had to do.

I had an old Huawei E398 laying around at home. I got it working with the instructions below but after a few hours it would disconnect and I had to reinsert it. I'm not sure if this was due to a faulty modem or something else.. So YMMV!

Now I'm using a Huawei E122 and it worked OOTB. lsusb for that modem gives

ID 12d1:1001 Huawei Technologies Co., Ltd. E169/E620/E800 HSDPA Modem

If you have another modem it might be worth checking out the device reference for usb_modeswitch if it's supported. There's also a parameter reference for those who don't like man-pages.

Switching usb mode

Most modems will present themselves as a USB storage when you insert them, mostly for Windows users to they can install drivers. So you need to tell it to switch to modem mode and for this we use usb-modeswitch.

Instructions for Huawei E398

sudo apt-get install gammu usb-modeswitch

Check what mode we're in, for my modem storage mode looked like this

lsusb
=> Bus 002 Device 019: ID 12d1:1505 Huawei Technologies Co., Ltd. E398 LTE/UMTS/GSM Modem/Networkcard

12d1 above is the vendor ID and 1505 the product ID.

Now you need to find the product ID of the modem mode. Some guides recommended restarting the computer with the modem plugged in but that didn't help for me so I had to google a bit. There are tons of forum posts for different modems and in my case I found the correct ID to be 1506.

You also need to find the "message-content" to send the modem with the -M flag.

Switch the modem mode

# -v is current vendor ID
# -p is current product ID
# -V is target vendor ID (usually the same)
# -P is target product ID
# -M is the message
sudo usb_modeswitch -v 12d1 -p 1505 -V 12d1 -P 1506 -M 
"55534243123456780000000000000011062000000100000000000000000000"

Check that the modem has switched mode

lsusb
=> Bus 002 Device 005: ID 12d1:1506 Huawei Technologies Co., Ltd. Modem/Networkcard

There should now be one or more devices to use

dmesg | grep tty
# Should give something like
GSM modem (1-port) converter now attached to ttyUSB0
GSM modem (1-port) converter now attached to ttyUSB1
GSM modem (1-port) converter now attached to ttyUSB2

Run the gammu config and set port to /dev/ttyUSB0

sudo gammu-config

Check if gammu can identify the modem

sudo gammu --identify
Device : /dev/ttyUSB0
Manufacturer : Huawei
Model : unknown (E398)
Firmware : xxx
IMEI : yyy
SIM IMSI : zzz

All good! Now you should be able to send an sms like so

sudo gammu sendsms TEXT [phonenumber] -text "Ohhai"

Automatically switch mode when inserted or after reboot

After a reboot or unplug and reinsert the modem usually switches mode again. To make sure we always are in modem mode you need to create a config for usb_modeswitch matching the vendor ID and product ID.

sudo nano /etc/usb_modeswitch.d/12d1:1505

DefaultVendor=0x12d1
DefaultProduct=0x1505
TargetVendor=0x12d1
TargetProduct=0x1506
MessageContent="55534243123456780000000000000011062000000100000000000000000000"

Symlink to /dev/sms for consistency

It might be a good idea to create a symlink for your device so your settings for gammu always works, otherwise sometimes after reboot it might switch to /dev/ttyUSB1 or whatever.

sudo nano /etc/udev/rules.d/999-sms-gateway.rules
# Add this line 
SUBSYSTEM=="tty", ATTRS{idVendor}=="12d1", ATTRS{idProduct}=="1506", SYMLINK+="sms"
# Then
sudo udevadm control --reload

Forward texts with WebHook

I wanted to send all received texts to my Huginn instance and from there to Pushover. Huginn supports a bunch of services so if you prefer Slack or even Twitter that's no problem!

I'm using a modified version of this script.

Install gammu-smsd and configure it.

sudo apt-get install gammu-smsd
sudo pip install requests

sudo nano /etc/gammu-smsdrc
# Change the port in the [gammu] section to your device
# Add the RunOnReceive in the [smsd] section.
# RunOnReceive = python /opt/gammu/receivesms.py

sudo service gammu-smsd restart

Feel free to change the path of the script, I put in it /opt/gammu and chown:ed the dir to gammu.

Modified version of the script

#!/usr/bin/env python

from __future__ import print_function
import os, sys, re, requests

def get_message():
    files = [os.path.join('/var/spool/gammu/inbox/', m) for m in sys.argv[1:]]
    files.sort() # make sure we get the parts in the right order
    number = re.match(r'^IN\d+_\d+_\d+_(.*)_\d+\.txt', os.path.split(files[0])[1]).group(1)
    text = ''
    for f in files:
        text += open(f, 'r').read()
    try:
        text = text.decode('UTF-8', 'strict')
    except UnicodeDecodeError:
        text = text.decode('UTF-8', 'replace')
    return number, text

number, text = get_message()

request = requests.post('YOUR_HUGINN_URL', data={
    'from': number,
    'message': text
})

To get the url for Huginn, all you need to do is to create a Webhook Agent. Now try sending a text to your phonenumber and it should show up in Huginn! From there you can forward it to whatever service you prefer, either with one of the built-in Agents or a regular WebHook with Post Agent.

If you have any questions drop me an email or tweet me.

DEV Community: Jon Neverland

Pixelfed (beta) with Docker and Traefik

Prerequisites

Setup

Configuration

Docker

Site admin

The end

Adding webhooks to Matrix

Setup

Registration file

docker-compose

Adding hook to a room

Posting them hooks

Updating hooks

Matrix homeserver (Synapse v1.0.0) with Traefik

🚀 Update 🚀

Prerequisites

Setup

docker-compose.yml

Nginx

Synapse config

Register account and login

Final notes

SRV Record

Troubleshooting

Hosting Riot

Thanks!

Element (Riot) web for Matrix with Docker and Traefik

Prerequisites

Setup

docker-compose, with Traefik

No Traefik

Fetch latest Element version

Config

Run it!

Updating

Web client for Gammu SMSD

Building Node Docker images for Raspberry Pi Zero

With local NPM

Multistage build

Building

Automation

Sample .dockerignore

Sharing is caring

Easy InfluxDB & Grafana setup with Traefik

Setup

docker-compose.yml

InfluxDB

Grafana

Grafana UI

Getting some data in there

The End

Traefik with Docker and Let's Encrypt

Prerequisites

Setup

Basic auth

Add a container

Updates

High memory usage in Influxdb

Raspberry Pi Zero as SMS gateway

Assembling

Setup

Connecting

Sending SMS!

Reading list

Matrix homeserver with Docker

Prerequisites

Setup

Base directory

Generating Synapse files

Creating a docker network

Setting up docker-compose

Editing homeserver.yaml

Editing log config (optional)

Obtaining Let's Encrypt cert

Nginx configuration

Start all the things!

Register account and login

If everything is ok you should be greeted by the friendly @riot-bot!

If everything is ok you should be greeted by the friendly `@riot-bot`!