Backup Grafana SQLite with Litestream using s6-overlay in a container app

I run my own small stack to monitor a set of nodes and use grafana to visualise all logs, metrics and events and deliver any alerts. For my use case running with sqlite is sufficient without adding an external dependency of postgres or mysql DB on the limited hardware that it runs on.

SQLite has recently gained lot of traction with Litestream, LiteFS and Turso making it a choice to be used in production at a larger scale than just on the edge for a smaller workload.

As grafana being a crucial component in my o11y stack backing up of the DB was equally crucial. There are many options ranging from running cron jobs and making copies upon booting and restarting to capture the state. None really captures the continuous changing state and there is greater chance of data loss between backups.

Litestream is a game changer allowing to backup all your changes to a an s3 compatible cloud storage at 1sec intervals using the WAL all done via the API and not interacting with the DB itself to avoid corruption or impacting performance.

But how to run Litestream within a container, Isn't container suppose to run 1 process as best practice ? Yes and no, I like the mantra s6-overlay gives of 1 process that can run multiple process that are supervised and linked based on dependency. A supervisor co-ordinates and makes sure that they all work in harmony.

Now on to how to use litestream with s6-overlay. benbjohnson creator of litestream has a repo over how to use litestream and s6-overlay with a go app container but the repo is publicly archived. Its probably so becuase its uses s6-0verlay version 2 while the newer version is 3 and its got a new design over how to organise your application and define the run scripts and dependencies.

s6-overlay has great instructions over how to set it up. It uses execline but most of the things are doable and most likely you won't face any shortcomings. In case of any limitations you can link to bash scripts to execute custom scripts.

The setup I have is where grafana and litestream are long running jobs that continue to run for the life of the container while litesream-restore is a oneshot job that only runs at the startup. with that sequence litestream depends on grafana to be running and grafana requires that litestream-restore executes successfully to restore any snapshots available before starting. The dependency graph would be as such

 litesream-restore --> grafana --> litestream

To setup jobs the files need to be setup as such for this sequence.

- /etc
  |- /s6-overlay
   |-/s6-rc.d
    |-/grafana
     |- type
     |- run
     |- finish
     |-/dependencies.d
     |- litesream-restore
    |-/litestream
     |- type
     |- run
     |-/dependencies.d
      |- grafana
    |-/litestream-restore
     |- type
     |- up
     |-/dependencies.d
      |- base
    |-/user
     |-/contents.d
      |- grafana
      |- litestream
      |- litestream-restore
   |-scripts
    |-/litestream-restore
     |- 00-litestream
  |-litestream.yml

type content will either be longrun or oneshot. We have 1 oneshot and 2 longrun jobs. There content would be as such

/etc/s6-overlay/s6-rc.d/grafana/type

longrun

/etc/s6-overlay/s6-rc.d/litestream/type

longrun

/etc/s6-overlay/s6-rc.d/litestream-restore/type

oneshot

Each application have their dependencies.d folder where an empty file is be defined with the name of its dependency application folder. In case of a application not having a dependency a file with a name base is to be defined.

The dependency graph

 litesream-restore --> grafana --> litestream

would be defined as such
/etc/s6-overlay/s6-rc.d/grafana/dependencies.d/base
/etc/s6-overlay/s6-rc.d/grafana/dependencies.d/litestream-restore
/etc/s6-overlay/s6-rc.d/litestream/dependencies.d/grafana

For longrun jobs a run file is to be defined while for oneshot up file is used. As oneshot could require complex steps and might be limited to execline you can define a bash script to be executed in the onshot up file.

To run oneshot litestream up define the path of the script file

/etc/s6-overlay/s6-rc.d/litestream-restore/up

/etc/s6-overlay/scripts/litestream-restore/00-litestream

/etc/s6-overlay/scripts/litestream-restore/00-litestream

#!/command/with-contenv bash

: ${MINIO_LITESTREAM_ENDPOINT?"Need to set MINIO_LITESTREAM_ENDPOINT environment variable"}
: ${MINIO_LITESTREAM_GRAFANA_BUCKET?"Need to set MINIO_LITESTREAM_GRAFANA_BUCKET environment variable"}
GRAFANA_DB_PATH="/var/lib/grafana/grafana.db"

# Restore the database if it does not already exist.
if [ -f $GRAFANA_DB_PATH ]; then
    echo "Database already exists, skipping restore"
else
    echo "No database found, restoring from replica if exists"
    exec /litestream restore -v -if-replica-exists -o $GRAFANA_DB_PATH "${MINIO_LITESTREAM_ENDPOINT}/${MINIO_LITESTREAM_GRAFANA_BUCKET}/grafana.db"
fi

• I use minio as a s3 compatible backup storage. Any s3 storage can be used

To start grafana using s6-overlay the run file would have following as we are using grafana-oss container
/etc/s6-overlay/s6-rc.d/grafana/run

#!/command/with-contenv bash

exec /run.sh

/etc/s6-overlay/s6-rc.d/litestream/run

exec /litestream replicate

The finish script is required for any cleanups or graceful shutdown when the container is turned off and can be defined in any longrun application that requires it. For us grafana being the main app based on which we run litestream the finish script is to be added there. Here I use the default script defined by s6-overlay repo README

/etc/s6-overlay/s6-rc.d/grafana/finish

#!/command/execlineb -S0

foreground { redirfd -w 1 /run/s6-linux-init-container-results/exitcode echo 0 }
/run/s6/basedir/bin/halt

Last part of the setup is to define empty files in /etc/s6-overlay/s6-rc.d/users/contents.d for all the applications that need to run by s6-overlay. In our case it would be as such

/etc/s6-overlay/s6-rc.d/users/contents.d/grafana
/etc/s6-overlay/s6-rc.d/users/contents.d/litestream
/etc/s6-overlay/s6-rc.d/users/contents.d/litestream-restore

For litestream you would need to define /etc/litestream.yml. For backing up to s3 compatible storage a minimal config would be as such

dbs:
  - path: /var/lib/grafana/grafana.db
    replicas:
      - type: s3
        bucket: ${MINIO_LITESTREAM_GRAFANA_BUCKET}
        path: grafana.db
        endpoint: http://${MINIO_LITESTREAM_ENDPOINT}
        force-path-style: true
        retention: 24h
        snapshot-interval: 1h
        validation-interval: 6h

The only part left now is how to install litestream and s6-overlay in the grafana-oss container. The grafna-oss container is hardened and uses USER 472 which restricts many operations. What worked for me is to escalate user privileges during build stage and reset them before starting.

FROM docker.io/grafana/grafana-oss:9.5.12-ubuntu

# Set USER to root escalating priviliges to perform installation of litestream and s6-overlay
USER root

RUN apt-get -qq update && \
  apt-get -qq install -y xz-utils \
  && rm -rf /var/libs/apt/lists/*

# https://github.com/benbjohnson/litestream-s6-example/blob/main/Dockerfile
# Download the static build of Litestream directly into the path & make it executable.
ADD https://github.com/benbjohnson/litestream/releases/download/v0.3.11/litestream-v0.3.11-linux-amd64.tar.gz /tmp/litestream.tar.gz
RUN tar -C / -xvzf /tmp/litestream.tar.gz

ARG S6_OVERLAY_VERSION="3.1.5.0"
# Download the s6-overlay for process supervision.
ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz /tmp
RUN tar -C / -Jxpf /tmp/s6-overlay-noarch.tar.xz
ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz /tmp
RUN tar -C / -Jxpf /tmp/s6-overlay-x86_64.tar.xz

# Copy s6 init & service definitions.
COPY etc/s6-overlay /etc/s6-overlay

# Copy Litestream configuration file.
COPY etc/litestream.yml /etc/litestream.yml

# The kill grace time is set to zero because our app handles shutdown through SIGTERM.
ENV S6_KILL_GRACETIME=0

# Sync disks is enabled so that data is properly flushed.
ENV S6_SYNC_DISKS=1

# Reset USER to 472 to reset the escalated privileges
USER 472

# # Run the s6 init process on entry.
ENTRYPOINT [ "/init" ]

SQLite is awesome, Litestream is a game changer and s6-overlay is superb. Run apps together without external dependencies and making them resilient couldn't be simpler.

Looking to use this for few more sqlite based apps and have the confidence of production readiness.