How to Backup your Applications Data to S3 with Walrus

👋 there!

I have been working lately to build a reliable and lightweight system for automated backups. Finally it is released 2 weeks ago 🚀

Clivern / Walrus

🔥 Fast, Secure and Reliable System Backup, Set up in Minutes.

Walrus

Fast, Secure and Reliable System Backup, Set up in Minutes.

Dashboard Screenshots

Walrus is a fast, secure and reliable backup system suitable for modern infrastructure. With walrus, you can backup services like SQLite, MySQL, PostgreSQL, Redis, etcd or a complete directory with a short interval and low overhead. It supports AWS S3, digitalocean spaces and any S3-compatible object storage service.

Documentation

Deployment

Download the latest walrus binary. Make it executable from everywhere.

$ export WALRUS_LATEST_VERSION=$(curl --silent "https://api.github.com/repos/Clivern/Walrus/releases/latest" | jq '.tag_name' | sed -E 's/.*"([^"]+)".*/\1/' | tr -d v)

$ curl -sL https://github.com/Clivern/Walrus/releases/download/v{$WALRUS_LATEST_VERSION}/walrus_{$WALRUS_LATEST_VERSION}_Linux_x86_64.tar.gz | tar xz

Then install etcd cluster or single node, please refer to etcd docs or bin directory inside this repository.

Run Walrus Tower:

Create the tower configs file tower.config.yml from config.dist.yml. Something like the following:

Please make sure to update the…

View on GitHub

Walrus is a fast, secure and reliable backup system suitable for modern infrastructure. With walrus, you can backup services like SQLite, MySQL, PostgreSQL, Redis, etcd or a complete directory with a short interval and low overhead. It supports AWS S3, digitalocean spaces and any S3-compatible object storage service.

Deploy with Docker

Install docker & docker-compose

$ apt-get update
$ sudo apt install docker.io
$ sudo systemctl enable docker
$ sudo apt install docker-compose

Clone Walrus for docker-compose.yml file

$ git clone https://github.com/Clivern/Walrus.git walrus
$ cd walrus/deployment/docker-compose

Feel free to update walrus tower port, api.key and api.encryptionKey. make sure you also use these values in walrus agent config file since agents require the tower URL, tower API key and encryptionKey to be able to reach and communicate with walrus tower.

Run the tower and etcd. It is also recommended to run etcd anywhere where data loss is mitigated.

$ docker-compose up -d

Now tower should be running. User your server public IP and tower port configured before to open the dashboard and setup the admin account.

# To get the public IP
$ curl https://ipinfo.io/ip

In the host where backups have to take place, download walrus binary.

$ curl -sL https://github.com/Clivern/Walrus/releases/download/v0.1.6/walrus_0.1.6_Linux_x86_64.tar.gz | tar xz

Create agent config file. Don't forget to replace agent.tower configs with the tower URL, apiKey and encryptionKey, you can get these values from tower configs you created earlier.

# Agent configs
agent:
    # Env mode (dev or prod)
    mode: ${WALRUS_APP_MODE:-prod}
    # HTTP port
    port: ${WALRUS_API_PORT:-8001}
    # URL
    url: ${WALRUS_API_URL:-http://127.0.0.1:8001}
    # TLS configs
    tls:
        status: ${WALRUS_API_TLS_STATUS:-off}
        pemPath: ${WALRUS_API_TLS_PEMPATH:-cert/server.pem}
        keyPath: ${WALRUS_API_TLS_KEYPATH:-cert/server.key}

    # API Configs
    api:
        key: ${WALRUS_API_KEY:-56e1a911-cc64-44af-9c5d-8c7e72ec96a1}

    # Async Workers
    workers:
        # Queue max capacity
        buffer: ${WALRUS_WORKERS_CHAN_CAPACITY:-5000}
        # Number of concurrent workers
        count: ${WALRUS_WORKERS_COUNT:-4}

    # Tower Configs
    tower:
        url: ${WALRUS_TOWER_URL:-http://127.0.0.1:8000}
        # This must match the one defined in tower config file
        apiKey: ${WALRUS_TOWER_API_KEY:-6c68b836-6f8e-465e-b59f-89c1db53afca}
        # This must match the one defined in tower config file
        encryptionKey: ${WALRUS_ENCRYPTION_KEY:-B?E(H+Mb}
        # Time interval between agent ping checks
        pingInterval: ${WALRUS_CHECK_INTERVAL:-60}

    # Backup settings
    backup:
        tmpDir: ${WALRUS_BACKUP_TMP_DIR:-/tmp}

    # Log configs
    log:
        # Log level, it can be debug, info, warn, error, panic, fatal
        level: ${WALRUS_LOG_LEVEL:-info}
        # output can be stdout or abs path to log file /var/logs/walrus.log
        output: ${WALRUS_LOG_OUTPUT:-stdout}
        # Format can be json
        format: ${WALRUS_LOG_FORMAT:-json}

Then run the host agent.

$ walrus agent -c /path/to/agent.config.yml

If everything is right, you should be able to see the host shown in the tower dashboard with one active agent. You can create backup crons under that host and update s3 configs in settings tab.