DEV Community: Kurt McAlpine

Unifi Autobackup Data Recovery and Restore

Kurt McAlpine — Sun, 02 Jun 2024 03:52:45 +0000

Unifi Autobackup Data Recovery and Restore

Unifi controller is a great piece of software that allows you to easily manage hundreds of WiFi access points, configuring multiple SSIDs, VLAN, etc. The insights dashboard can help you identify clients performance issue and rogue APs. In general I think Unifi is a great tool for home or small business use. So you might have guess, this is a cautionary tale that I hope you don't have to experience yourself since you have a robust backup process that is tested regularly!

Some background: at my work we run Unifi network controller on AWS EC2. There is a redundant backup process, each day we make a full snapshot of the data directory of the Unifi controller application. We do this by first stopping the application, running a simple tar command tar -czvf $(unifi--data-$(date +%s).tar.gz) config/data, upload the file to AWS S3 then starting the application back up again. Simple and effective. The restore process has been tested many times and works just as you'd expect. Just for good measure, Unifi provides an incremental backup file that it creates each day too, it creates an autobackup up with a name that looks something like this autobackup_8.0.24_20240527_1200_1716811200004.unf, and these are synced to AWS S3 as an insurance policy.

We made a couple mistakes with our S3 backup approach, we put a lifecycle rule on the bucket to delete files after a certain age. The intention was to be able to keep the last n backups, but in practice if something goes wrong with the backup process after a few days all the backups will get deleted. We should have implemented a monitor that checks that backups are being produced and alarm early if something is wrong so it can be fixed before any backups get deleted. Hindsight is 20/20.

Some work was being carried out on the EC2 instance that runs Unifi and as part of a normal procedure with these types of third party applications that we run, the instance was terminated and a new one created that would usually look for the latest backup and run the restore procedure automatically. The new instance never restored the data and when it was investigated, we noticed that all the backups were missing.

No worries, the autobackups were still on S3, surely we can just use that right?!

Error restoring backup
"{filename}" is not a valid backup.

*For the purposes of helping out others who might be in this situation, this all relates to version 8.0.24 of Unifi controller and specifically we were using this docker image https://hub.docker.com/r/linuxserver/unifi-controller which is now deprecated and you should be using the following instead https://hub.docker.com/r/linuxserver/unifi-network-application *

WHAT!

Ok, there must be something useful in the HTTP response:

HTTP/1.1 400 
X-Frame-Options: DENY
Content-Type: application/json;charset=UTF-8
Content-Length: 63
Date: Sun, 02 Jun 2024 03:00:25 GMT
Connection: close
{"meta":{"rc":"error","msg":"api.err.InvalidBackup"},"data":[]}

Nope just a 400.

Can anything be done?

So there is some hope here, I found some things online which can help us inspect the data. First step, there is this repository which provides a bash script to decrypt the backup: https://github.com/zhangyoufu/unifi-backup-decrypt

wget https://raw.githubusercontent.com/zhangyoufu/unifi-backup-decrypt/master/decrypt.sh
chmod +x decrypt.sh
./decrypt autobackup_8.0.24_20240527_1200_1716811200004.unf autobackup.zip

This produces a zip file, that seems to be completely broken:

$ unzip autobackup.zip 
Archive:  autobackup.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of autobackup.zip or
        autobackup.zip.zip, and cannot find autobackup.zip.ZIP, period.

but, this can be extracted further if you use 7zip, which can be installed on Linux or Mac:

# Ubuntu
sudo apt install p7zip-full

# Arch Linux
pacman -S p7zip

# Mac
brew install p7zip

Now extract the zip:

7z x autobackup.zip

This produces db.gz, which can further be extacted with gunzip to produce a BSON file named db

gunzip db.gz

Now the db file can be converted to plain text with MongoDB Database Tools, which can be downloaded from here: https://www.mongodb.com/try/download/database-tools

bsondump db > dump.json

This may produce a very large file depending on your Unifi deployment size and if you look into the data you see sections and data that seems to be for that section. For for example the lines after {"__cmd":"select","collection":"devices"} contain all the mongodb objects in the devices collection

I wrote a small program in go that can be used to load this data directly into the mongodb database. Which you can do by following these steps:

Copy the dump.json file into the container:

docker container cp dump.json 5ab135e2d58b:/dump.json

Download and run the program:

docker exec -it 5ab135e2d58b bash
curl -O https://github.com/kurtmc/blog/raw/master/2024-06/unifi-autobackup-data-recovery-and-restore/files/unifi-restore
./unifi-restore /dump.json

Depending on the size of your backup, this can take hours, but once it has completed you should restart the Unifi application. If the program fails, make sure you read the error, it may be due to the buffer size being too small or the max token size being too small, both of which can be configured using environment variables.

I hope you don't find yourself in this situation where you must rely on the autobackups, but if you do, I hope this helps!

Lightweight artifact repository with Python and GitHub

Kurt McAlpine — Sun, 10 Jul 2022 03:54:59 +0000

Lightweight artifact repository with Python and GitHub

Code reuse is fundamental to reducing the cost of software development, reusing a function implementation rather than developing it again is faster. Fixing a bug in a library rather than in multiple re-implementations is easier. A common way to reuse code is to package up related functionality and publish it as a library. Usually you could publish to Arifactory or Nexus but occasionally you may have a business constraint that makes it painful and slow to onboard a new tool, often for valid reasons, they maybe be expensive to onboard and support.

Coming from a background of Node.js and Go programming, it had been quite a shock to me when I saw the state of Python dependency management. Node.js and Go have canonical dependency management practices, with node you have npm and yarn and Go it's built into the toolchain. When you are ready to abstract some logic into it’s own library, it’s as simple as creating a new git repository, writing the appropriate metadata files (package.json, go.mod) push and you have a dependency you can import into your project!

Here are some examples of importing directly from git in the tools I am familiar with:

Node.js: yarn add https://github.com/octokit/rest.js.git
Go: go get github.com/google/go-github/v45

This is so incredibly easy and I want to have the same experience with Python. Is this possible? Almost!

I discovered that I could achieve similar ergonomics using existing and widely used tools, and I would like to demonstrate that here.

Python Library Project setup

The first step is to setup a Python library project, the best way to go about doing this is to follow the official documentation which can be found here: https://packaging.python.org/en/latest/tutorials/packaging-projects/

I will summarise what needs to be done to demonstrate a working example.

You will need to create the following structure in your git repository:

.
├── pyproject.toml
├── README.md
├── src
│   ├── example_package
│   │   ├── example.py
│   │   └── __init__.py
└── tests
    └── test_example.py

The contents of these files are listed here, you should update the fields in
pyproject.toml to match your organisation or project.

pyproject.toml Update this to match your organisation.

[build-system]
  build-backend = "hatchling.build"
  requires = ["hatchling"]

[project]
  classifiers = ["Programming Language :: Python :: 3", "License :: OSI Approved :: MIT License", "Operating System :: OS Independent"]
  dependencies = ["boto3==1.23.6"]
  description = "A small example package"
  name = "python-library-test"
  readme = "README.md"
  requires-python = ">=3.8"
  version = "0.0.1"

  [[project.authors]]
    email = "kurt.mcalpine@sourcedgroup.com"
    name = "Kurt McAlpine"

  [project.urls]
    "Bug Tracker" = "https://github.com/kurtmc/python-library-test/issues"
    Homepage = "https://github.com/kurtmc/python-library-test"

Note: you may add additional dependencies this project may have to the dependencies field under [project] . In this example I have added boto3 as a dependency.

src/example_package/example.py example code

def add_one(number):
    return number + 1

tests/test_example.py example test

import unittest

import sys
import os
sys.path.append(os.path.dirname(os.path.realpath(__file__)) + "/../src")
from example_package.example import add_one


class TestExamplePackage(unittest.TestCase):

    def test_add_one(self):
        expected = 1
        actual = add_one(0)
        self.assertEqual(expected, actual)


if __name__ == '__main__':
    unittest.main()

Once you have this structure setup, you can commit it and push it to your git repository. The next incredibly useful feature to add will be automatic versioning and tagging. We can use GitHub actions to automatically increment a version number and apply git tags. Later we will use the git tag to specify exactly which version of the library we want to include as a dependency to a new project.

Create the following files:

.github/workflows/update-version.yml You may want to change the branch name if main is not your default branch name.

name: Updates version and tags
on:
  push:
    branches:
      - main
permissions:
  contents: write
jobs:
  update_version_and_tag:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Install Python 3
      uses: actions/setup-python@v2
      with:
        python-version: 3.8
    - name: Update version
      uses: kurtmc/github-action-python-versioner@v1

Now any changes to main will be tagged and the version in pyproject.toml will be updated by GitHub actions:

Whilst we are here, we should add a GitHub action that runs on pull requests to enforce code style consistency and validate that the unit tests pass.

Create .github/workflows/pull-request.yml:

name: Run tests against pull requests
on: pull_request
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          ref: ${{ github.head_ref }}
      - name: Install Python 3
        uses: actions/setup-python@v2
        with:
          python-version: 3.8
      - name: Lint
        run: |
          pip install flake8==4.0.1
          flake8 ./src --ignore E501
  unit_tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          ref: ${{ github.head_ref }}
      - name: Install Python 3
        uses: actions/setup-python@v2
        with:
          python-version: 3.8
      - name: Install dependencies
        run: |
          pip install -e .
      - name: Run Python unittest
        run: |
          python -m unittest tests/*.py

Now we can ensure that all new code added to the library follows consistent code style and the unit tests pass.

We now have a Python library project in GitHub, following code style best practices and automatic version incrementing. How do we import it into a Python project?

Using the git URL in requirements.txt:

example-python-library @ git+https://github.com/YourOrg/example-python-library.git@0.0.1

Now lets try install it:

$ pip install -r requirements.txt
Collecting example-python-library@ git+https://github.com/YourOrg/example-python-library.git@0.0.1
  Cloning https://github.com/YourOrg/example-python-library.git (to revision 0.0.1) to /tmp/pip-install-1h4qrmmg/example-python-library_22bdfa1c6ab242c18e0e17b700c1be60
  Running command git clone --filter=blob:none --quiet https://github.com/YourOrg/example-python-library.git /tmp/pip-install-1h4qrmmg/example-python-library_22bdfa1c6ab242c18e0e17b700c1be60
Username for 'https://github.com':
Password for 'https://github.com':
  remote: Repository not found.
  fatal: Authentication failed for 'https://github.com/YourOrg/example-python-library.git/'
  error: subprocess-exited-with-error

  × git clone --filter=blob:none --quiet https://github.com/YourOrg/example-python-library.git /tmp/pip-install-1h4qrmmg/example-python-library_22bdfa1c6ab242c18e0e17b700c1be60 did not run successfully.
  │ exit code: 128
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× git clone --filter=blob:none --quiet https://github.com/YourOrg/example-python-library.git /tmp/pip-install-1h4qrmmg/example-python-library_22bdfa1c6ab242c18e0e17b700c1be60 did not run successfully.
│ exit code: 128
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

This fails to install because in the case of a private repository. We need to tell git to use our SSH credentials when cloning this private repository, which we can do with git config:

git config --global url."git@github.com:".insteadOf "https://github.com/"

Attempting the install again:

$ pip install -r requirements.txt
Collecting example-python-library@ git+https://github.com/YourOrg/example-python-library.git@0.0.1
  Cloning https://github.com/YourOrg/example-python-library.git (to revision 0.0.1) to /tmp/pip-install-z0q1jh7e/example-python-library_0e007d22fbd1439d9481e28d224387bf
  Running command git clone --filter=blob:none --quiet https://github.com/YourOrg/example-python-library.git /tmp/pip-install-z0q1jh7e/example-python-library_0e007d22fbd1439d9481e28d224387bf
  Running command git checkout -q 32600d1874df73fc209736eef6bbd09553cf2dc0
  Resolved https://github.com/YourOrg/example-python-library.git to commit 32600d1874df73fc209736eef6bbd09553cf2dc0
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: boto3==1.23.6 in /usr/local/lib/python3.10/site-packages (from example-python-library@ git+https://github.com/YourOrg/example-python-library.git@0.0.1->-r requirements.txt (line 1)) (1.23.6)

🥳 This is working locally! Now we should configure our CI/CD platform in the same way if we use SSH authentication, however if you are using personal access tokens registered against a service user you can configure git like this (assuming the personal access token is available under the GITHUB_TOKEN environment variable):

git config --global url."https://${GITHUB_TOKEN}@github.com/".insteadOf "https://github.com/"

Conclusion

Above, is a demonstration on how to build your own private dependency management system for Python using git and GitHub actions. Is this the best solution for private dependency management? Probably not, if you are in the position to pick technologies and services or are starting a greenfield project, you will be able to pick something that works out of the box (examples include: Artifactory, Nexus, AWS CodeArtifact) and establish best practices from the beginning. Not everyone is so lucky, and you may not be able to onboard a new tool so you need to stick with what you already have, and you almost certainly already have GitHub, this may be a solution for you.