DEV Community: Justin Cummings

🧟‍♂️ adventures in software development: automatic, scheduled wsl2 backups

Justin Cummings — Sun, 15 Aug 2021 15:18:16 +0000

As we rejoin our brave adventurer, we find that he has hurdled another piece of the backup puzzle, to be faced by yet even more hurdles when automatically scheduling Windows Subsystem for Linux (WSL2) backups. He has scoured and squirmed his way to timely exports of the distribution of concern and beat back weird interactions with the foul, yet, beloved beast, Docker.

The first step was creating an archive of the instance. That is made possible by the ‘wsl’ command using the ‘export’ sub-command.

# Get a list of WSL `distributions` and their status
$ wsl -l -v
  NAME                   STATE           VERSION
* Ubuntu                 Running         2
  docker-desktop         Running         2
  docker-desktop-data    Running         2

# Note the current directory/drive and change to a location to save the exported distribution to; I recommend not saving this to a location that is cloud synchronized, because it will be large and extremely time consuming to upload (and then sync to all connected devices).
$ pwd
Path
----
C:\Users\justi\Documents

# Export a distribution for backup
# Note that this takes it offline and for several minutes, but ensures absolute consistency
$ wsl --export Ubuntu
$ wsl -l -v
  NAME                   STATE           VERSION
* Ubuntu                 Converting      2
  docker-desktop         Running         2
  docker-desktop-data    Running         2


# Verify it completed once the 'State' shows as 'Stopped' for the given distribution.
$ ls

    Directory: C:\Users\justi\Documents

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---           4/12/2021  2:52 PM    12001935360 Ubuntu.tar

Now that we are confident that we can archive the distribution, the next step is to do it under automatic execution.

After fiddling around from near zero powershell or programatic task-scheduler knowledge, I currently can run the backup under the task scheduler for reoccurring, time-of-day and day-of-week based execution. Personally, I set it to the late evening-to-very-early morning hours, like 2:30AM locally, but only weekly, because that is my risk vs cost threshold and don't feel more frequent backups would be helpful; however, you might. I could register the task through the 'Task Scheduler' GUI, but what fun is that? Lets look a little more 😎.

Initial registration of the task is roughly prepared using power shell script. The archives are date-tagged in the filename scheme to easily identify when they were created and to make sorting and visualizing other aspects like growth easy to see.

# .\Register-ScheduledBackup-WSL2.ps1
...
# this is the meat; a lot of sides surround the main course, but this is how you add a scheduled task
Register-ScheduledTask -Action $action -Trigger $trigger -TaskPath $TaskPath -TaskName $taskName -Description $taskDescription -Principal $principle -Settings $settingsSet
...
# using a subscription to the task scheduler log
$subscription = @"
<QueryList>
    <Query Id="0" Path="Microsoft-Windows-TaskScheduler/Operational">
        <Select Path="Microsoft-Windows-TaskScheduler/Operational">
            *[EventData[@Name="ActionSuccess"][Data[@Name="TaskName"]="\$TaskPath\$taskName"][Data[@Name="ResultCode"]="0"]]
        </Select>
    </Query>
</QueryList>
"@
...
# add a task that fires based on that subscription
Register-ScheduledTask -Action $dockerRestartAction -Trigger $dockerRestartTrigger -TaskPath $TaskPath -TaskName 'Restart Docker' -Description "Restart Docker on completion of '$taskName'" -Principal $dockerRestartPrinciple -Settings $dockerRestartSettings
...

Note: the above example only creates the scheduled task. To update the scheduled task requires a different script. That is not shown. It is not implemented. Instead, I just unregister, and register anew. The actual backup script that is run via the above registration process is:

# .\Backup-WSL2.ps1
# parameter handling and logging omitted; here is the crux


$command = (Get-Command wsl).Definition
$dateExecuted = $(Get-Date -Format FileDate)
$commandArgs = "--export $Distribution $DestinationPath\$Distribution-$dateExecuted.tar"
# execute the backup
Invoke-Expression "& $command $commandArgs"

There is a problem with this. For some reason when the export/archive of the WSL distribution finishes, the connection to the docker socket of the host is lost or corrupted or something, so Docker needs to be restarted before your distribution can interact with the host docker system: We 😵

I could reboot the entire machine, but just restarting docker seems to be sufficient. I could use the Docker Desktop GUI/menus, but automatic restart would be nice while the underlying challenge remains. TLDR; Be 😎.

Note: restarting all of docker has it's own consequences, so proceed cautiously as any other running containers on the same host, your machine, will also be stopped.

The process to accomplish a docker restart via powershell is something that I am less comfortable showing, as it is based on a general instruction demonstrated in the following link:

Stackoverflow: Restart docker Windows 10 command line

The reason I am less comfortable showing how I do it because:

Licensing attached to Stackoverflow user-contributed code may impede personal or professional efforts.
The example solution does not work cleanly as is. So modification is required, and then I refer back to '#1' 😧.

But I will highlight the relevant commands here (with parameter handling, logging, and sleep or await commands omitted):

...
# kill the underlying Windows services used by docker, as gracefully as possible
Stop-Service -InputObject $_ -ErrorAction Continue -Confirm:$false -Force
...
# kill the user facing docker desktop app without delay (the app has little consequence on the underlying docker system, and is mostly just a dashboard; not a criticism) 
Stop-Process -InputObject $_
...
# start the desktop app because it generally takes the longest to completely start and won't complain terribly if the underlying docker system isn't ready or available
$_.Start()
...
# start the underlying Windows services for docker
Start-Process -FilePath $clientAppPath -PassThru |
...
# execute a `docker info` command and parse the results for indicators of readiness
$healthCheckResult = $($ServiceHealthCommand)
...

The complete script used to restart docker is:

.\Restart-Suite.ps1

In any case, once Docker is restarted, I am able to rejoin/restart my now safely archived distribution.

The complete effort to-date is found here:

backup-wsl2

Repeating the disclaimer in the repository README: this is very rookie stuff you will find here, and more importantly, it may be a terrible approach to managing wsl2 backups at the end of the day; but it is an option and maybe inspirational to someone with more foresight, talent, and time.

Insert Tim Allen as Tim the Toolman Taylor trope. I connected a service restart to the end of the backup script to automatically restart Docker after exporting the wsl instance at a regular time of day when I wasn't heavily using the machine. I should add more steps:

modify my process to also copy a complete, intact archive periodically to another location; one other option might be ‘rysnc’ the archive chunks to remote storage for isolation from the source (likely you want to stage it to local network storage first and let that NAS/SAN sync to offsite during off-peak periods)
test the recovery process; if you can't recover from it, the backup is not worth making
shrink the live distribution somehow, as the growth of the image has a negative affect on the archive process in terms of time and space (experience shows this is going to be fruitless in most cases)

🧟‍♂️ adventures in software development: first run of swiftlint analyze

Justin Cummings — Sun, 15 Aug 2021 14:29:56 +0000

Effective, efficient, safe, easy to reason about and maintain code is what professionals strive towards. In essence, clean. Using tools that can help us clean-up code should be common practice.

For Swift, the compiler finds many issues. To go beyond a valid build based on legal swift and allowed Xcode configurations when building on Mac, adding linting and AST static analysis is not hard, and raises the bar towards more idiomatic code. While Apple continues to refine and expand the basic built in tooling, the community has several tools to augment the build process in the meantime. One of those tools is swiftlint

The instructions on the website for installing and running the linter (both from the command line, and as a build step inside Xcode) are straightforward. I won't go into installing or linting here. How to run your first AST static analysis using swiftlint, which is experimental, is what I will describe from hands-on-experience beyond the somewhat sparse instructions on the tool's site.

To run the swiftlint AST static analysis, you must provide a successful, full-build log (incremental builds will not work, supposedly). Let me underline this point: Xcode build logs are required, so you won't be doing this outside of a Mac at this time. Normally, the build log from running the build process from the IDE is not exposed or kept in the project directory, so you have to do things a bit differently.

Creating a project-local build log is possible, and configuring it to do as a default process is possible, but beyond the scope of this article. There are reasons to and not to include a build log by default locally. At this time, Apple feels the convention should be to exclude them locally, so that is how we will roll.

Using the Xcode command-line tools (v12.5) in the terminal from the project root directory, issue the following command:

xcodebuild -list

Example output:

Command line invocation:
    /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -list

User defaults from command line:
    IDEPackageSupportUseBuiltinSCM = YES

Information about project "DifferenceOfSquares":
    Targets:
        DifferenceOfSquares
        DifferenceOfSquaresPackageDescription
        DifferenceOfSquaresPackageTests
        DifferenceOfSquaresTests

    Build Configurations:
        Debug
        Release

    If no build configuration is specified and -scheme is not passed then "Release" is used.

    Schemes:
        DifferenceOfSquares-Package

The relevant parts:

...
    Schemes:
        DifferenceOfSquares-Package
...

Using the information from that list of schemes, you can now create a build log that can be passed to the analyzer.

Using the Xcode command-line tools (v12.5) in the terminal from the project root directory, issue the following commands:

xcodebuild -scheme DifferenceOfSquares-Package clean build > build.log
swiftlint analyze --compiler-log-path ./build.log

Example output:

Analyzing Swift files in current working directory
Analyzing 'DifferenceOfSquares.swift' (1/1)
Done analyzing! Found 0 violations, 0 serious in 1 file.

There you have it. A static-analysis on my exercism project, 'DifferenceOfSquares' in the swift track.