DEV Community: Jacob Korsgaard

[Solved] iPhone "The device is unreachable"

Jacob Korsgaard — Sat, 06 May 2023 14:09:05 +0000

If you've ever tried copying photos from your iPhone using a PC, you may have encountered an issue that goes something like:

"The device is unreachable"

You may also have had issues with the iPhone disconnecting from the PC while you're trying to copy photos.

I have finally found out why this is happening. It turns out it's because the iPhone runs out of space!

For every copy operation you do from the phone to the PC, an additional copy is created on the phone in a buffer!

To ensure success, clear up space on your iPhone if you can, then reboot the iPhone to clear out any buffers. Then copy off the biggest files off the iPhone 1 by 1, delete them from the phone and reboot. Slowly build up space on the phone so further operations succeed.

Whenever you get the error, reboot the iPhone to clear up the buffer used and try again.

Get Azure Pipelines UniversalPackages version

Jacob Korsgaard — Fri, 16 Apr 2021 07:29:39 +0000

I was building an azure pipelines job and during it I uploaded a Universal Package using the UniversalPackages task.

I then had to contact a REST API with the package and version of the newly published Universal Package. To do that I need to get the version and package name out of the task.

That looks like this:

    - task: UniversalPackages@0
      displayName: Universal Publish
      inputs:
        command: publish
        publishDirectory: '$(System.DefaultWorkingDirectory)/dist'
        vstsFeedPublish: 'MyProject/MyFeed'
        vstsFeedPackagePublish: 'package.name'
        packagePublishDescription: 'My super package'
        versionOption: patch #increment the patch part of the semantic versioning
        publishedPackageVar: someVariableName

After this task runs $(someVariableName) will hold a string like so "package.name 1.0.1", but that variable will not be available outside this current job.

To make it available for other jobs you need to move it to an output variable. I did that like this:

    - powershell: echo "##vso[task.setvariable variable=PackageNameAndVersion;isOutput=true]$(someVariableName)"
      name: PackagePublish

Now the value is available in other jobs through PackagePublish.PackageNameAndVersion.

You can consume it in subsequent jobs by making sure those jobs depend on this job and then grabbing the variable from the output.

- job: 'RestAPICallingJob'
  dependsOn: 'BuildProductionStuff'
  pool: server
  variables:
    PackageNameAndVersion: $[ dependencies.BuildProductionStuff.outputs['PackagePublish.PackageNameAndVersion'] ]

Now I can add a step to call a REST API using the package name and version of my universal package.

NOTE: I ended up not using Universal Packages, because you can only download Universal Packages from Azure using the AzureCLI. Instead, I created a NuGet package that contained just my loose files using a .nuspec file.

RClone in a Docker to backup OneDrive locally

Jacob Korsgaard — Sat, 20 Mar 2021 15:54:27 +0000

Today I'm trying to setup my Unraid server to take a daily backup of the contents of my OneDrive folder. That's in case of catastrophic failure on Microsoft's side, or more likely, I mess up and nuke my OneDrive folder for whatever reason. It's also just nice to have the data available locally.

To accomplish this task I'm using RClone and I'm using it from a Docker container, primarily to continue on my learning journey with Docker. It's also nice that I could test things out on my Windows PC and easily move the setup to my Unraid server.

The RClone image I used is rclone/rclone:latest. That said, in order to actually get started with RClone you need to configure it first, that is give it access to whatever cloud service you want to access.

To do this I downloaded the Windows binary instead, because RClone has a simple interactive flow you can use to create a configuration file. See instructions here.

After running the configuration I had an rclone.conf file ready to go. I deployed this to the Unraid server and set about defining the command-line to use to backup my Unraid settings.

I went with a sync command parameterized with --backup-dir. Then I set the command to run using the "Community Applications User Scripts" plugin on Unraid (which is basically just an easy way to setup cron jobs from the Unraid WebUI)

The final bash script looks something like this:

#!/bin/bash
NOW=$(date +"%Y-%m-%d")
docker run --rm \
  --volume /mnt/user/appdata/rclone:/config/rclone \
  --volume /mnt/user/onedrive:/data \
  rclone/rclone sync onedrive: /data/root \
  --backup-dir /data/backup/$NOW

First we create a date string for the backup folder. The backup folder is used whenever a file is deleted by the sync operation. Note: It's likely I'll want to run a deduplication script after syncing so the backup folder doesn't end up full of files that I already backed up.

Then we mount in the config folder for rclone as well as the target folder to copy the files to.

Finally, we run the RClone command with the root OneDrive folder as source onedrive: and the mounted folder as the target /data/root.

The onedrive: label is the one I created for my connection to OneDrive during configuration of RClone.

The one remaining niggle with the result is that the files created are all read/write only by root and read-only by everyone else. For my purpose, that's fine.

Node.js Docker Service Tips

Jacob Korsgaard — Tue, 16 Mar 2021 21:46:23 +0000

Last time I deployed some docker containers remotely to Unraid. Now let me share a tip or two for the node.js part of those containers.

Graceful exit

Node.js applications do not automatically know how to gracefully shut down when a container is stopped. To do this you'll have to handle signals.

I followed this nice guide: Building Graceful Node Applications in Docker and they got the signal handling from here Trapping signals in Docker containers.

The crux of the code is to add a listener to process:

function shutdown() {
  // close your server here
  process.exit();
}

process.on('SIGTERM', shutdown); 
process.on('SIGINT', shutdown);

Node.JS Signal Events Documentation doesn't call process.exit() but it turns out Docker does funny things due to the process having ID 1, this is the most up to date post I've found about that issue: Top 4 Tactics To Keep Node.js Rockin’ in Docker see "Start Node Directly in Dockerfiles".

Container content not updating

Another issue I ran into with my node.js container was that when I made changes to my index.js and ran the command to deploy my containers to Unraid, the Unraid nodeserver container would not have my changes.

I searched for a solution and -no-cache and --force-recreate were both suggested, but these methods did not fix the problem.

The only thing I found that actually fixed the issue was to remove the container and images from Unraid before building new ones and deploying those.

docker-compose --context unraid -f docker-compose.yml -f docker-compose.prod.yml down --rmi local
docker-compose --context unraid -f docker-compose.yml -f docker-compose.prod.yml up --build -d

Now, arguably an even better way would be to not put the index.js into the container, but instead mount a path to run index.js from. Then just restart the nodeserver container when I made changes. Which is probably what I would do in the future for iteration.

Remote Deploy with Docker-Compose to Unraid

Jacob Korsgaard — Tue, 16 Mar 2021 21:17:23 +0000

In my previous post I built a couple of containers with docker-compose. Now I'd like to share how I deploy those containers to my Unraid server using SSH.

I've had two goals, from my Windows machine I'd like to:

Deploy my containers to Unraid
Iterate on the containers locally

Multiple configurations with docker-compose

The first thing I tackled was figuring out how to customize my docker compose files based on the target I'm building for.

It turns out the tool for that is Multiple Compose Files.

By default when you run docker-compose, the tool read two files from the target directory docker-compose.yml and docker-compose.override.yml. These two files are effectively merged to create the configuration. The thought is you'd put the common configuration into docker-compose.yml and your development settings into the override.

docker-compose.yml

version: "3.8"
services:
    nodeserver:
        build:
            context: ./home-video-server
    nginx:
        restart: always
        build:
            context: ./home-server-nginx

docker-compose.override.yml

version: "3.8"
services:
    nodeserver:
        volumes:
            - c:\Users\jacob\videos:/media
    nginx:
        restart: always
        ports:
            - "81:80"
        volumes:
            - ./home-video/www:/data
            - c:\Users\jacob\videos:/media

When you want to run a command using a different configuration, what you do is you explicitly declare exactly which files to use in the configuration. I created a docker-compose.prod.yml with the settings for the Unraid containers.

docker-compose.prod.yml

version: "3.8"
services:
  nodeserver:
    volumes:
      - /mnt/user/kids:/media
  nginx:
    restart: always
    ports:
      - "9000:80"
    volumes:
      - /mnt/user/web:/data
      - /mnt/user/kids:/media

the command I run when building for unraid is then

docker-compose -f docker-compose.yml -f docker-compose.prod.yml up --build

Using Contexts to target Unraid

Now to actually send those containers to Unraid using docker-compose.

There are a number of different methods you can use, I found this nice article going over some of them: How to deploy on remote Docker hosts with docker-compose

Contexts is the weapon of choice and it's fairly straightforward. First we have to create a context, which is a named connection string basically. Then we have to use that context in all our docker commands to make them execute on the target.

Create a context called 'remote'

docker context create remote ‐‐docker “host=ssh://user@remotemachine”

This command will use SSH to connect to a 'docker endpoint' (that's the --docker part). Docker Context Documentation

For Unraid, it'll look something like this:

docker context create unraid ‐‐docker “host=ssh://root@tower”

Our command from before now becomes the following:

docker-compose --context unraid -f docker-compose.yml -f docker-compose.prod.yml up --build

SSH to Unraid (Use SSH Keys)

If you tried running this you would be asked for your root password roughly a 100 times and it's likely the job would fail anyway.

Because docker-compose executes so many commands over the SSH connection, it will need the password many times. We can't have that and will have to setup an SSH key.

Before that though: SSH doesn't work on Unraid by default (Access Denied error). You have to login to Unraid and give the root user a password before you can SSH. It's not enough to create a new user and give that a password, root has to have a password.

What we need to do now is generate an SSH key on our Windows 10 machine. Then copy the public part of the key to Unraid.

I used this guide to help me with SSH keys How To Set Up SSH Keys

You should already have a version of OpenSSH installed on your Windows machine in c:\Windows\System32\OpenSSH. So from a command line, you can generate a key like this:

ssh-keygen -t rsa

Let it install this into your user folder (c:\users\jacob\.ssh\id_rsa).

Now we have to authorize that key on Unraid, by getting the public part of the key over to it.

Because you're playing with Docker on Windows with WSL2, we can use the WSL bash to get it across to Unraid. Fire up your WSL and use this command:

ssh-copy-id -i /mnt/c/Users/jacob/.ssh/id_rsa.pub root@tower

That's it! ...or is it?

ssh-agent: agent returned different signature type

If you try to run the docker command again, you'll have to put in your password still and you'll get this error/warning ssh-agent: agent returned different signature type.

I found the answer and solution to this here.

It turns out the OpenSSH installation in Windows is old and incompatible. Specifically the ssh-agent tool isn't good enough. So you'll have to go download another OpenSSH binary release extract it to your machine and use the instructions in the link above to register the ssh-agent.exe from that location instead of the default installation.

This is crappy, but it did solve the issue on my machine. A better way to do all this is probably to run all your docker stuff directly from WSL, which would have an updated SSH.

Now you should be set to run your docker-compose command against the Unraid context!

docker-compose --context unraid -f docker-compose.yml -f docker-compose.prod.yml up --build

Next up are tips for building and running the node.js applications.

Playing with Docker on Unraid from Windows 10

Jacob Korsgaard — Tue, 16 Mar 2021 20:22:53 +0000

I've recently setup an Unraid server in my house and wanted to use the opportunity to learn a bit about Docker by developing a simple web app and deploy it to the server.

I'm writing this series to leave a paper trail for other people to follow since I ran into a number of issues.

I am by no means a docker expert. This is all from the point of view of a docker novice.

First a few basic details about my setup:

It's March 2021
Development PC is a Windows 10 version: 10.0.19042.867
Unraid is version: 6.9.1
Docker Engine is version: 20.10.5
I've installed WSL2

Here are some links to get the basics up and running:

The web app I'm building has a frontend (Angular/Ionic) and a backend (NodeJS). I'm not really going to go into those things as this is mostly about just getting docker working on Unraid.

Creating the Docker images and containers for my App

Initially I was going to use PHP for the server - because it's what I already had - but I found this excellent:

NGINX with Docker and Node.js — a Beginner’s guide

The end product of this guide is an NGINX container that serves as a reverse proxy for a Node.js server.

I didn't know what a reverse proxy was, but essentially it lets you access another service through the endpoint of NGINX. You can configure NGINX to pass your requests to another server and ferry the response back. This can be helpful if you want to hide the actual location of the other server. NGINX can also take care of all the HTTPS security stuff for your reversed proxied server.

This guide doesn't add a frontend so I thought I'd try to do that myself and followed the documentation from NGINX: NGINX Docs | Serving Static Content

Setting up the NGINX docker container

It only requires 2 files to setup NGINX in a container

Dockerfile the instructions to create the image.
default.conf the configuration of the server.

That's about as easy as anyone could hope for.

The Dockerfile literally just splats the configuration onto the server and you're done.

FROM nginx
COPY default.conf /etc/nginx/conf.d/default.conf

Now my default.conf looks like this:

server {
  root /data;

  location /media {
    rewrite ^/media(.*)$ $1 break;
    root /media;
  }

  location /nodeserver {
    rewrite ^/nodeserver(.*)$ /$1 break;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_pass http://nodeserver:5000;
  }
}

This is somewhat similar to Ashwin's guide, but I'll try to explain what I'm doing here.

First up root /data; is the root directory used to try to find the static content. I'm expecting the container to be able to find some HTML (particularly index.html) in the /data folder. I'll accomplish this later by mounting a folder on my system to the /data folder in the container.

The location /media block I use to serve some media files from another folder separate from the /data folder. Whenever someone requests a url that starts with /media/, then we serve from root /media; instead of the default root.

I did this without the rewrite line first, but then my requests would all be prefixed with /media so when I request http://localhost/media/myfile.mp4 NGINX would try to serve the file /media/media/myfile.mp4. To remove the first /media I used this rewrite rule. A solution I found here: serverfault | how to remove location block from $uri in nginx configuration?

Finally, there is the /nodeserver location, this is my access to the backend. Here I setup the reverse proxy as described in Ashwin's guide. I use the same rewrite rule here to remove the prefix from the requests forwarded to the backend. It's really crucial to note the / prefix in /$1, this won't work with just $1. I found the solution to that here

Node.JS container setup

I used the same approach as in Ashwin's guide.

Here is the Dockerfile I used to create my node server:

FROM node:alpine
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 5000
CMD ["node", "index.js"]

docker-compose for local deployment

I found Docker quite cumbersome before I made it to the docker-compose stage. It does make it a lot easier.

For a local deployment of this app, I need to spin up my two containers together and mount in a /data and a /media folder.

docker-compose.yml for that looks something like this:

version: "3.8"
services:
  nodeserver:
    build:
      # home-video-server has the nodeserver node.js DockerFile
      context: ./home-video-server
  nginx:
    restart: always
    build:
      # home-server-nginx has the NGINX DockerFile
      context: ./home-server-nginx
    ports:
      - "81:80"
    volumes:
      # Map the subfolder home-video/www to the /data folder
      - ./home-video/www:/data
      # and put my videos in the media folder
      - c:\Users\jacob\videos:/media

You can use docker-compose up --build to update the images and restart the containers.

In the default.conf for the NGINX server, you may have noticed that it refers to nodeserver by http://nodeserver:5000/ this kind of reference is only possible because we used docker compose with both containers. It's facilitated by a docker network that's created automatically. See Docker Compose Documentation on Networking.

Next time I'll explain how I managed to deploy the containers to my Unraid server using SSH.