DEV Community: Miro

Oauth2 certificate authentication in bash script

Miro — Mon, 08 Mar 2021 01:00:08 +0000

I was exploring the possibility of getting the auth token in bash using certificate authentication. Here are two examples how to obtain access token, one for Microsoft Graph and the other one for Google APIs.

Both scripts are similar:

create jwt payload/claims
sign header +payload with private key of the certificate uploaded to Microsoft/Google
post data to token endpoint
acquire access_token

A bit of explanation

base64 -w 0 | tr '/+' '_-' | tr -d '=' - this is simple base64url encoding base64 -w 0 means that data won't be broken in multiple lines, tr '/+' '_-' is used to replace / with _ and + with -, and finally tr -d '=' is used to remove = which is standard base64 padding
openssl dgst -sha256 -sign $key_file is signing input with certificate key stored in the $key_file
date +%s is current timestamp, and $(($ts + 600)) adds 10 minutes of token validity

Microsoft uses certificate fingerprint (thumbprint) in jwt header and it is a binary value encoded as base64url.

openssl x509 -fingerprint -noout -in $cert_file will produce something like SHA1 Fingerprint=AA:BB:CC:DD:EE... after which cut -d '=' -f 2 is used to get everything right of the = sign (the fingerprint), which is then converted into binary using xxd -r -p and finally converted into base64url

It also uses "a Guid" for unique identifier of a jwt where cat /proc/sys/kernel/random/uuid is used to create it.

In the end, jq is used to extract auth_token into the file or to display the error message in case authentication was not successful.

It is probably easier just to use vendor provided tools or libraries, but sometimes having a simple shell script is just good enough.

Microsoft Graph

#!/bin/bash
set -e

client_id='<replace_with_client_id>'
tenant_id='<replace_with_tenant_id>'
cert_file='my.crt' #certificate (used for fingerprint)
key_file='my.key'  #certificate private key (for signing)
cert_hash=$(openssl x509 -fingerprint -noout -in $cert_file | cut -d '=' -f 2 | xxd -r -p | base64 -w 0| tr '/+' '_-' | tr -d '=')
jwt_header="{\"alg\":\"RS256\",\"typ\":\"JWT\",\"x5t\":\"$cert_hash\"}"

ts=$(date +%s)
part_aud="\"aud\":\"https://login.microsoftonline.com/$tenant_id/oauth2/v2.0/token\""
part_nbf="\"nbf\":$ts"
part_exp="\"exp\":$(($ts + 600))"
part_jti="\"jti\":\"$(cat /proc/sys/kernel/random/uuid)\""
part_iss="\"iss\":\"$client_id\""
part_sub="\"sub\":\"$client_id\""
jwt_payload="{$part_aud,$part_exp,$part_iss,$part_jti,$part_nbf,$part_sub}"

token_data="$(echo -n $jwt_header | base64 -w 0 | tr '/+' '_-' | tr -d '=').$(echo -n $jwt_payload | base64 -w 0 | tr '/+' '_-' | tr -d '=')"

signature=$(echo -n $token_data | openssl dgst -sha256 -sign $key_file | base64 -w 0 | tr '/+' '_-' | tr -d '=')

assertion="$token_data.$signature"

resp=$(curl -Ssl -X POST "https://login.microsoftonline.com/$tenant_id/oauth2/v2.0/token" \
  --data-urlencode "client_id=$client_id" \
  --data-urlencode 'scope=https://graph.microsoft.com/.default' \
  --data-urlencode 'client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer' \
  --data-urlencode "client_assertion=$assertion" \
  --data-urlencode 'grant_type=client_credentials' \
)

if echo $resp | grep -q 'access_token'; then
    echo $resp | jq -j '.access_token' >access_token
else
    echo $resp | jq -r '.error + " error:\n" + .error_description'
    exit 1
fi

Source:

https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-client-creds-grant-flow#second-case-access-token-request-with-a-certificate

https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-certificate-credentials

Google APIs

#!/bin/bash
set -e

client_email='client@your-project-name.iam.gserviceaccount.com'
subject_email='subject@example.com' #user that will be impersonated
scopes='https://www.googleapis.com/auth/<scope1> https://www.googleapis.com/auth/<scope2>'
key_file='my.key' #certificate private key (for signing)

jwt_header="{\"alg\":\"RS256\",\"typ\":\"JWT\"}"

ts=$(date +%s)
part_iss="\"iss\":\"$client_email\""
part_sub="\"sub\":\"$subject_email\""
part_scope="\"scope\":\"$scopes\""
part_aud="\"aud\":\"https://oauth2.googleapis.com/token\""
part_exp="\"exp\":\"$(($ts + 600))\""
part_iat="\"iat\":\"$ts\""
jwt_payload="{$part_iss,$part_sub,$part_scope,$part_aud,$part_exp,$part_iat}"

token_data="$(echo -n $jwt_header | base64 -w 0 | tr '/+' '_-' | tr -d '=').$(echo -n $jwt_payload | base64 -w 0 | tr '/+' '_-' | tr -d '=')"

signature=$(echo -n $token_data | openssl dgst -sha256 -sign $key_file | base64 -w 0 | tr '/+' '_-' | tr -d '=')

assertion="$token_data.$signature"

resp=$(curl -Ssl -X POST 'https://oauth2.googleapis.com/token' \
  --data-urlencode 'grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer' \
  --data-urlencode "assertion=$assertion" \
)

if echo $resp | grep -q 'access_token'; then
    echo $resp | jq -j '.access_token' >access_token
else
    echo $resp | jq -r '.error + " error:\n" + .error_description'
    exit 1
fi

Source:

https://developers.google.com/identity/protocols/oauth2/service-account#httprest

Code

WSL2 Containers

Miro — Fri, 07 Aug 2020 16:05:55 +0000

Because, why not?

Playing around with WSL2, I figured it could be used to make one-off build containers. In a similar way I usually use docker.

For example, building TDLib for Telegram Messenger.

My run.cmd is simple:

@echo off
wsl --import alpine-build .\VM alpine-minirootfs-3.12.0-x86_64.tar.gz
wsl -d alpine-build /bin/sh /mnt/e/Temp/TDLib/build.sh
wsl --unregister alpine-build

Install alpine distribution by running wsl --import
Execute the build script
Unregister the distribution

Build script, build.sh , is made using TDLib Build instrunction generator, with the addition of the OUT_DIR variable just to make it easier.

OUT_DIR=/mnt/e/Temp/TDLib/lib

cd ~
apk update
apk upgrade
apk add --update alpine-sdk linux-headers git zlib-dev openssl-dev gperf php php-ctype cmake
git clone https://github.com/tdlib/td.git
cd td
git checkout v1.6.0
rm -rf build
mkdir build
cd build
export CXXFLAGS=""
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=$OUT_DIR ..
cmake --build . --target install
cd ..
cd ..
ls -l $OUT_DIR

Running it and it works.

PS E:\Temp\TDLib> .\run.cmd
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
v3.12.0-214-g4076381e49 [http://dl-cdn.alpinelinux.org/alpine/v3.12/main]
v3.12.0-213-gfedcac4c0f [http://dl-cdn.alpinelinux.org/alpine/v3.12/community]
OK: 12749 distinct packages available
(1/6) Upgrading musl (1.1.24-r8 -> 1.1.24-r9)
<snip>
-- The CXX compiler identification is GNU 9.3.0
-- The C compiler identification is GNU 9.3.0
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ - works
<snip>
-- Configuring done
-- Generating done
-- Build files have been written to: /root/td/build
Scanning dependencies of target tdsqlite
[  0%] Building C object sqlite/CMakeFiles/tdsqlite.dir/sqlite/sqlite3.c.o
[  0%] Linking C static library libtdsqlite.a
[  0%] Built target tdsqlite
<snip>
Scanning dependencies of target bench_queue
[100%] Building CXX object benchmark/CMakeFiles/bench_queue.dir/bench_queue.cpp.o
[100%] Linking CXX executable bench_queue
[100%] Built target bench_queue
Install the project...
-- Install configuration: "Release"
-- Installing: /mnt/e/Temp/TDLib/lib/lib/libtdjson.so.1.6.0
-- Installing: /mnt/e/Temp/TDLib/lib/lib/libtdjson.so
-- Installing: /mnt/e/Temp/TDLib/lib/lib/libtdjson_static.a
-- Installing: /mnt/e/Temp/TDLib/lib/lib/libtdjson_private.a
<snip>
total 0
drwxrwxrwx    1 root     root           512 Aug  7 14:03 include
drwxrwxrwx    1 root     root           512 Aug  7 14:03 lib
Unregistering...
PS E:\Temp\TDLib>

Not as versatile as docker run but does the job.

Using differencing disks

Using differencing disks, I can have all build tools installed in the base vhdx and spawn "build container" just for a single build.

For example, based on above, I've created base

PS E:\Temp> wsl --import Alpine-BuildBase E:\WSL2\Alpine-BuildBase E:\WSL2\alpine-minirootfs-3.12.0-x86_64.tar.gz

And installed everything I need

apk update
apk upgrade
apk add --update alpine-sdk linux-headers git zlib-dev openssl-dev gperf php php-ctype cmake

Now my build.sh omits tools installation and looks like:

OUT_DIR=/mnt/e/Temp/TDLib/lib

cd ~
git clone https://github.com/tdlib/td.git
cd td
git checkout v1.6.0
rm -rf build
mkdir build
cd build
export CXXFLAGS=""
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=$OUT_DIR ..
cmake --build . --target install
cd ..
cd ..
ls -l $OUT_DIR

And the final part, run.ps1 PowerShell script that will:

Create differencing vhdx
Make a registry key for the linux distribution "container"
Execute the build script
Unregister the distribution

#Must be run as an Administrator
if (!([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) {
    Start-Process PowerShell -Verb RunAs "-NoProfile -ExecutionPolicy Bypass -Command `"cd '$pwd'; & '$PSCommandPath';`"";
    exit;
}

$DISTRO_NAME = "alpine-build"

#Create a new differencing VHD
New-Item -Path VM -ItemType Directory -Force | Out-Null
New-VHD -ParentPath E:\WSL2\Alpine-BuildBase\ext4.vhdx -Path .\VM\ext4.vhdx -Differencing -BlockSizeBytes 1048576

#Add a registry key
$guid = New-Guid
$RegKey = "HKCU:\SOFTWARE\Microsoft\Windows\CurrentVersion\Lxss\" + $guid.ToString("B")
New-Item -Path $RegKey
New-ItemProperty -Path $RegKey -Name BasePath -PropertyType String -Value $PWD\VM | Out-Null
New-ItemProperty -Path $RegKey -Name DefaultUid -PropertyType DWord -Value 0 | Out-Null
New-ItemProperty -Path $RegKey -Name DistributionName -PropertyType String -Value $DISTRO_NAME | Out-Null
New-ItemProperty -Path $RegKey -Name Flags -PropertyType DWord -Value 15 | Out-Null
New-ItemProperty -Path $RegKey -Name State -PropertyType DWord -Value 1 | Out-Null
New-ItemProperty -Path $RegKey -Name Version -PropertyType DWord -Value 2 | Out-Null

#Run the build script
wsl -d $DISTRO_NAME /bin/sh /mnt/e/Temp/TDLib/build.sh

#Unregister the distribution
wsl --unregister $DISTRO_NAME

#Pause
Read-Host -Prompt "Press Enter key to continue"

Just testing

To make sure above script works as expected instead of running the build script (line 25 above) I can swap it with

wsl -d $DISTRO_NAME /bin/sh -c "date && cat /etc/*-release"

Result is

ComputerName            : PC123
Path                    : E:\Temp\TDLib\VM\ext4.vhdx
VhdFormat               : VHDX
VhdType                 : Differencing
FileSize                : 6291456
Size                    : 274877906944
MinimumSize             :
LogicalSectorSize       : 512
PhysicalSectorSize      : 4096
BlockSize               : 1048576
ParentPath              : E:\WSL2\Alpine-BuildBase\ext4.vhdx
DiskIdentifier          : A96ADE57-4838-48EA-97A9-DE02790ADCB4
FragmentationPercentage :
Alignment               : 1
Attached                : False
DiskNumber              :
IsPMEMCompatible        : False
AddressAbstractionType  : None
Number                  :

Property      : {}
PSPath        : Microsoft.PowerShell.Core\Registry::HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Lxss\{4
                e9ced54-7d28-48d2-92de-1df93ddc55a9}
PSParentPath  : Microsoft.PowerShell.Core\Registry::HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Lxss
PSChildName   : {4e9ced54-7d28-48d2-92de-1df93ddc55a9}
PSDrive       : HKCU
PSProvider    : Microsoft.PowerShell.Core\Registry
PSIsContainer : True
SubKeyCount   : 0
View          : Default
Handle        : Microsoft.Win32.SafeHandles.SafeRegistryHandle
ValueCount    : 0
Name          : HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Lxss\{4e9ced54-7d28-48d2-92de-1df93ddc55a9}

Fri Aug  7 15:27:17 UTC 2020
3.12.0
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.12.0
PRETTY_NAME="Alpine Linux v3.12"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
Unregistering...
Press Enter key to continue:

Conclusion

I'm not sure if this has a practical value. Maybe in scenarios where it is easier to have a single file virtual disk rather than a docker container stored in some registry. But it was an interesting concept to explore. Let me know if you find it useful.

Manually installing WSL2 distributions

Miro — Fri, 07 Aug 2020 15:55:29 +0000

There are a few different ways of installing Linux distributions for WSL. The easiest, using Microsoft Store, or using tools like LxRunOffline or wsldl. But distribution can be installed manually as well.

WSL command line allows to export distribution to a tar file, or to import from a tar file as a new distribution.

Let's say I have Ubuntu installed from Microsoft Store, if I list them from a command line I will get something like this:

PS E:\Temp> wsl --list --all -v
  NAME            STATE           VERSION
* Ubuntu-20.04    Stopped         2

And if I export it:

PS E:\Temp> wsl --export Ubuntu-20.04 E:\WSL2\ubuntu.tar

I will get a tar file containing a root filesystem of my Ubuntu distribution. And I can import that as a new distribution:

PS E:\Temp> wsl --import Ubuntu-Copy E:\WSL2\Ubuntu-Copy E:\WSL2\ubuntu.tar

This will create a folder Ubuntu-Copy folder in the E:\WSL2 and the ext4.vhdx file in that folder. Virtual hard disk is an ext4 partitioned disk containing the contents of the tar file. Listing distributions now shows two of them.

PS E:\Temp> wsl --list --all -v
  NAME            STATE           VERSION
* Ubuntu-20.04    Stopped         2
  Ubuntu-Copy     Stopped         2

If I start the second distribution, I will be logged in as a root account:

PS E:\Temp> wsl -d Ubuntu-Copy
root@PC123:/mnt/e/Temp#

Ok, so if import works for the exported root filesystem, will it work for any root filesystem? And the answer is Yes!

For example, if I download alpine-minirootfs-3.12.0-x86_64.tar.gz from https://alpinelinux.org/downloads/ I can import it like this:

PS E:\Temp> wsl --import Alpine-3.12 E:\WSL2\Alpine-3.12 E:\WSL2\alpine-minirootfs-3.12.0-x86_64.tar.gz

It even works with tar.gz files, not just .tar files. In fact, Alpine distribution available in Microsoft Store does exactly that. Downloads this file and installs it. Source for the launcher is available here: https://github.com/agowa338/WSL-DistroLauncher-Alpine

Similar to the reference implementation of the launcher (WSL-DistroLauncher) it allows user creation on start-up and it has some additional fixes. See the DistroSpecial.h file.

After distribution is complete I can manually create user, the same way launcher does

PS E:\Temp> wsl -d Alpine-3.12 /usr/sbin/adduser -g '' -D user1
Changing password for user1
New password:
Retype password:
passwd: password for user1 changed by root

And then logging in like

PS E:\Temp> wsl -d Alpine-3.12 -u user1
PC123:/mnt/e/Temp$

In a nutshell this is it. A new distribution is manually installed.

Important thing to note:
By unregistering the distribution like wsl --unregister Ubuntu-20.04 it will delete the associated ext4.vhdx file. Make sure to backup any important data form the virtual hard disk before unregistering.

Start-up scripts

Instead manually creating users and setting everything up, I can write all commands in a text file and execute them.

For example, a dummy init file that displays current date and release name

date
cat /etc/*-release

Start like

E:\Temp> wsl -d Debian-10-Slim < init

And the result is

Fri Aug  6 23:08:29 BST 2020
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Or run that file directly as a shell script from a know location

PS E:\Temp> wsl -d Alpine-3.12 /bin/sh /mnt/e/Temp/init

Fri Aug  6 22:17:05 UTC 2020
3.12.0
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.12.0
PRETTY_NAME="Alpine Linux v3.12"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"

Registry keys

When distribution is installed its settings are stored in the registry HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Lxss\{GUID}

Guid is, as far as I can tell, randomly generated for every distribution installation

For example, official installation of Ubuntu creates similar to the following:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Lxss\{2506e37c-8479-4433-9ea8-7549090aa5bf}]
"State"=dword:00000001
"DistributionName"="Ubuntu-20.04"
"Version"=dword:00000002
"BasePath"="C:\\Users\\User\\AppData\\Local\\Packages\\CanonicalGroupLimited.Ubuntu20.04onWindows_79rhkp1fndgsc\\LocalState"
"Flags"=dword:0000000f
"DefaultUid"=dword:000003e8
"PackageFamilyName"="CanonicalGroupLimited.Ubuntu20.04onWindows_79rhkp1fndgsc"
"KernelCommandLine"="BOOT_IMAGE=/kernel init=/init"
"DefaultEnvironment"=hex(7):<snipped>

Which means that virtual hard disk is stored in the C:\Users\User\AppData\Local\Packages\CanonicalGroupLimited.Ubuntu20.04onWindows_79rhkp1fndgsc\LocalState folder and default Uid is 1000 (0x3e8).

Looking at the Alpine distribution:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Lxss\{b9a51057-c168-4798-a89e-483c97278f3c}]
"State"=dword:00000001
"DistributionName"="Alpine-3.12"
"Version"=dword:00000002
"BasePath"="\\\\?\\E:\\WSL2\\Alpine-3.12"
"Flags"=dword:0000000f
"DefaultUid"=dword:00000000

it omits some settings, but BasePath points to a folder containing ext4.vhdx file and DefaultUid is the Uid of the user when starting the distribution without the -u <user> option.

Moving a distribution to a different location requires just to move the ext4.vhdx to a new folder and update the BasePath value. Similarly, changing a default user requires an update to the DefaultUid value.

To "restore" a missing distribution, copy a registry key, update the DistributionName, BasePath, and DefaultUid values, and it is ready to go. Or import an empty tar file (created like tar -zvcf empty.tar.gz -T /dev/null) and swap out the resulting empty ext4.vhdx with the real one.

Other distributions

Following the examples from the wsldl project, a simple way of installing different linux distributions is to take root filesystem used to make docker images. For example Dockerfile for Debian Buster Slim says ADD rootfs.tar.xz /. I can download the rootfs.tar.xz, unpack it since WSL doesn't recognize tar.xz file format, import it and run it like:

PS E:\Temp> wsl --import Debian-10-Slim E:\WSL2\Debian-10-Slim E:\WSL2\debian-buster-slim-20200803-rootfs.tar
PS E:\Temp> wsl -d Debian-10-Slim
root@PC123:/mnt/e/Temp#

Official `install.tar.gz`

When installing Ubuntu or Debian Linux distribution from the Microsoft Store root filesystem is downloaded as a part of Store App installation. This means that initial filesystem tar, install.tar.gz can be found in the C:\Program Files\WindowsApps \CanonicalGroupLimited.Ubuntu20.04onWindows_2004.2020.424.0_x64__79rhkp1fndgsc\ folder, for the Ubuntu 20.04 installation.

For a different version, folder name in the C:\Program Files\WindowsApps will be different, but it should contain install.tar.gz file.

And that file can be imported as a new distribution in the same way as mentioned above.

Differencing disks

Since WSL2 is using vhdx disks, I was wondering can I use differencing disks. And the answer is again Yes.

Having 5 instances of Ubuntu will take about 5GB of space immediately. Base vhdx is about 1GB. But, by leveraging differencing disks, I can have one base disk, and 5 smaller, differencing disks.

First stop all distributions with wsl --shutdown, then move existing ext4.vhdx to a new folder (one level up in my case), and create a differencing disk with the ext4.vhdx name.

PS E:\WSL2\Ubuntu-20.04> New-VHD -ParentPath ..\base.vhdx -Path ext4.vhdx -Differencing -BlockSizeBytes 1048576

When creating virtual hard disks for linux, make sure that you use 1MB block size. Default block size of 32MB will result in fairly large vhdx file once it starts to get filled.

Only article where I found this recommendation is Understanding and Working with VHD(X) Files

For Linux systems, there is a soft recommendation to use a 1 megabyte block size

By examining ext4.vhdx created by the installation, details also show 1MB block size

PS E:\WSL2\Ubuntu-Copy> Get-VHD .\ext4.vhdx | Format-List

ComputerName            : PC123
Path                    : E:\WSL2\Ubuntu-Copy\ext4.vhdx
VhdFormat               : VHDX
VhdType                 : Dynamic
FileSize                : 1164967936
Size                    : 274877906944
MinimumSize             :
LogicalSectorSize       : 512
PhysicalSectorSize      : 4096
BlockSize               : 1048576
ParentPath              :
DiskIdentifier          : A96ADE57-4838-48EA-97A9-DE02790ADCB4
FragmentationPercentage : 53
Alignment               : 1
Attached                : False
DiskNumber              :
IsPMEMCompatible        : False
AddressAbstractionType  : None
Number                  :

BlockSize effect on a dynamic disk file size

A little digression. Comparing a file size for vhdx with 1MB block size vs default 32MB block size.

I've created two vhdx files:

PS E:\Temp> New-VHD -Path .\256_default.vhdx -SizeBytes 256GB

Path                    : E:\Temp\256_default.vhdx
VhdFormat               : VHDX
VhdType                 : Dynamic
FileSize                : 4194304
Size                    : 274877906944
MinimumSize             :
LogicalSectorSize       : 512
PhysicalSectorSize      : 4096
BlockSize               : 33554432

and

PS E:\Temp> New-VHD -Path .\256_1mb.vhdx -SizeBytes 256GB -BlockSizeBytes 1MB

Path                    : E:\Temp\256_1mb.vhdx
VhdFormat               : VHDX
VhdType                 : Dynamic
FileSize                : 6291456
Size                    : 274877906944
MinimumSize             :
LogicalSectorSize       : 512
PhysicalSectorSize      : 4096
BlockSize               : 1048576

Initial sizes were 4MB and 6MB.

Attached them to a Hyper-V VM, booted the alpine-virt-3.12.0-x86_64.iso .

Then I ran mkfs.ext4 /dev/sda and mkfs.ext4 /dev/sdb and shut down the VM.

Resulting files are

PS E:\Temp> Get-VHD .\256_default.vhdx | Format-List

Path                    : E:\Temp\256_default.vhdx
VhdFormat               : VHDX
VhdType                 : Dynamic
FileSize                : 4869586944
Size                    : 274877906944
MinimumSize             :
LogicalSectorSize       : 512
PhysicalSectorSize      : 4096
BlockSize               : 33554432

and

PS E:\Temp> Get-VHD .\256_1mb.vhdx | Format-List

Path                    : E:\Temp\256_1mb.vhdx
VhdFormat               : VHDX
VhdType                 : Dynamic
FileSize                : 162529280
Size                    : 274877906944
MinimumSize             :
LogicalSectorSize       : 512
PhysicalSectorSize      : 4096
BlockSize               : 1048576

As you can see, after making the ext4 filesystem on a 256GB dynamic disk, the one with a 32MB block size grew to 4,53GB while while the one with the 1MB block size grew to just 155MB.

The end

Anyway, WSL2 seems interesting enough 😁

Socket activation of containers using Traefik's ForwardAuth

Miro — Sat, 25 Jul 2020 17:25:09 +0000

This is just an idea of mixing & matching a few services to get something like socket activation for docker containers.

Let's say there is a web hook which isn't triggered very often and it is created as a docker container. There is no point of running the container all the time just for an occasional calls to a web hook. But there is a problem how to start a container when needed, when a request is sent for that hook. Of course I could have something like OpenFaaS but that seems like a overkill for an intermittent process. In addition, OpenFaaS doesn't support mounting volumes by design so if I want to receive a file, confirm that it is received, process it in the background and store it somewhere on a filesystem, it requires an additional effort.

I'm already using Traefik as a reverse proxy and webhook for some scenarios and I wanted to see if I could use that to achieve something similar to socket activation.

Traefik doesn't have "execute this shell script" middleware, but it does have ForwardAuth middleware. It sends the request to the AuthServer and if the response code is 2XX access is granted ad original request is performed.

So my idea was: If I use webhook as an "AuthServer" with a hook being a shell script that starts a container, could this work?

As it turns out, it does work.

The setup

Instead of an actual container doing some work, this example will just be starting simple nginx container, and stopping it after 20 seconds.

In a proper setup, instead of a nginx there would be a real container doing real processing and after processing is complete it would exit gracefully. Instead of blindly running a container, and stopping it after 20 seconds, there would be check in place whether the container is already running etc. Pretty much everything that OpenFaaS Gateway module does. But this is just a general idea.

First I have a docker container which has webhook and docker-cli installed, built as:

FROM        golang:alpine3.11 AS build
WORKDIR     /go/src/github.com/adnanh/webhook
ENV         WEBHOOK_VERSION 2.7.0
RUN         apk add --update -t build-deps curl libc-dev gcc libgcc \
            && curl -sSL https://github.com/adnanh/webhook/archive/${WEBHOOK_VERSION}.tar.gz \
              | tar -xz -C /go/src/github.com/adnanh/webhook --strip 1 \
            && go get -d \
            && go build -o /usr/local/bin/webhook

FROM        alpine
COPY        --from=build /usr/local/bin/webhook /usr/local/bin/
RUN         apk update \
            && apk add --no-cache docker-cli
WORKDIR     /etc/webhook
ENTRYPOINT  ["/usr/local/bin/webhook"]

That container has configuration mounted as /etc/webhook/hooks.json:

[
  {
    "id": "start-nginx1",
    "execute-command": "/etc/webhook/scripts/nginx1",
  }
]

A shell script that will be executed, which starts a container (mounted as /etc/webhook/scripts/nginx1)

#!/bin/sh
set -e
docker run -d --name nginx1 --network skynet nginx:alpine
sleep 5
nohup /etc/webhook/scripts/nginx1-remove >& /dev/null &

And a shell script that will stop the container after 20 seconds
(mounted as /etc/webhook/scripts/nginx1-remove)

#!/bin/sh
set -e
sleep 20 && docker stop nginx1 && docker rm nginx1

With that in place, let's call this container activator and start it like:

docker run -d --name=activator \
-v /srv/activator:/etc/webhook \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
--network skynet \
test/activator -hooks=/etc/webhook/hooks.json -verbose -hotreload

For Traefik, configuration (dynamic part) is:

[http.routers]
  [http.routers.nginx1-http]
    entryPoints = ["web"]
    rule = "Host(\"nginx1.myservers.local\")"
    middlewares = ["auth-run"]
    service = "nginx1"

[http.middlewares]
  [http.middlewares.auth-run.forwardAuth]
    address = "http://activator:9000/hooks/nginx1"
[[http.services.nginx1.loadBalancer.servers]]
  url = "http://nginx1/"

The execution

With all of the above set, and given that dns entries are correct and all containers are on the same network once the http://nginx1.myservers.local is opened in a browser, the following will happen:

Browser will send a request to the Traefik
Traefik will, as a part of the authorization, call the nginx1 hook
Webhook will run a shell script
Shell script will start docker container and start remove script detached
If starting a shell script was successful, webhook will return code 200 to Traefik
Traefik will accept that as a authorization confirmation and pass the request to the, now running, nginx1 container
Nginx will respond with a welcome page
Welcome page (from nginx) will be displayed in a browser
After 20 seconds, nginx1 container will be removed
It works!

In a scenario where something wrong happens with a script (for example, a subsequent request within the 20sec period will fail to start a container with the same name while it is already running), webhook will not return 200, Traefik's authentication sequence will fail and browser will get unauthorized error.

In a way this is exploiting ForwardAuth middleware for something totally different, but it does the trick.

The application

Where I see this idea fit is for intermittent web hooks that do some processing in the background. For example, when a web hook is triggered, response is sent right away (confirmation that data is received), while the processing continues in the background, and once complete, container can shut down.

Reload cron jobs in an Alpine linux container

Miro — Sat, 25 Jul 2020 14:42:44 +0000

TL;DR

After updating /etc/crontabs/root , putting an empty cron.update file in the /etc/crontabs will update directory's last modification, cron daemon will reload jobs from the files in the /etc/crontabs and cron.update file will be deleted.

A bit more about why and how

In a situation where container's crontabs directory (/etc/crontabs) is mounted from a host, just editing file that container sees as a /etc/crontabs/root file outside of a container will not trigger a cron daemon to reload jobs.

Looking at the busybox source code, there is a comment saying

The file 'cron.update' is checked to determine new cron jobs. The directory is rescanned once an hour to deal with any screwups.

Additionally, the code is checking if the /etc/crontabs directory's last modification time changed which triggers a job update. To update last modification time of a directory, something should be added, removed or renamed inside that directory (excluding subdirectories). For example, adding a random file put into the /etc/crontabs will update last modification time which will trigger reloading jobs in the next cycle (the next minute). This is good, since container doesn't have to be restarted for new jobs to take effect. The downside is a lack of feedback that this really happened.

So, instead of a random file, a file named cron.update can be used to get a confirmation that jobs have been reloaded. File is used by crontab to notify cron daemon that a cron file has been updated and a cron daemon will delete cron.update once it reloads the jobs, meaning that if the file is gone, jobs are reloaded. Crontab writes the changed file name into the cron.update file, but since adding a file updates directory's last modification time, an empty file will do the trick.

Quickly create a self signed certificate for a custom domain

Miro — Mon, 13 Apr 2020 20:03:55 +0000

I often find myself needing a simple self signed certificate that is either for a wildcard domain or that cover multiple domain names. And then, even when I have openssl somewhere the questions are "do I have the config file?", "do I really need to fill in all those distinguished name values?", "where do I put additional domains", etc.

So to make my life simpler I made a small batch script that takes domain names as arguments, creates configuration file and instructs openssl to create a self signed certificate for those domains.

Example usage

Requesting certificates like:

E:\Temp> self-signed-cert mydomain.local *.mydomain.local

will generate one certificate with 2 domain names (mydomain.local and *.mydomain.local) and will produce the following files:

mydomain.local.conf - configuration file for openssl
mydomain.local.key - private key
mydomain.local.crt - certificate
mydomain.local.pfx - a pkcs#12 file with both certificate and private key

Content of the generated mydomain.local.conf file:

[req]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = req_distinguished_name
x509_extensions = v3_req

[req_distinguished_name]
CN = mydomain.local

[v3_req]
subjectAltName = @san

[san]
DNS.1 = mydomain.local
DNS.2 = *.mydomain.local

And the generated certificate looks like this:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            fc:55:a1:88:a5:68:67:0c
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = mydomain.local
        Validity
            Not Before: Apr 13 12:12:57 2020 GMT
            Not After : Apr 11 12:12:57 2030 GMT
        Subject: CN = mydomain.local
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
                    00:cb:d9:5c:bb:df:23:1e:69:5d:b2:45:06:d8:1f:
                    d9:d1:4a:37:07:ab:c7:86:77:e0:2f:88:36:dc:f4:
                                      <...>
                    7d:70:b6:23:aa:9a:bc:57:83:c9:f8:13:3f:9f:b4:
                    01:54:71
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name: 
                DNS:mydomain.local, DNS:*.mydomain.local
    Signature Algorithm: sha256WithRSAEncryption
         b0:7b:4c:2c:f6:5e:40:2b:9d:9d:71:b1:d3:f8:0b:44:84:50:
         d1:ea:b0:91:fb:61:7d:d8:f8:d9:ca:5d:c8:b2:bf:d4:5d:75:
                               <...>
         5c:73:66:d1:52:10:11:2b:68:85:da:8a:16:3d:82:4f:0c:c5:
         50:a5:85:81:4f:06:27:33

Batch file explained

if [%1]==[] (
  echo Usage: %0 ^<domain_name^> [additional_domain] [additional_domain] ...
  exit /b 1
)

Just making sure that there is at least one argument

if [%2]==[] (set addsan=) else (set addsan=1)

Checking if it is a single domain name or multiple domain name

set fn=%~1
if [%fn:~0,2%] == [*.] set fn=_wildcard.%fn:~2%

Setting a file name for configuration and certificates. If the (first) domain name in the list is a wildcard domain, replace the asterisk character with "wildcard" word. Otherwise just take the domain name. So requesting a *.mydomain.local certificates will generate wildcard.mydomain.local files, while requesting web1.mydomain.local certificate will generate web1.mydomain.local files.

%~1 - removing surrounding quotes around first argument. E.g. if the first argument is "something with spaces", the %~1 will remove the quotes.
%fn:~0,2% - substring, first two characters of the fn variable
%fn:~2% - substring, everything after 2nd character

echo [req]>%fn%.conf
echo default_bits = 2048>>%fn%.conf
echo prompt = no>>%fn%.conf
echo default_md = sha256>>%fn%.conf
echo distinguished_name = req_distinguished_name>>%fn%.conf
if [%addsan%] == [1] echo x509_extensions = v3_req>>%fn%.conf
echo.>>%fn%.conf
echo [req_distinguished_name]>>%fn%.conf
echo CN = %~1>>%fn%.conf
echo.>>%fn%.conf

Making the openssl configuration, named %fn%.conf.

if not [%addsan%] == [1] goto :makecert

Skip additional configuration if there is only one domain

echo [v3_req]>>%fn%.conf
echo subjectAltName = @san>>%fn%.conf
echo.>>%fn%.conf
echo [san]>>%fn%.conf
set /a sanid = 0

Additional configuration sections for multiple domain names.

:sanloop
set /a sanid+=1
echo DNS.%sanid% = %~1>>%fn%.conf
shift
if [%~1]==[] goto :makecert
goto :sanloop

Looping through the arguments.
set /a sanid+=1 - increment counter so that each DNS entry has it's own number
shift - shift arguments so that %2 becomes %1, %3 becomes %2, etc
if [%~1]==[] goto :makecert - once there are no more arguments, break the loop

:makecert
openssl req -new -x509 -nodes -days 3650 -sha256 -newkey rsa:4096 -keyout %fn%.key -out %fn%.crt -config %fn%.conf

openssl pkcs12 -export -out %fn%.pfx -inkey %fn%.key -in %fn%.crt -name %friendly% -passout pass:

Finally, the openssl call. Frist create a new self signed certificate which is valid for 10 years, and then export it to pkcs#12 pfx file with an empty password.

Make sure that openssl is somewhere in PATH.

self-signed-cert.cmd

Simple Docker HTTP socket certificates

Miro — Thu, 09 Apr 2020 21:00:29 +0000

I have a couple of Docker servers and I wanted to be able to manage them remotely. Reading about it, adding -H tcp://0.0.0.0:2375 argument when starting docker daemon was simple enough solution but it also came with a warning that it is not secure.

In addition, there is a Protect the Docker daemon socket guide which covers how to expose http socket in a secure manner but it is also marked as an "Advanced topic" so I decided to make a simple shell script out of it.

Tested on a clean Debian 10 after Docker installation. Full docker-certs.sh is at the bottom of the post.

Usage

docker-cert.sh [options]
OPTIONS:
  --all           Same as --ca --server --client --service combined, default behaviour
  --ca            Creates only self-signed Root CA
  --server        Creates a server certificate, requires CA to be created first
  --client        Creates a client certificate with a default name, requires CA to be created first
  --client=Name   Creates a client certificate with provided name, requires CA to be created first
  --service       Updates /lib/systemd/system/docker.service with socket listener and tls verification

Note: all files will be created in /etc/docker/certs dir

Running it without any options like sudo docker-cert.sh will execute default behaviour, same as with the --all option. Script will do the following:

Check if /etc/docker/certs directory exists, create if not
Create a self-signed root CA with a common name Docker ($FQDN) CA. FQDN is hostname plus the first suffix from /etc/resolv.conf search line, or just hostname if search line is not found.
Create a server certificate with a common name Docker ($FQDN) Server, signed by the root CA
Create a client certificate with a common name Docker ($FQDN) Client, signed by the root CA
Update /lib/systemd/system/docker.service to append socket listener and certificates to the start command. Like the following:

-H tcp://0.0.0.0:2376
--tlsverify
--tlscacert=/etc/docker/certs/ca.pem
--tlscert=/etc/docker/certs/server.pem
--tlskey=/etc/docker/certs/server.key

Reloading systemctl daemon, systemctl daemon-reload
Restart docker service, systemctl restart docker

There are checks in place so if a file that is to be created already exist, script will exit. In the same way if docker.service contains -H tcp:// in the ExecStart line, script will exit.

Additional clients

Additional clients can be created afterwards by running sudo docker-cert.sh --client=Another which will create client_another.key and client_another.pem files, and common name will be Docker ($FQDN) Client Another

Accessing remotely from Windows

Once certificates are created and docker daemon is running, it is time to test connectivity from a client windows machine. Tutorial suggests

docker --tlsverify --tlscacert=ca.pem --tlscert=cert.pem --tlskey=key.pem \
  -H=$HOST:2376

or setting environment variables like DOCKER_HOST or DOCKER_CERT_PATH for a directory where certificates are stored.

First approach looks cumbersome to type every time, and second approach doesn't work for multiple servers with different certificates.

Batch file

I opted to create a small batch file to interact with docker.

Let's say that my docker server is named dock1, domain suffix is local. I've created certificates using the script so I have Docker (dock1.local) Server certificate.

After that I copied ca.pem, client.key, client.pem from server to my windows machine into the %USERPROFILE%\.docker\dock1 folder.

Created a batch file named dock1.bat with the following content:

@echo off
set DATADIR=%USERPROFILE%\.docker\%~n0
if exist "%DATADIR%\host.var" (set /p DOCKERHOST=<"%DATADIR%\host.var") else (set DOCKERHOST=%~n0)
docker --tlsverify --tlscacert="%DATADIR%\ca.pem" --tlscert="%DATADIR%\client.pem" --tlskey="%DATADIR%\client.key" -H=%DOCKERHOST%:2376 %*

and put that batch file somewhere in the PATH.

(broken into multiple lines for readability):

set DATADIR=%USERPROFILE%\.docker\%~n0

if exist "%DATADIR%\host.var" (
  set /p DOCKERHOST=<"%DATADIR%\host.var"
) else (
  set DOCKERHOST=%~n0
)

docker
--tlsverify
--tlscacert="%DATADIR%\ca.pem"
--tlscert="%DATADIR%\client.pem"
--tlskey="%DATADIR%\client.key"
-H=%DOCKERHOST%:2376
%*

So when I run dock1 info from the command line that translates into the following command (broken into multiple lines for readability):

docker
--tlsverify
--tlscacert="C:\Users\user\ca.pem"
--tlscert="C:\Users\user\client.pem"
--tlskey="C:\Users\user\client.key"
-H=dock1:2376
info

and running it displays docker system-wide information.

It works for all docker commands, likedock1 images, dock1 ps, dock1 run .... etc.

More fun with batch file

Ok, so the batch file above has some additional lines and variables. Let me explain them:

%~n0 - this is the name of the file without extension. So if have file named dock1.bat it will have dock1 value. If I name it dock365.bat it will have dock365 value.

set DATADIR=%USERPROFILE%\.docker\%~n0 - I'm using it to set the path where certificates are stored to a folder named the same as the batch file within my user profile > .docker folder.

if exist "%DATADIR%\host.var" (set /p DOCKERHOST=<"%DATADIR%\host.var") else (set DOCKERHOST=%~n0) - if hostname of docker server is different from the batch file name, this allows me to put real hostname into the file which will be read into the variable.
So if host.var file does not exist, %DOCKERHOST% variable will contain name of the batch file without extension (dock1 for example). But if I want to access docker using dock1.local hostname, I can put dock1.local into the host.var file and then the %DOCKERHOST% variable will contain dock1.local value.

set /p DOCKERHOST=<"%DATADIR%\host.var" - set variable to a value being read from the input file.

%* - passing all arguments that batch file received.

What this batch script allows me is to have one txt file, let's name it docker_batch.txt, instead of the batch file. And then a symbolic link for each docker server I have. Created like this:

mklink dock1.bat docker_batch.txt
mklink dock365.bat docker_batch.txt

Pretty cool.

docker-certs.sh

SED - Update line that starts with but does not contain

Miro — Thu, 09 Apr 2020 18:26:28 +0000

I wanted to update my docker.service file to include tls certificates in the start command. The idea was to search for the line that starts with ExecStart= but also not to change anything if that line already contains tcp header -H tcp://. It turned out to be a bit more complicated than expected.

Result

sed -i '\|^ExecStart=.*-H tcp://|! s|^ExecStart=.*|\0 -H tcp://0.0.0.0:2376 --tlsverify --tlscacert='$PWD'/ca.pem --tlscert='$PWD'/server.pem --tlskey='$PWD'/server.key|' /lib/systemd/system/docker.service

Broken in multiple lines for readability

sed -i '\|^ExecStart=.*-H tcp://|!
        s|^ExecStart=.*|\0 -H tcp://0.0.0.0:2376 --tlsverify
        --tlscacert='$PWD'/ca.pem --tlscert='$PWD'/server.pem
        --tlskey='$PWD'/server.key|' /lib/systemd/system/docker.service

In a nutshell, it excludes all lines that start with ExecStart= and contain -H tcp:// and out of remaining lines, if a line starts with ExecStart=, it appends additional arguments to the end of the line.

Explanation

sed -i means that we will change a file in place.

\|A|! s|S|R| - using pipes instead of slashes makes it easier when dealing with paths. Note that syntax starts with a backslash when using special delimiters with addresses (A part).
Using standard slashes would be like /A/! s/S/R/, without leading backslash.

\|^ExecStart=.*-H tcp://|! - using the addresses in sed, exclude lines that start with ExecStart= and have -H tcp:// down the line.

s|^ExecStart=.*|\0 -H tcp://0.0.0.0:2376 ... - search for a line that starts with ExecStart= and grab the rest of the line using .* . In the replace part, \0 will hold the complete line match and the rest is additional text that is appended to the line.

/lib/systemd/system/docker.service - file that is being edited

Conclusion

For a single line change, a file can be checked if the update is actually required before updating it, and then the exclusion part is not needed at all. But if there are more lines that require similar update this could do the trick.

Installing Windows on VHD to use with Hyper-V

Miro — Fri, 17 Jan 2020 16:38:32 +0000

One PowerShell script that I've been using for a long time to prepare disk and install Windows for Hyper-V VMs. Since Microsoft released Windows Sandbox I'm not using it as often because sandbox starts faster and for simple application testing that's good enough. But every now and then I need a full VM and here you can find the script I'm using.

It takes a few of minutes to complete and then a few more minutes for windows setup to complete the first boot. Compared that to manual installation, faster and compared to Hyper-V "Install an OS from a bootable CD/DVD" it's more flexible.

It does the following:

Mounting Installation ISO file
Presenting a menu to choose edition to install
Creating virtual hard disk
Installing windows using DISM tool
Setting up boot configuration
Copying unattend.xml
Cleaning up

Example Usage

.\Prepare-VirtualWinVhd.ps1
 -VhdFile "TestWin.vhdx"
 -WinIso "en_windows_10_multi-edition_version_1709_updated_dec_2017_x64_dvd_100406711.iso"
 -WinEdition "Windows 10 Enterprise"
 -VhdSize 60GB
 -EfiLetter R
 -VirtualWinLetter W
 -UnattendFile ".\unattend.xml"

Script explained

Some defaults for the script. Installing Windows 10 Enterprise on a 40GB VHD. We will be using drive letter R for Efi partition and W for Windows partition. Make sure to pick letters are not assigned on your current windows installation. In addition to the installation, the unattend.xml file will also be copied to the VHD.

param (
  [string]$WinIso,
  [string]$VhdFile,
  [string]$WinEdition = "Windows 10 Enterprise",
  [long]$VhdSize = 40GB,
  [char]$EfiLetter = 'R',
  [char]$VirtualWinLetter = 'W',
  [string]$UnattendFile = "unattend.xml"
)

In the very beginning of the script we are setting $ErrorActionPreference to Stop so that it stops on any error. In addition we need to make sure that script is run with elevated privileges. Both DISM and Diskpart require elevated privileges:

$ErrorActionPreference = 'Stop'

if(-not [bool]([Security.Principal.WindowsIdentity]::GetCurrent().Groups -contains 'S-1-5-32-544')) {
  Write-Error "Script must be started as an Administrator"
}

Mounting Installation ISO file

Mounting iso file using Mount-DiskImage, and getting the drive letter of mounted image

$WinIso = Resolve-Path $WinIso
Write-Progress "Mounting $WinIso ..."
$MountResult = Mount-DiskImage -ImagePath $WinIso -StorageType ISO -PassThru
$DriveLetter = ($MountResult | Get-Volume).DriveLetter
if (-not $DriveLetter) {
  Write-Error "ISO file not loaded correctly" -ErrorAction Continue
  Dismount-DiskImage -ImagePath $WinIso | Out-Null
  return
}
Write-Progress "Mounting $WinIso ... Done"

Presenting a menu to choose edition to install

Once iso is mounted, we are using dism tool to enumerate all windows editions stored in install.wim file.

$WimFile = "$($DriveLetter):\sources\install.wim"
Write-Host "Inspecting $WimFile"
$WimOutput = dism /get-wiminfo /wimfile:`"$WimFile`" | Out-String

Output of dism /get-wiminfo command looks like this:

C:\WINDOWS\system32>dism /get-wiminfo /wimfile:f:\sources\install.wim

Deployment Image Servicing and Management tool
Version: 10.0.18362.1

Details for image : f:\sources\install.wim

Index : 1
Name : Windows 10 Education
Description : Windows 10 Education
Size : 14.780.927.379 bytes

Index : 2
Name : Windows 10 Education N
Description : Windows 10 Education N
Size : 13.958.508.090 bytes

Index : 3
Name : Windows 10 Enterprise
Description : Windows 10 Enterprise
Size : 14.781.260.269 bytes

...

So parsing it with regex we want to extract Index and Name

$WimInfo = $WimOutput | Select-String "(?smi)Index : (?<Id>\d+).*?Name : (?<Name>[^`r`n]+)" -AllMatches
if (!$WimInfo.Matches) {
  Write-Error "Images not found in install.wim`r`n$WimOutput" -ErrorAction Continue
  Dismount-DiskImage -ImagePath $WinIso | Out-Null
  return
}

And build the menu selecting the Edition that was passed to the script as a parameter just to make things easier. In addition we are checking that selected value is valid.

$Items = @{ }
$Menu = ""
$DefaultIndex = 1
$WimInfo.Matches | ForEach-Object { 
  $Items.Add([int]$_.Groups["Id"].Value, $_.Groups["Name"].Value)
  $Menu += $_.Groups["Id"].Value + ") " + $_.Groups["Name"].Value + "`r`n"
  if ($_.Groups["Name"].Value -eq $WinEdition) {
    $DefaultIndex = [int]$_.Groups["Id"].Value
  }
}

Write-Output $Menu
do {
  try {
    $err = $false
    $WimIdx = if (([int]$val = Read-Host "Please select version [$DefaultIndex]") -eq "") { $DefaultIndex } else { $val }
    if (-not $Items.ContainsKey($WimIdx)) { $err = $true }
  }
  catch {
    $err = $true;
  }
} while ($err)
Write-Output $Items[$WimIdx]

It will present a menu like this:

1) Windows 10 Education
2) Windows 10 Education N
3) Windows 10 Enterprise
4) Windows 10 Enterprise N
5) Windows 10 Pro
6) Windows 10 Pro N
7) Windows 10 Pro Education
8) Windows 10 Pro Education N
9) Windows 10 Pro for Workstations
10) Windows 10 Pro N for Workstations

Please select version [3]:

Creating virtual hard disk

Now that we know what we want to install, it is time to prepare disk for installation. First we ensure that $VhdFile is full path, expanding it as needed, and that the file does not exists. If filename was not given as a parameter we are using Windows edition name.

If (!$VhdFile) {
  $VhdFile = $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath($Items[$WimIdx] + ".vhdx")
}else{
  $VhdFile = $ExecutionContext.SessionState.Path.GetUnresolvedProviderPathFromPSPath($VhdFile)
}

if (Test-Path $VhdFile) {
  Write-Error "Vhd file '$VhdFile' allready exists." -ErrorAction Continue
  Dismount-DiskImage -ImagePath $WinIso | Out-Null
  return
}

Then we create a new VHD using New-VHD command, mount it and get the disk number used in diskpart.

$disk = New-VHD $VhdFile $VhdSize
Mount-DiskImage -ImagePath $VhdFile
$disknumber = (Get-DiskImage -ImagePath $VhdFile | Get-Disk).Number

Additionally just as an precaution, make sure that disk number is not 0 which is usually main system disk

if ($disknumber -eq 0) {
  Write-Error "Unexpected disk number" -ErrorAction Continue
  Dismount-DiskImage -ImagePath $VhdFile | Out-Null -ErrorAction Continue
  Dismount-DiskImage -ImagePath $WinIso | Out-Null -ErrorAction Continue
  return
}

To partition disk we are using a list of commands piped to diskpart. In addition we are assigning known letters to both EFI partition to configure boot record, and to partition for windows installation (future C drive)

@"
select disk $DiskNumber
convert gpt
select partition 1
delete partition override
create partition primary size=300
format quick fs=ntfs
create partition efi size=100
format quick fs=fat32
assign letter="$EfiLetter"
create partition msr size=128
create partition primary
format quick fs=ntfs
assign letter="$VirtualWinLetter"
exit
"@ | diskpart | Out-Null

Installing windows using DISM tool

Once the disk is ready we can apply image (install Windows) using dism tool. Since we are using all those variables it makes sense to build command line as a string and run it using Invoke-Expression

Invoke-Expression "dism /apply-image /imagefile:`"$WimFile`" /index:$WimIdx /applydir:$($VirtualWinLetter):\"

Setting up boot configuration

And then we make disk bootable

Invoke-Expression "$($VirtualWinLetter):\windows\system32\bcdboot $($VirtualWinLetter):\windows /f uefi /s $($EfiLetter):"
Invoke-Expression "bcdedit /store $($EfiLetter):\EFI\Microsoft\Boot\BCD"

Copying unattend.xml

If there was unatted.xml file set, make sure to copy it so Windows can be configured on the first boot.

if($UnattendFile) {
  Write-Host "Copying unattended.xml"
  New-Item -ItemType "directory" -Path "$($VirtualWinLetter):\Windows\Panther\" | Out-Null
  Copy-Item $UnattendFile "$($VirtualWinLetter):\Windows\Panther\unattend.xml" | Out-Null
}

Cleaning up

In the end we just remove all assigned letters and unmount everything

@"
select disk $disknumber
select partition 2
remove letter="$EfiLetter"
select partition 4
remove letter="$VirtualWinLetter"
exit
"@ | diskpart | Out-Null
Dismount-DiskImage -ImagePath $VhdFile | Out-Null
Dismount-DiskImage -ImagePath $WinIso | Out-Null

Unattend.xml

Simple unattend.xml that just sets user (name user, password login) and skips all installation questions. There are many more options to set. You can find more from Microsoft Answer files (unattend.xml) or use online tool Windows Answer File Generator

<?xml version="1.0" encoding="utf-8"?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
  <settings pass="oobeSystem">
    <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35"
               language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
      <AutoLogon>
        <Password>
          <Value>login</Value>
          <PlainText>true</PlainText>
        </Password>
        <Enabled>true</Enabled>
        <Username>user</Username>
      </AutoLogon>
      <OOBE>
        <HideEULAPage>true</HideEULAPage>
        <HideOEMRegistrationScreen>true</HideOEMRegistrationScreen>
        <HideOnlineAccountScreens>true</HideOnlineAccountScreens>
        <HideWirelessSetupInOOBE>true</HideWirelessSetupInOOBE>
        <NetworkLocation>Home</NetworkLocation>
        <SkipUserOOBE>true</SkipUserOOBE>
        <SkipMachineOOBE>true</SkipMachineOOBE>
        <ProtectYourPC>1</ProtectYourPC>
      </OOBE>
      <UserAccounts>
        <LocalAccounts>
          <LocalAccount wcm:action="add">
            <Password>
              <Value>login</Value>
              <PlainText>true</PlainText>
            </Password>
            <Description></Description>
            <DisplayName>user</DisplayName>
            <Group>Administrators</Group>
            <Name>user</Name>
          </LocalAccount>
        </LocalAccounts>
      </UserAccounts>
      <RegisteredOrganization></RegisteredOrganization>
      <RegisteredOwner></RegisteredOwner>
      <DisableAutoDaylightTimeSet>false</DisableAutoDaylightTimeSet>
      <TimeZone>GMT Standard Time</TimeZone>
    </component>
  </settings>
</unattend>

Final script

Downloading GAL as individual vCards using OWA

Miro — Sun, 12 Jan 2020 19:30:21 +0000

Some time ago, a friend asked me about syncing Exchanges's GAL (Global Address List) with a mobile phone. So I started looking into Exchange Web Services, but as it turns out I need GAL id in order to query it. And to get it I need either more privileged access to Office 365 tenant or snoop OWA's traffic as described in the article: https://gsexdev.blogspot.com/2013/05/using-ews-findpeople-operation-in.html

Looking at the network log in a browser I figured that I could effectively do everything within a browser ran from Puppeteer and send it to Radicale CardDAV server.

But as it turned out, one required field, "Pager", was not present in Exchange's Persona object. So good old "copy everything from GAL into Contacts" worked, at least for one-off sync, and this idea was abandoned.

Here is JavaScript PoC that can be used to download GAL as vCard files. Bear in mind that it doesn't have any error handling and might not work in every browser. It works in Chrome. Generated vCards are not 100% by specification since I'm serializing whole persona object into it, but I haven't had any issue opening it in Windows (People app), Outlook or on iOS/Android phones.

Before posting this I just cleaned up code a bit and replaced XHR with fetch.

How to use in 5 steps:

Open https://outlook.office365.com/people/ in a browser and sign in with your O365 account
Open browser's console
Paste this script
Run one of the functions, suggestion is to test with OwaGalExtractor.downloadGal(null, 5)
Look at a bunch of file being downloaded (chrome might ask you to allow multiple file download)

Making Dynamics 365 CE reports available in UCI on iOS devices

Miro — Sat, 11 Jan 2020 00:06:13 +0000

Some time ago, working on a Dynamics 365 CE project, I wanted to display reports within the app on an iOS device. Originally posted in May 2019 (https://gist.github.com/milolav/14fc9a27e0d3ebc8f15ae992dff9f182)
but looks like dev.to is a better place to share ideas, so here goes...

At the time of writing, Microsoft does not officially support running reports in an app on iOS / Android / Windows devices in the Unified Interface (UCI), but as it turns out, it's actually possible to display a report within the app using SiteMap entries pointing to Web Resources.

Reports are just another url in Dynamics and it seems the only reason why is that not working in the app is because every time you click on a item in the reports view it opens it in a new window.

First Idea

The first idea was just to use url of specific report, put it into Site Map like:

/crmreports/viewer/viewer.aspx?action=filter&helpID=User Summary.rdl&id={e5838060-8866-e911-a819-000d3ab29ccc}&appid=3ef28a1b-5666-e911-a9c5-000d3ab78b73

The problem is that menu items with this url do not show in the menu.

Web Resource to the rescue

Second idea was just to create a simple web resource and put in the following code:

<html>
  <body>
    <iframe src="/crmreports/viewer/viewer.aspx?action=filter&helpID=User Summary.rdl&id={e5838060-8866-e911-a819-000d3ab29ccc}" style="height: 100%; width: 100%;" frameborder="0"></iframe>
  </body>
</html>

And that worked!

Since it is not preferable to to have a separate web resource for every report next challenge was to pass data to that web resource. Microsoft provides sample on how to pass data to web resources outlined here:

https://docs.microsoft.com/en-us/dynamics365/customer-engagement/developer/sample-pass-multiple-values-web-resource-through-data-parameter

What's more interesting, passing data parameter also works in the Site Map. Not through the visual editor in Dynamics, but it does work by directly editing the SiteMap XML or using SiteMap Editor in XRMToolBox. Just adding data parameter after the web resource name like:

$webresource:new_reportwrapper.html?data=mydatavalue1

Which means that report guid value can be passed like this:

$webresource:new_reportwrapper.html?data=e5838060-8866-e911-a819-000d3ab29ccc

And the only thing left is to make use of passed value.

Code

Web Resource:

<html><body style="background: white; position: absolute; margin: 0.5em; top: 0; left: 0; right: 0; bottom: 0;">
<script>
function loadReport() {
  if (location.search != "") {
    params = location.search.substr(1).split("&");
    for (p in params) {
      var kv = params[p].replace(/\+/g, " ").split("=");
      if(kv[0].toLowerCase() === "data") {
        var guid = kv[1];  
        var url = "/crmreports/viewer/viewer.aspx?id=" + guid;
        document.getElementById("report").src = url;
        return;
      }
    }
  }
  document.getElementById("msg").innerHTML = "Report parameters not set!"; 
}

document.onreadystatechange = function () {  
  if (document.readyState == "complete") {  
    document.getElementById("report").onload = function() {
        document.getElementById("loading_overlay").style.display = "none";
    }
    loadReport();  
  }  
}
</script>
<iframe id="report" src="" style="height: 100%; width: 100%;" frameborder="0"></iframe>
<div id="loading_overlay">
<div style="background: none repeat scroll 0 0 black; position: fixed; display: block; opacity: 0.5; left: 0; top: 0; height: 100%; width: 100%;"></div>
<p id="msg" style="position: absolute; top: 50%; left: 50%; transform: translateX(-50%) translateY(-50%);">LOADING ...</p>
</div>
</body>
</html>

Site Map item:

<SubArea
  Id="tempId_636927761987954911"
  Url="$webresource:new_reportwrapper.html?data=e5838060-8866-e911-a819-000d3ab29ccc"
  Title="User Summary Report"
  AvailableOffline="false"
  PassParams="false"
  Client="All"
  Sku="All"
/>

Final result

Additional stuff

This approach can also be used for getting diag tool inside the app by creating a web resource with the following code and linking that in the site map.

<html>
  <body>
    <iframe src="/tools/diagnostics/diag.aspx" style="height: 100%; width: 100%;" frameborder="0"></iframe>
  </body>
</html>

Which allows you to access it from within the app rather than typing or pasting this in a browser.

Tracking Service Updates for MS Dynamics 365 CE in calendar

Miro — Sat, 11 Jan 2020 00:06:00 +0000

Another script from a couple of months ago

With Microsoft pushing "Service Update XX" almost every week it became hard to track down when particular update is going to be released.

Powershell script uses application registered in AAD with privileges to read messages from Office 365 Communication API and access D365 to create appointments. Appointments are set category and subcategory to identify unique messages without customizations. Recipients of the appointment are defined as a list of activity parties.

DEV Community: Miro

Oauth2 certificate authentication in bash script

A bit of explanation

Microsoft Graph

Google APIs

Code

WSL2 Containers

Using differencing disks

Just testing

Conclusion

Manually installing WSL2 distributions

Start-up scripts

Registry keys

Other distributions

Official install.tar.gz

Differencing disks

BlockSize effect on a dynamic disk file size

The end

Socket activation of containers using Traefik's ForwardAuth

The setup

The execution

The application

Reload cron jobs in an Alpine linux container

A bit more about why and how

Quickly create a self signed certificate for a custom domain

Example usage

Batch file explained

self-signed-cert.cmd

Simple Docker HTTP socket certificates

Usage

Additional clients

Accessing remotely from Windows

Batch file

More fun with batch file

docker-certs.sh

SED - Update line that starts with but does not contain

Result

Explanation

Conclusion

Installing Windows on VHD to use with Hyper-V

Example Usage

Script explained

Mounting Installation ISO file

Presenting a menu to choose edition to install

Creating virtual hard disk

Installing windows using DISM tool

Setting up boot configuration

Copying unattend.xml

Cleaning up

Unattend.xml

Final script

Downloading GAL as individual vCards using OWA

Making Dynamics 365 CE reports available in UCI on iOS devices

First Idea

Web Resource to the rescue

Code

Final result

Additional stuff

Tracking Service Updates for MS Dynamics 365 CE in calendar

Official `install.tar.gz`