DEV Community: Abhishek

Run Airflow on Windows without Virtualization

Abhishek — Wed, 12 Feb 2025 12:31:07 +0000

Setting up Podman on Windows with MSYS2

This guide walks through the process of setting up Podman on Windows using MSYS2, including the installation of required tools and configuration steps.

Prerequisites

1. Install MSYS2

Download and install MSYS2 from the official website:

MSYS2 Installation Guide

2. Update MSYS2

Update the package database and base packages:

pacman -Syuu

Run this command multiple times until no updates are available.

Required Tools Installation

3. Install SSH

pacman -S openssh

4. Install Podman and Compose

pacman -S mingw-w64-x86_64-podman
pacman -S mingw-w64-x86_64-podman-compose

5. Install Astro CLI

Download the Windows x64 binary from Astro CLI Releases
Place the binary in C:\msys64\usr\bin

Podman Configuration

6. Initialize Podman Machine

podman machine init

7. Configure Rootful Mode (Optional)

If you need rootful mode:

podman machine set --rootful

8. Start Podman Machine

podman machine start

9. Install Docker Credential Helper

Download the Windows AMD64 binary from Docker Credential Helpers Releases

Example: docker-credential-wincred-v0.8.2.windows-amd64.exe

Rename and copy the binary to C:\msys64\usr\bin\docker-credential-wincred.exe and C:\msys64\usr\bin\docker-credential-desktop.exe (this is purely because astro-cli is reverting back to dockerdesktop credential helper for starting local deployment)

10. Configure Container Authentication

Create the authentication configuration file:

mkdir -p ~/.config/containers
echo '{"credHelpers": {"docker.io": "wincred"}}' > ~/.config/containers/auth.json
export REGISTRY_AUTH_FILE=~/.config/containers/auth.json

Do Not Configure Docker Host

⚠️ WARNING: DO NOT SET THIS ENVIRONMENT VARIABLE ⚠️

The following environment variable setting can cause conflicts with Podman's functionality:

# DO NOT USE - Left here for reference only
export DOCKER_HOST=npipe:////./pipe/docker_engine

Podman has its own socket configuration and doesn't require Docker's pipe configuration.

Ready to Use

You can now use Astro CLI commands:

astro dev init - Initialize a new project
astro dev start - Start development environment
astro dev parse - Parse configurations
astro dev kill - Stop development environment

My Home Lab setup 💻

Abhishek — Sun, 03 Sep 2023 18:53:23 +0000

Here is my homelab setup that my brother and I created after so many horrific experiences.

The list consists of mostly software requirements.

Virtualization: Hyper-V (Because with Easy-GPU-PV I can split my GPU as much as possible.)

For VMs I use:

AtlasOS(It's a very trimmed down version of windows with almost 0 security)
Linux Lite

Anti-Virus: Kaspersky

Printer: PaperCut

Remote Desktop: Parsec

Media Server: EMBY

Media collection:

TV Shows: Sonarr
Movies: Radarr
Seedbox: Seedr
Torrent Client: Combination of deluge (for python automations) and qbit.
Indexer: ~~Jackett~~ switched to Prowlarr

I also do run an Airflow with few DAGs that take care of notifications to my telegram about Emby conversions and torrent downloads.

Tunneling: Twingate

Software Load Balancer: NPM
Static IP VPN: PureVPN
Proxy Server: ~~CCProxy~~ switched to Squid

For Local text sharing: Ssavr.com

Daily Driver: Windows 11 Pro, MacOS Sonoma, ~~Pop!OS~~ elementaryOS

Internet Load Balancer: Tplink ER605
re-flashed with OpenWRT. [credits]

Domain Service: Google Domains

========================================================

Problems I faced:

I started with Proxmox, but migrations and handling storage were becoming a bit of a hassle. GPU passthrough was good but needed another Windows with Hyper-V to split even further. When I discovered AtlasOS, it had issues with Hyper-V. I started sponsoring to boost the development. Now there are no issues with the hypervisor, and I'm happily able to run as many GPU splits.
I tried the many solutions for tunneling from this repo awesome-tunneling. But none of if it was enough and finally settled on Tailscale (uses WireGuard) before Twingate. Twingate was a game changer, is miles better than any solution available.
For Media server, I tried Plex, even took the Plex pass. But the support is not good. Sometimes subtitles won't render even if we tried to burn. Also, my biggest complaint was the exposing to internet. I don't have static IP because I use 2x1Gbps connection load balanced. So, it becomes difficult to manage. Plex doesn't work well under NAT networks. Then I tried Jellyfin which works like a breeze with routing through my static IP VPN. But Jellyfin is just not practical to pay for. Has so many playback related issues. So landed on Emby and it works nicely enough. Also took the premium.
I do have a local torrent client which I can route through VPN, but it takes a heavy toll to performance. Have been using seedr for years now. Love the speed. The premium version for API support is a bit costly, so need a little extra work to download from Sonarr/Radarr to seedr. Still looking to improve this. Adding Airflow DAGs were a huge success.
NgnixProxyManager works like a charm with Let's encrypt and GoogleDNS.
TPlink's firmware is trash and was unable to handle the multi WAN setup. It was only able to fallback with a significant downtime. Flashing ER605 took a few minutes, but is a massive performance boost.

WSL change LocalState path

Abhishek — Fri, 24 Mar 2023 06:05:21 +0000

List the ditros you have wsl -l -v
Change the directory to a folder with space to export the distro and then use command: wsl --export <Distro-name> <filename.tar>
Unregister the old distro wsl --unregister <Distro-name>
Import again with a different install path. The import command syntax is :

    --import <Distro> <InstallLocation> <FileName> [Options]
        Imports the specified tar file as a new distribution.
        The filename can be - for standard input.

        Options:
            --version <Version>
                Specifies the version to use for the new distribution.

            --vhd
                Specifies that the provided file is a .vhdx file, not a tar file.
                This operation makes a copy of the .vhdx file at the specified install location.

Comand to import:
wsl --import <Distro-name> <InstallLocation> <filename.tar>

Additional
Sample .wslconfig to be placed in User home

# Settings apply across all Linux distros running on WSL 2
[wsl2]
# Limits VM memory to use no more than 4 GB, this can be set as whole numbers using GB or MB
# memory=4GB
# Sets the VM to use two virtual processors
# processors=2
# Specify a custom Linux kernel to use with your installed distros. The default kernel used can be found at https://github.com/microsoft/WSL2-Linux-Kernel
# kernel=C:\\temp\\myCustomKernel
# Sets additional kernel parameters, in this case enabling older Linux base images such as Centos 6
# kernelCommandLine = vsyscall=emulate
# Sets amount of swap storage space to 8GB, default is 25% of available RAM
swap=8GB
# Sets swapfile path location, default is %USERPROFILE%\AppData\Local\Temp\swap.vhdx
swapfile=D:\\WSL\\wsl-swap.vhdx
# Disable page reporting so WSL retains all allocated memory claimed from Windows and releases none back when free
# pageReporting=false
# Turn off default connection to bind WSL 2 localhost to Windows localhost
# localhostforwarding=true
# Disables nested virtualization
# nestedVirtualization=false
# Turns on output console showing contents of dmesg when opening a WSL 2 distro for debugging
debugConsole=false
kernelCommandLine = "sysctl.vm.swappiness=100"

Is python slow ?

Abhishek — Thu, 16 Jun 2022 05:10:04 +0000

Here are a few superchargers 🚀 for your python.

PYPY
- Plane simple easy to use but 🏎️fast
Numba
- Boost performance with JIT. Or better yet, have an NVIDIA GPU ? 😏💵
Pyjion
- Easy to use, small compiler. Increase performance of our 🐌 CPython.

Let's race now 😎

Mac OS Python build from source with optimizations and debugging

Abhishek — Fri, 08 Apr 2022 16:56:15 +0000

brew install openssl xz gdbm zlib
export LDFLAGS="-L/opt/homebrew/opt/openssl@3/lib -L/opt/homebrew/lib -L/opt/homebrew/opt/sqlite3/lib -L/opt/homebrew/opt/zlib/lib"
export CPPFLAGS="-I/opt/homebrew/opt/openssl@3/include -I/opt/homebrew/include -I/opt/homebrew/opt/sqlite3/include -I/opt/homebrew/opt/zlib/include"
export PKG_CONFIG_PATH="/opt/homebrew/lib/pkgconfig:/opt/homebrew/opt/openssl@3/lib/pkgconfig:/opt/homebrew/opt/sqlite3/lib/pkgconfig"

#configure python source
./configure --enable-optimizations --with-pydebug --with-openssl=$(brew --prefix openssl)

Manage homebrew on M1 mac with rosetta 2

Abhishek — Fri, 19 Nov 2021 16:23:09 +0000

If you don't have rosetta installed:

softwareupdate --install-rosetta

Install homebrew x86_64 :

arch -x86_64 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

Highly suggested to modify .zshrc to include :

alias ibrew='arch -x86_64 /usr/local/bin/brew'
alias mbrew='arch -arm64e /opt/homebrew/bin/brew'

Going forward if needed to install x86_64 based module, use:

ibrew install pypy3

Mac OS Big Sur CPPFLAGS and LDFLAGS

Abhishek — Tue, 07 Sep 2021 08:48:58 +0000

While building python packages, dependency development libraries are required. We can install the libraries from homebrew were not found by pip for some reason.
Export the path used by homebrew for "include" and "lib".

export CPPFLAGS=-I/opt/homebrew/include
export LDFLAGS=-L/opt/homebrew/lib

Macbook Flapdown (Lid) settings

Abhishek — Sun, 27 Jun 2021 13:45:35 +0000

You can use the command line utility pmset to change the behavior of the lid closed/open state.

To disable wake/sleep on lid state:

sudo pmset -a lidwake 0

To undo the last setting

sudo pmset -a lidwake 1

By default, closing the lid will put the Mac to sleep. Setting the lidwake 0 value will disable it meaning closing the lid will do nothing.

Expose docker api on Ubuntu

Abhishek — Sun, 23 May 2021 17:48:24 +0000

Change file /etc/default/docker

# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock"

Change file /lib/systemd/system/docker.service

EnvironmentFile=/etc/default/docker
ExecStart=/usr/bin/dockerd -H fd:// $DOCKER_OPTS

Run

sudo systemctl daemon-reload
sudo service docker restart

Run following to see open ports

lsof -i -P -n | grep LISTEN

If the port 2375 doesn't show up do a system reboot.

Citrix reciever for ubuntu add ssl certs

Abhishek — Tue, 16 Feb 2021 13:38:35 +0000

Just few commands:

sudo ln -s /usr/share/ca-certificates/mozilla/* /opt/Citrix/ICAClient/keystore/cacerts/
sudo c_rehash /opt/Citrix/ICAClient/keystore/cacerts/

Docker Desktop on Windows expose daemon to LAN

Abhishek — Wed, 10 Feb 2021 18:49:37 +0000

Steps :

Enable "Expose daemon on tcp://localhost:2375 without TLS" in settings.
From admin Command Prompt

netsh interface portproxy add v4tov4 listenport=2375 listenaddress=192.168.0.2 connectaddress=127.0.0.1 connectport=2375

NOTE : replace listenaddress to your LAN ip

On Remote machine Set DOCKER_HOST to tcp://192.168.0.2:2375