DEV Community: Jonathan Bowman

Linux on the Dell XPS: Fixing AX201 Wi-Fi performance

Jonathan Bowman — Wed, 06 Apr 2022 11:03:39 +0000

I am very happy with Linux on my Dell XPS 13 9310. I use the latest version of Fedora (35 at the time of this writing, soon to be upgraded to 36).

However, the Wi-Fi connection gave me great difficulty for months before I learned that performance is much better with the power saving functionality turned off. With power saving on, I would often lose packets right and left. This was a typical ping session:

[bowmanjd@lappy386 ~]$ ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=223 ms
64 bytes from 192.168.0.1: icmp_seq=4 ttl=64 time=1.15 ms
64 bytes from 192.168.0.1: icmp_seq=5 ttl=64 time=15.3 ms
64 bytes from 192.168.0.1: icmp_seq=6 ttl=64 time=1.56 ms
^C
--- 192.168.0.1 ping statistics ---
6 packets transmitted, 4 received, 33.3333% packet loss, time 5075ms
rtt min/avg/max/mdev = 1.147/60.182/222.714/94.010 ms

Note the first two dropped packets. Sometimes more, sometimes less. At first, I blamed our wireless access point, yet no other client device had this problem. Finally, I booted into Windows on the XPS and noticed much better wireless performance. Clearly, this is a driver issue of some sort.

There may be a variety of solutions for this, and I suspect this will one day be fixed in the kernel, but here is how I solved it for now.

The NetworkManager solution

For Linux distros using NetworkManager, add a new configuration file to the /etc/NetworkManager/conf.d/ directory. Name it something like disable_power_save.conf with the contents:

[connection]
wifi.powersave = 2

Elevate to root first using sudo or similar.

One command can do it all:

printf "[connection]\nwifi.powersave = 2\n" | sudo tee /etc/NetworkManager/conf.d/disable_power_save.conf

Then restart NetworkManager with:

sudo systemctl restart NetworkManager

Test the performance with ping, and there should be a marked improvement.

wifi.powersave config options

NetworkManager has several configurable options for wireless adapter settings, including powersave. You can read about them in the API documentation. There, we discover the following powersave options:

NM_SETTING_WIRELESS_POWERSAVE_DEFAULT (0) (use the globally configured value).
NM_SETTING_WIRELESS_POWERSAVE_IGNORE (1) (don't touch currently configured setting)
NM_SETTING_WIRELESS_POWERSAVE_DISABLE (2) (disable Wi-Fi power saving)
NM_SETTING_WIRELESS_POWERSAVE_ENABLE (3) (enable Wi-Fi power saving)

In this case, I want to disable Wi-Fi power saving, thus the value of 2 in the config file, set with

[connection]
wifi.powersave = 2

Detecting and setting the power-save setting on the wireless card

With or without NetworkManager, the power save setting of the AX201 and other Wi-Fi cards can be read with the iw command.

First, find the name of your wireless adapter with iw dev or to get just the name:

iw dev | grep -o 'Interface.*'

The wireless adapter likely starts with "wl". Mine is wlp0s20f3, so the following works to get the current powersave setting:

iw dev wlp0s20f3 get power_save

If you have already made the NetworkManager config changes and restarted NetworkManager, then the result should be Power save: off.

A similar command can be used to turn power_save back on:

iw dev wlp0s20f3 set power_save on

Awaiting other solutions...

Of course, I hope that a driver solution comes soon with new kernel releases. Likely the above solution will not be necessary long term. If you have encountered other ways to solve this problem, please feel free to use the comments!

Other resources

Jean-Christophe Berthon's gist with NetworkManager scripts and notes
javamarket's comments in this archlinux forum topic

How to Upgrade to Fedora 37 In Place on Windows Subsystem for Linux (WSL)

Jonathan Bowman — Sun, 18 Apr 2021 21:42:29 +0000

The time has once again arrived to upgrade Fedora. As detailed in another article, I installed Fedora on WSL 2. Now I want to upgrade to Fedora version 37. I could do a clean install of course, using the steps detailed in that article, but I want to upgrade in place.

How do we do that?

Here are the steps I use, based on official Fedora instructions.

Backup first

Just in case, yes?

First, clean up downloaded packages, etc. within Fedora:

sudo dnf clean all

Then, exit WSL and export the whole installation to a tarball (this assumes your distro name is "fedora"):

wsl --export fedora $HOME\Downloads\fedora-wsl.tar

You may want a different folder than Downloads; specify the location you desire.

Depending on what packages you installed, it may be as small as a quarter GB, or it could be far larger. You could gzip it if you want the storage size to be even smaller. Next time you want to start fresh, you can do something like this:

mkdir $HOME\wsl\freshfedora
wsl --import freshfedora $HOME\wsl\freshfedora $HOME\Downloads\fedora-wsl.tar

Begin the upgrade: freshen up

It is important to have a refreshed package index, and upgrade all packages to the latest. You can do so with

sudo dnf upgrade --refresh

Install the upgrade system

We will need DNF System Upgrade in order to make the leap, so let's install it now.

sudo dnf install dnf-plugin-system-upgrade

Download new release packages

To prepare and download for a system upgrade to Fedora 37, the following should do the trick:

sudo dnf system-upgrade download --releasever=37

This will take a bit of time.

Yes, you should feel fine about importing the new GPG for Fedora 37, so you can answer "y" to that.

Reboot?

If using WSL, this is where it gets a little strange, but only a little.

I first set a flag to indicate reboots are not necessary.

export DNF_SYSTEM_UPGRADE_NO_REBOOT=1

Not sure how necessary that is, but it doesn't hurt. We can restart WSL ourselves, if we need to, but I have not found that to be required. If you are doing this on a full Fedora system (not WSL), please do not set this flag; you need to do an actual reboot.

Now we trigger the update, strangely, with the upgrade and reboot command (pass the -E flag to sudo in order to utilize the DNF_SYSTEM_UPGRADE_NO_REBOOT variable defined earlier):

sudo -E dnf system-upgrade reboot

You should see Reboot turned off, not rebooting.

Moment of truth: launch the upgrade

Now trigger the actual upgrade with

sudo -E dnf system-upgrade upgrade

Finalization

You should now have a fresh Fedora 37 system.

In case you upgraded from Fedora 32 or earlier, the RPM database backend has changed somewhat. Refresh it with

sudo rpmdb --rebuilddb

Again, this should not be necessary unless you were on a much earlier version of Fedora.

Then refresh and upgrade all the packages.

sudo dnf upgrade --refresh

Just to be as clean as possible (you may still have processes running that are actually older versions than installed), why not restart your WSL distro. Open a Powershell window, and terminate your WSL instance. Assuming your instance is named "fedora" you can wsl -t fedora.

Then, relaunch with wsl -d fedora

Reference: all the steps in one place

For your reference, here are all the steps in one block:

sudo dnf upgrade --refresh
sudo dnf install dnf-plugin-system-upgrade
sudo dnf system-upgrade download --releasever=37
export DNF_SYSTEM_UPGRADE_NO_REBOOT=1
sudo -E dnf system-upgrade reboot
sudo -E dnf system-upgrade upgrade
sudo dnf upgrade --refresh

Enjoy your shiny new Fedora!

Keep upgrading

If you chose to upgrade to a beta or prerelease, there should be no need to reinstall. Just keep upgrading, as often as you like; the process is pretty seamless:

sudo dnf upgrade

If you daringly chose to use Fedora 37, for instance, upgrade as often as you like with the above command, and you will eventually (by the end of October 2022) be at release.

Two Methods for Testing HTTPS API Calls with Python and pytest and Also Communicating with the in-Laws

Jonathan Bowman — Tue, 16 Mar 2021 23:19:32 +0000

API endpoints and web URLs are, thankfully, more secure than ever, usually requiring encrypted HTTPS. Python works well as an HTTPS client, and pytest simplifies testing Python-based tools or libraries. Tools like VCR.py or the combination of pytest-httpserver and trustme provide an additional testing layer that is fast and convenient, and well-suited for HTTPS work. This will help with family gatherings. Let me show you.

A potentially awkward social situation, solved by HTTPS

Imagine this scenario: you recently married into a wonderful family, but your spouse's parents only speak pig Latin. Unfortunately, you are not fluent in this exotic language, and you are worried about the next family gathering. As you can guess, this is a common situation for many, so, thankfully, there is an API for that.

Being the enterprising Python developer that you are, you write a Python client to serve as a realtime translator.

It looks like this:

import json
from urllib.parse import urlencode
from urllib.request import urlopen


def translate(text, server="api.funtranslations.com"):
    data = {"text": text}
    form_encoded_data = urlencode(data).encode()
    url = f"https://{server}/translate/piglatin.json"
    with urlopen(url, form_encoded_data) as response:
        result = json.load(response)
        print(result)
    return result["contents"]["translated"]

Less than a dozen lines of code for marital bliss. Seems worthwhile.

You might save the above as pypiglatin.py in a directory of your choosing.

The script and associated tests can be found in the companion Github repo.

The tests

If you are like me, you have already tried the above enough times to receive a 429 "Too Many Requests" error. The funtranslations.com service allows you to sign up and become a paying customer in order to remove the rate limiting, but the anonymous limits certainly make a point. Your many tests are using up someone's server bandwidth and compute cycles. Maybe we should stop doing that. Even if the API you are accessing isn't rate limited.

Instead, find ways of testing locally. It will be faster, and kinder. Tests will make sure that the pig Latin translation is working smoothly, in advance of any high-pressure family gatherings.

I believe tools for HTTP client testing should:

As already mentioned, easily replicate API endpoints locally, for performance and respect of API limits.
Locally test the same protocol (HTTPS or HTTP) that is used in production. Usually API endpoints are HTTPS only, and so the URLs, even in testing, should start with "https://"
HTTP client library flexibility. Yes, requests pairs well with responses or requests-mock, and HTTPX has RESPX or pytest_httpx. Those testing helpers are an excellent match for the corresponding library, and should certainly be recommended. However, I don't always want to face the risk of rewriting all my tests if I replace the client library some day. And sometimes I am using an altogether different tool (even though both requests and HTTPX are quite awesome) such as urlopen, urllib3, httplib2, tornado, or aiohttp.

Let's explore two methods for meeting the above goals. Both assume the use of pytest:

An easy-to-configure HTTP server using pytest-httpserver, coupled with trustme for quick SSL certificate setup.
While not an actual HTTP server, pytest-vcr records and then mocks HTTP requests using the VCR.py library.

pytest-httpserver

It is hard to beat an actual HTTP server for testing reassurance, and pytest-httpserver provides a quick way to configure responses based on expected requests. The web application library Werkzeug is used to build the server.

Here is an example that tests if a request to the "/hello" endpoint indeed returns "Hello, World!" as expected:

from urllib.request import urlopen

import pytest


@pytest.fixture(scope="session")
def httpserver_listen_address():
    return ("localhost", 8888)


def test_hello(httpserver):
    body = "Hello, World!"
    endpoint = "/hello"
    httpserver.expect_request(endpoint).respond_with_data(body)
    with urlopen(httpserver.url_for(endpoint)) as response:
        result = response.read().decode()
    assert body == result

To use the above test, save it in your current directory as test_http.py or similar, then install pytest and pytest-httpserver in your preferred manner (pip install pytest pytest-httpserver works well in a Python virtual environment). Then run pytest from the command line in that directory.

While not strictly necessary, I like defining the httpserver_listen_address fixture from pytest-httpserver to explicitly define the server and port. This makes it easy to define URLs later, as the port is known. Otherwise, pytest-httpserver will assign a random port.

The heart of the test server setup is the line

httpserver.expect_request(endpoint).respond_with_data(body)

We tell the server what sort of request to expect, and the data with which to respond to the expected request. This is quite configurable, on both ends. See the docs for more advanced examples. For instance, request and/or response headers can be defined, the method (GET, POST, PUT, DELETE) specified, and various formats of data can be expected or returned, such as JSON.

Adding SSL with trustme

What the sample test above is missing is HTTPS. Thankfully, pytest-httpserver allows a Python ssl.SSLContext to be specified by defining the httpserver_ssl_context fixture.

However, building a functioning certificate authority and the rest of the cert chain is not exactly a trivial matter. Enter trustme, a Python-based tool for enabling SSL for testing purposes. You can install it in your virtual environment with pip install trustme.

Here is a test mechanism for our previously-built translate function:

import ssl
from urllib.parse import urlencode

import pytest
import trustme

import pypiglatin

SERVER = "localhost"
PORT = 8888
ENDPOINT = "/translate/piglatin.json"


@pytest.fixture(scope="session")
def httpserver_listen_address():
    return (SERVER, PORT)


@pytest.fixture(scope="session")
def httpserver_ssl_context():
    ca = trustme.CA()
    client_context = ssl.SSLContext()
    server_context = ssl.SSLContext()
    server_cert = ca.issue_cert("test-host.example.org")
    ca.configure_trust(client_context)
    server_cert.configure_cert(server_context)

    def default_context():
        return client_context

    ssl._create_default_https_context = default_context

    return server_context


def test_post(httpserver):
    text = "Thank you for your hospitality"
    translated = "ank-Thay ou-yay or-fay our-yay ospitality-hay "
    response = {
        "success": {"total": 1},
        "contents": {
            "translated": translated,
            "text": text,
            "translation": "pig-latin",
        },
    }

    httpserver.expect_request(
        ENDPOINT, method="POST", data=urlencode({"text": text})
    ).respond_with_json(response)
    assert pypiglatin.translate(text, f"{SERVER}:{PORT}") == translated

A few notables:

We take advantage of the httpserver_ssl_context fixture to not only build a server SSL context and return it, but also a build a client context that trusts the server cert. We then make ssl._create_default_https_context() return that client context. Thus, any HTTPS client that seeks to utilize the default context from the ssl module will respect our new cert authority.
If your client does not use the ssl module to provide an SSL context, you will want to find another way to pass the client SSL context to your HTTP client. If need be, see the trustme documentation on how to write the certs to files so they can be utilized later.
The expected request is a little more complex than our test_hello example. It expects application/x-www-form-urlencoded POST data (encoded with urllib.parse.urlencode()).
The response is a Python dict populated with a known API response, then converted to JSON by the respond_with_json() method.
In my tests, the server SSL context seemed to work as the client SSL context as well. It just didn't seem like the right way to do it, so I separated them. Feel free to post advice in the comments.
This is only one test. We can add tests to make sure incorrect requests elicit an error, and other messages are translated correctly. Neither the production API server nor the in-laws will know how fast and how often we fail. Beauty.

The end result is a live local HTTPS service intended to behave very similarly to the remote production service when called with specific data. The primary difference is that the URL is "https://localhost:8888/translate/piglatin.json" when testing, but "https://api.funtranslations.com/translate/piglatin.json" in production.

For a nice introduction to pytest-httpserver, see the tutorial. For a deeper dive, see the how-to.

VCR.py

I love the convenience and realism that pytest-httpserver provides. The only downside is the mild tedium in recording and setting up responses and expected requests manually.

An alternate approach involves using VCR.py, a tool that records HTTP interactions in YAML files, then intercepts future HTTP requests and plays back the recorded responses. In this tutorial, we will use pytest-vcr to interface with VCR.py, although pytest-recording is another good option for doing the same.

While this doesn't provide an actual HTTPS server like pytest-httpserver does, the URLs used in the client code do not need to change. They can still start with "https://"; in fact, because the HTTPS calls are intercepted, URLs can use the same domains as production, without fear that calls will be emitted to the production servers (after the initial recording is saved).

Unfortunately, VCR.py only supports a certain number of HTTP client libraries. Thankfully, some very popular implementations are included: the Python built-in urllib and http.client, as well as requests, urllib3, and aiohttp. And, while it may not be listed in the docs at the time of this writing, the changelog notes that HTTPX is also supported. Sadly, pycurl is not supported.

To use VCR.py with pytest, start by installing pytest-vcr in your virtual environment:

pip install pytest-vcr

Implementing a test for the translate() function we built is quite simple using pytest-vcr:

import pytest

import pypiglatin


@pytest.mark.vcr()
def test_translate():
    text = "Thank you for your hospitality"
    translated = "ank-Thay ou-yay or-fay our-yay ospitality-hay "
    assert pypiglatin.translate(text) == translated

You might save this as test_vcr.py in the current directory. Now we can run pytest.

Afterward, there should be a cassettes directory with a test_translate.yaml file in it. Feel free to take a look: you will get an idea for the many facets of an HTTP request and response. The file is also easily editable, which could be useful, for instance, to munge authentication/authorization information (just do that before saving the file to version control, of course).

If you run pytest again, the test should run much faster, using the existing cassette.

This YAML cassette can be saved to version control, alongside the tests, so that later runs on any machine will not need to hit the live server at all.

And there it is. VCR.py is actually this easy.

Improved family dynamics

Hopefully, you are now gaining fluency in pig Latin. Meanwhile, you can be confident in your fallback plan: a thoroughly-tested translator, available at a moment's notice. Feel the reliability.

Please feel free to leave comments with your experiences, questions, or marital advice!

Alternatives

Thes two methods are solid and flexible.

Of course, as noted earlier, there are other options out there, especially if you use requests or HTTPX:

For requests:

For HTTPX:

Also of interest

If this article was helpful, you may also be interested in some of my other articles:

Hardening and Simplifying Python's urlopen

Jonathan Bowman — Wed, 10 Mar 2021 23:05:00 +0000

I recently wrote an article detailing the use of Python's urlopen() for performing HTTP calls. While researching and writing, I learned of the OpenerDirector class. This class offers the opportunity to streamline urlopen(), make it more secure, and provide custom error handling.

Security problems with the default `urlopen()`

You might try the following on a Linux system:

from urllib.request import urlopen

url = "file:///etc/passwd"

with urlopen(url) as response:
  print(response.read().decode())

A little disturbing, yes? Bandit agrees. Perhaps you want to consider scanning with that security tool or its related flake8 plugin.

The code above certainly communicates the lesson "sanitize your user inputs". Of course, if you control that url string, or can ensure that it starts with the correct https:// scheme, then things are looking better. If this url comes from user input, though, it would be good to check for protocol at least, and, better yet, ensure that the domain is as expected.

A suggested solution

The following code results in a urlopen() command that only opens https:// URLs by default:

import urllib.request


class SafeOpener(urllib.request.OpenerDirector):
    def __init__(self, handlers: typing.Iterable = None):
        super().__init__()
        handlers = handlers or (
            urllib.request.UnknownHandler,
            urllib.request.HTTPDefaultErrorHandler,
            urllib.request.HTTPRedirectHandler,
            urllib.request.HTTPSHandler,
            urllib.request.HTTPErrorProcessor,
        )

        for handler_class in handlers:
            self.add_handler(handler_class())


opener = SafeOpener()
urllib.request.install_opener(opener)

After running the above, using urllib.request.urlopen() should fail with a URLError if attempting to open http:, ftp:, file:, data:, or any other URL that doesn't have https: at the beginning. It will still follow redirects automatically, just as urlopen() does, and raise an exception for any HTTP status code that isn't in the 200s or 300s.

By the way, if you prefer not to override the opener in the urlopen() function, you could remove the install_opener(opener) line. Then call opener.open() instead of using urlopen().

The above code assumes that all HTTP calls will be encrypted with TLS (aka "SSL", using https:). That also means that testing will need to use https: URLs as well. Consider using the vcr library to mock-reproduce HTTP calls, or the trustme library to actually set up certs for testing. If you need to use unencrypted http: URLs, though, you can simply add urllib.request.HTTPHandler to handlers.

The handlers, defined

This is the chain of default handlers normally used by urlopen():

ProxyHandler: searches system settings for proxies. If you are 100% sure that your tool or library will never be used with a proxy, then this is not necessary.
UnknownHandler: raises a URLError if the protocol requested in the URL is not supported by a handler in this chain. Very helpful and recommended.
HTTPHandler: handles unencrypted http:// connections. Only add this if you are sure you need URL support other than https:
HTTPDefaultErrorHandler: a prefilter of sorts that turns all responses into exceptions, for handling downstream. This is necessary unless you plan on handling statuses, exceptions, and redirects yourself.
HTTPRedirectHandler: handles redirects (status codes 301, 302, 303, or 307) and is necessary if automatic following of redirects is desired.
FTPHandler: handles ftp: URLs. Not necessary for HTTP calls.
FileHandler: handles file: URLs, and poses security risks. Rarely should this be necessary, if ever.
HTTPErrorProcessor: The final response handler, raising any non-20x (OK) responses
DataHandler: handles data: URLs. Hard to imagine why this would be necessary in normal use, and could pose potential security risks with user input.

As may be apparent, several of the above are rarely necessary, if ever, for HTTP API work. Instead, I recommend this list as a happy medium between security and usability:

ProxyHandler
UnknownHandler
HTTPHandler
HTTPDefaultErrorHandler
HTTPRedirectHandler
HTTPSHandler
HTTPErrorProcessor

Of the above, ProxyHandler could possibly be removed if you know you don't need it, and even HTTPHandler could be removed if you know that only https: URLs will be called. Actually, this is a pretty good combination: the point of https: is to ensure that nothing is intercepting the connection, and that there are no proxies. So a most-secure list is the same as what is in the example code above:

UnknownHandler
HTTPDefaultErrorHandler
HTTPRedirectHandler
HTTPSHandler
HTTPErrorProcessor

Five handlers, not the original nine.

Flexibility

The use cases for custom OpenerDirector instances go beyond just security and simplicity. By subclassing BaseHandler then adding custom status handling methods with names like http_error_401, you create your own handlers that then can be appended to the handler list. These can be used for authorization, retry cadence, and other goals.

Curious how these suggestions work for you! Feel free to suggest improvements and share experiences in the comments.

HTTP Calls in Python Without Requests or Other External Dependencies

Jonathan Bowman — Sun, 07 Mar 2021 22:44:17 +0000

In addition to great Python HTTP client tools such as Requests and HTTPX, the standard library itself supplies the necessary ingredients to make a working HTTP client for API calls. This tutorial shares how to construct and customize such a tool for your own scripts.

Consider installing a library

Before proceeding, I should note that in many cases, the approach in this article is not best practice. Instead, I highly recommend using a third-party Python library for features, security, and reliability.

Some suggested libraries:

urllib3 is the dependency for many other tools, including requests. By itself, urllib3 is quite usable. It may be all you need.
requests is ubiquitous and well documented.
HTTPX has an interface almost identical to requests, but with the added benefit of asyncio support. You may be interested in a series of articles I wrote on using HTTPX both synchronously and asynchronously.
pycurl is less popular as a Python library, but interfaces with the well-known libcurl.
aiohttp has an asyncio-based HTTP client that is well-documented and well-liked.

If, however, you find yourself needing a solution that does not require external dependencies other than what is already available in the Python standard library, then you may wish to read on.

Summary code

import json
import typing
import urllib.error
import urllib.parse
import urllib.request
from email.message import Message


class Response(typing.NamedTuple):
    body: str
    headers: Message
    status: int
    error_count: int = 0

    def json(self) -> typing.Any:
        """
        Decode body's JSON.

        Returns:
            Pythonic representation of the JSON object
        """
        try:
            output = json.loads(self.body)
        except json.JSONDecodeError:
            output = ""
        return output


def request(
    url: str,
    data: dict = None,
    params: dict = None,
    headers: dict = None,
    method: str = "GET",
    data_as_json: bool = True,
    error_count: int = 0,
) -> Response:
    if not url.casefold().startswith("http"):
        raise urllib.error.URLError("Incorrect and possibly insecure protocol in url")
    method = method.upper()
    request_data = None
    headers = headers or {}
    data = data or {}
    params = params or {}
    headers = {"Accept": "application/json", **headers}

    if method == "GET":
        params = {**params, **data}
        data = None

    if params:
        url += "?" + urllib.parse.urlencode(params, doseq=True, safe="/")

    if data:
        if data_as_json:
            request_data = json.dumps(data).encode()
            headers["Content-Type"] = "application/json; charset=UTF-8"
        else:
            request_data = urllib.parse.urlencode(data).encode()

    httprequest = urllib.request.Request(
        url, data=request_data, headers=headers, method=method
    )

    try:
        with urllib.request.urlopen(httprequest) as httpresponse:
            response = Response(
                headers=httpresponse.headers,
                status=httpresponse.status,
                body=httpresponse.read().decode(
                    httpresponse.headers.get_content_charset("utf-8")
                ),
            )
    except urllib.error.HTTPError as e:
        response = Response(
            body=str(e.reason),
            headers=e.headers,
            status=e.code,
            error_count=error_count + 1,
        )

    return response

You are certainly welcome to copy and use the above function, or browse or clone the Github repo.

If, however, you are reading this article for a do-it-yourself approach, I encourage you to build your own function that suits your needs. It may grow simpler or more flexible than the above.

Let's discuss the building blocks.

An introduction to `urllib.request.urlopen()`

The recommended high-level function for HTTP requests is urlopen(), available in the standard urllib.request module.

Unlike the lower-level http.client module, urlopen() provides error handling, follows redirects, and provides convenience around headers and data.

An example:

from urllib.request import urlopen, Request

url = "https://jsonplaceholder.typicode.com/posts/1"

if not url.startswith("http"):
    raise RuntimeError("Incorrect and possibly insecure protocol in url")

httprequest = Request(url, headers={"Accept": "application/json"})

with urlopen(httprequest) as response:
    print(response.status)
    print(response.read().decode())

I highly appreciate and recommend JSONPlaceHolder's free fake API, used above. Useful precisely for what we are doing here: testing HTTP clients intended for API work.

Please note the security measures in the above code. Before passing a URL to urlopen(), make sure that it is a web url and not a local file ("file:///"). If you want a wake-up call, try urlopen("file:///etc/passwd").read() on a Linux system (not in production code, though!) and see what happens. Of course, this protocol check is only necessary if the URL comes from user input. If you control the URL string and can assure that it does not start with "file:" then that is a good thing. You may also be interested in an another approach to hardening urlopen() by redefining the list of protocol handlers.

Protocol checking aside, the urlopen() call is fairly simple, as you can see in the above example. I recommend using it alongside with in a context manager for tidiness, so that closing the response is handled automatically.

We passed a Request object to the urlopen() function. While we could simply pass a URL string, the Request object offer much more flexibility: we can specify HTTP method (GET, POST, PUT, HEAD, DELETE), request headers, and request data.

The response returned by urlopen has 4 useful attributes:

It has a file-like interface that can be read(), returning bytes
url
status returns the HTTP status code
headers returns an EmailMessage object. This functions somewhat like a dict but with case-insensitive keys. It also has some helpful methods such as get_content_type() and get_content_charset(). The get_all() method is another useful one, for when there may be multiple key/value pairs for the same header name. See the helpful Wikipedia article for a list of possible reponse headers.

HTTP errors

Out of the box, urlopen handles redirects (status codes 301, 302, 303, or 307). Other than these codes, though, if the status code is not between 200 and 299 (HTTP "OK" codes according to RFC 2616) then an HTTPError exception is raised.

The HTTPError can be captured and analyzed with the appropriate try... except... block, such as this:

from urllib.error import HTTPError
from urllib.request import urlopen

try:
    urlopen("https://github.com/404")
except HTTPError as e:
    print(e.status)
    print(e.reason)
    print(e.headers.get_content_type())

The error (assigned to the "e" variable in the above) has the following useful properties:

status to get the error code (such as 404)
headers as an EmailMessage object. Again, this can be treated like a case-insensitive dict.
reason with the text of the error

In the function at the top of this article, I catch and silence all errors, to make the response uniform, and pass error-handling responsibility downstream. However, this may not be desirable. Perhaps, instead of continuing no matter the error, you want to fail on anything other than a 401 or 429 error:

except urllib.error.HTTPError as e:
    if e.code in (401, 429):
        response = Response(
            body=str(e.reason),
            headers=e.headers,
            status=e.code,
            error_count=error_count + 1,
        )
    else:
        raise e

Of course, logic could be added to deal with errors as appropriate, depending on the status code.

Note the entirely optional auto-incrementing error_count attribute in my code. Sometimes, I wish to call the http request function recursively. This allows the number of calls to be tracked, and dealt with downstream, hopefully preventing infinite recursion. For instance, I may want to catch 401 errors, parse the "Www-Authenticate" header for a token, then retry the request with the token. But if this fails repeatedly (say, 5 tries), it should stop. I could test for error_count >= 5 and raise and exception if so, meanwhile making sure to pass the current error_count back to the request function as a parameter, so it continues to be incremented appropriately.

An alternative way to customize error handling is to construct your own subclasses of BaseHandler, then build an OpenerDirector chain of handlers as appropriate. For instance, you could subclass BaseHandler and add a method http_error_401 to handle authorization as desired, then pass an instance of that custom class to build_opener(). Obviously, this requires a deeper dive into the opener innards.

A versatile `Response` object

I find it helpful to create a Python class that can contain the bits of the HTTP response that I care about. This could be a dict, but I like to add a method or two, such as a JSON decoder.

If using Python 3.7 or later, consider using a dataclass. An example:

@dataclass(frozen=True)
class Response():
    body: str
    headers: Message
    status: int
    error_count: int = 0

    def json(self) -> typing.Any:
        """
        Decode body's JSON.

        Returns:
            Pythonic representation of the JSON object
        """
        try:
            output = json.loads(self.body)
        except json.JSONDecodeError:
            output = ""
        return output

Enabling frozen is strictly optional, and reflects my preference for this object being immutable (attributes can't be changed after initialization).

Another option, as demonstrated in the code at the beginning of the article, is a typed NamedTuple. I chose this for its immutability, ease of setup, and backwards compatibility.

Of course, a custom class will work, or attrs, or whatever container works for you.

Requests with data

Depending on the API with which you are interfacing, you may encounter various scenarios for accepting data. In each scenario, we can start with a Python dict and convert it into the required format.

The query string

Sometimes, data is passed in through the query string. To encode a Python dict as a query string that can be appended to the URL (after the "?"), use urllib.parse.urlencode:

from urllib.parse import urlencode
from urllib.request import urlopen

url = "https://jsonplaceholder.typicode.com/posts"
params = {"userId": 1, "_limit": 3}
url += "?" + urlencode(params, doseq=True, safe="/")

with urlopen(url) as response:
    print(response.read().decode())

While not relevant to the above request, I did pass two parameters to urlencode that I have found helpful:

doseq will allow lists to be encoded as multiple parameters. For instance, if we passed in {"usernames": ["John Doe", "Jane Doe"]}, then the end result would be "usernames=John+Doe&usernames=Jane+Doe".
safe defines the characters that will not be url-encoded. In some APIs I encounter, such as the Docker API, it is better to leave slashes unencoded, so I added that to the safe string. Adapt as you see fit.

Sending data in the request body

Similarly, data can be encoded with urllib.parse.urlencode and then passed into the Request object via the data parameter:

from urllib.parse import urlencode
from urllib.request import Request, urlopen

url = "https://api.funtranslations.com/translate/yoda.json"
data = {"text": "HTTP POST calls are remarkably easy"}

postdata = urlencode(data).encode()

httprequest = Request(url, data=postdata, method="POST")

with urlopen(httprequest) as response:
    print(response.read().decode())

Sending JSON in the request body

Many APIs accept and even require the request parameters to be sent as JSON. In these cases, it is important to first encode the Python dict (or other object) as JSON, then set the "Content-Type" request header appropriately:

import json
from urllib.parse import urlencode
from urllib.request import Request, urlopen

url = "https://jsonplaceholder.typicode.com/posts"
data = {
    "userid": "1001",
    "title": "POSTing JSON for Fun and Profit",
    "body": "JSON in the request body! Don't forget the content type.",
}

postdata = json.dumps(data).encode()

headers = {"Content-Type": "application/json; charset=UTF-8"}

httprequest = Request(url, data=postdata, method="POST", headers=headers)

with urlopen(httprequest) as response:
    print(response.read().decode())

In the above, we used Python's built-in JSON module to dump the data dict into a string, then encode it to bytes so it could then handled as POST data.

We set the content type header to application/json. In addition, we specified the character encoding as UTF-8. Given that UTF-8 is the required JSON character encoding, this is redundant and probably unnecessary, but it never hurts to be explicit.

Parsing JSON in the response body

Because most APIs I use return JSON, and some return other formats such as XML unless JSON is specified, I typically set the Accepts header in the request to application/json. If you are pulling other types of data, such as text/csv, you would want to tweak that header. Set it to */* if you don't care.

In the Response object we created earlier, there is an example of a JSON decoder, the result of which will likely be a Python dict or list, but could conceivably be a string or boolean, depending on what the server returns. Here is another example of similar functionality, but loading the JSON directly from the file-like response:

import json
from urllib.parse import urlencode
from urllib.request import Request, urlopen

url = "https://jsonplaceholder.typicode.com/posts?_limit=3"

with urlopen(url) as response:
    try:
        jsonbody = json.load(response)
    except json.JSONDecodeError:
        jsonbody = ""

print(jsonbody)

In the above example I decided to fail silently, offering an empty string when JSON decoding fails. If this is not desired, just use json.load() (or json.loads() from a string) and let any exceptions float up as they occur.

Other tricks or suggestions?

I am very curious if you use urlopen and how. Are there optimizations to the above that I am missing? Does this raise any questions or confusion? Feel free to post in the comments.

A "POSIX Playground" Container for Shell Script Testing

Jonathan Bowman — Tue, 23 Feb 2021 11:36:27 +0000

After exploring portable scripting in the previous article, I built a container available on Docker Hub, useful for testing and experimentation.

The container is an Alpine Linux container with the following tools:

posh shell (this is the default)
dash shell
rlwrap to offer readline support for either of the above shells (rlwrap posh is the default command if none specified)
checkbashisms for rooting out Bash-specific idiosyncrasies in scripts
shellcheck for shell script linting

See the "Writing Bash Scripts that are not only Bash" for explanation and usage of the above.

Using the container

(Podman also works fine in any of the examples below.)

First, pull the image:

docker pull docker.io/bowmanjd/posix-playground

Note that the default user is shelly and the home directory is /home/shelly

To launch posh: Policy-compliant Ordinary SHell with readline support:

docker run -it bowmanjd/posix-playground

To instead launch dash shell with readline support:

docker run -it bowmanjd/posix-playground rlwrap dash

(Without rlwrap, line editing in either of these shells is impossible, making experimentation rather painful. Using rlwrap is recommended.)

Note that the container runs as an unprivileged user (as common sense dictates!) named shelly and the home directory is /home/shelly.

To make every file in the host's current working directory available from within the home directory inside the container, you might bind mount the current directory like this:

docker run -it -v "$(pwd):/home/shelly" bowmanjd/posix-playground

If using rootless podman, you will find it convenient to add the :z SELinux label to the bind mount to indicate a shared volume, allowing read-write acces, then use the keep-id user namespace in order to map the user id of shelly to your current user, as follows:

podman run -it --userns=keep-id -v "$(pwd):/home/shelly:z" bowmanjd/posix-playground

To check a helloworld.sh shell script in the current directory for "bashisms" (make sure the first line, the "shebang", reads #!/bin/sh):

cat helloworld.sh | docker run -i bowmanjd/posix-playground checkbashisms

To check a helloworld.sh shell script in the current directory for a variety of gotchas:

cat helloworld.sh | docker run -i bowmanjd/posix-playground shellcheck -

Be sure to include the - on the end: shellcheck - to let shellcheck know to expect stdin, as you are "piping" the script file contents into shellcheck as standard input.

To simply run a script using posh:

docker run -i -v "$(pwd):/home/shelly" bowmanjd/posix-playground posh myscript.sh

Posh: an obscure shell with great usefulness

Of the tools included in the image, posh is the winner for me. It is not available on every distro, so having a container available is quite convenient.

Posh is also quite strict, and this is the point. It does not have any syntactic sugar to make it more Bash-like. I feel quite reassured when a script runs correctly in posh.

If you have other tools to recommend, or want to reflect on your experiences with this container image, feel free to contact me!

Writing Bash Scripts that are not only Bash: Checking for Bashisms and testing with Dash

Jonathan Bowman — Sun, 21 Feb 2021 20:52:25 +0000

Shells like Bash or Zsh are advanced and user-friendly, and include features beyond what a simpler POSIX-compliant shell might offer. You will do well to utilize the full features of your shell when writing scripts.

There are situations, however, when portability should be a valued feature, allowing the script to run on a variety of shells.

Bash scripts are most portable when "bashisms" are avoided. Let's explore writing POSIX-compliant shell scripts that work on Ash/Dash and other shells.

A summary checklist

Here is a checklist I use to keep tabs on my own script writing. Does my script:

Have #!/bin/sh as the first ("shebang") line of the script, not #!/usr/bin/bash or other shell
Avoid double-bracket tests [[ ]] and instead use single-brackets [ ]
Use printf instead of echo -e when newlines '\n' need to be printed
Use no other read flag other than -r, as in read -r
Avoid Bash's convenience redirects: use >myfile 2>&1 to redirect stdout and stderr to a file rather than &>myfile
Test accurately with dash or posh: Policy-compliant Ordinary SHell
Only use standard flags and options with common utilities such as sed, grep, cut, test, and others
Avoid issues discovered by shellcheck

An example

#!/bin/sh

read -p "Who would you like to greet? "

if [[ -z $REPLY ]] ; then
  recipient="$REPLY"
else
  recipient="World"
fi

echo -e "Hello\n$recipient\n"

You might save the following in the current working directory of your choice, as example-noncompliant.sh

In human language, the above script prompts for a greeting recipient, then sets the recipient to "World" if none was given, then greets the recipient on multiple lines.

The above works on Bash, but has issues on other shells. You may wish to run it once in Bash, just to feel good. Even in Zsh, though, it may raise some complaints.

Dash, a POSIX compliant shell

Dash is a derivative of the Ash shell. It is meant to be POSIX-compliant.

Debian and Ubuntu come with dash installed. In fact, scripts invoked with /bin/sh will run with dash by default. On Alpine, Tiny Core Linux, OpenWRT, and other distros that use BusyBox by default, the standard shell is also dash (although labeled as ash). On Fedora, dash can be installed with sudo dnf install dash. Other distros may also include dash in their repositories.

Using Docker or Podman, running dash is easy, as in this example:

docker run -it debian dash

You may also try my POSIX playground container, with a variety of tools, including dash. By default, it uses posh: Policy-compliant Ordinary SHell, which is slightly stricter than dash. It can be launched with:

docker run -it docker.io/bowmanjd/posix-playground

See the article for a deeper explanation.

In all of the above, podman can replace docker without a problem.

Testing the example

Can you try running the example-noncompliant.sh script above, but with dash, not Bash?

dash example-noncompliant.sh

Or, using Docker or Podman:

docker run -it -v "$(pwd):/work" debian dash /work/example-noncompliant.sh

The output is likely something resembling:

dash: 3: read: arg count
dash: 5: [[: not found
-e Hello

A few learning points can be derived from that output.

Avoid double-bracket tests `[[ ]]`

The [[ construct is a safe one if using Bash or another shell that supports bashisms. It has some convenient features, like regex matching using =~, and has less risks with string matching.

That said, you will generally not go wrong with the single bracket approach: [ ] (an alias for test). Always be sure to quote variables, but that is good advice anyway. If you need regular express matching, use grep.

Bottom line: not all shells support [[; use [ instead.

Use vanilla `read`

When using the read command to get input, here are a few suggestions:

Always specify the variable, rather than relying on Bash's default $REPLY
Use read -r and no other flags. Using -r prohibits the user from using backslash to escape characters, which can cause issues later. And no other flag is supported by POSIX read.
Instead of specifying a prompt with -p, just use a printf call prior to the read command. Again, POSIX read does not support such a flag, plus the -p option means something different to Zsh's read.

Given these rules, our script should not use read -p "Who would you like to greet? " but rather:

printf "Who would you like to greet? "
read -r recipient

Use `printf` when newlines are at issue

The echo command works great when we know we want to output a simple string, followed by a newline.

However, if we have newlines in a string we want to print, or if printing without a trailing newline is desired, then printf, not echo -e will be our friend.

So, instead of echo -e "Hello\n$recipient\n" in our code above, this would be better:

printf "Hello\n%s\n\n" "$recipient"

Note the variable substitution going on with %s in the first string (the format string). Do not put shell variables like $recipient in the format string. This is the way.

Refactoring the example

Given the above concerns, let's completely rewrite our greeting script:

#!/bin/sh

printf "Who would you like to greet? "
read -r recipient

if [ -z "$recipient" ] ; then
  recipient="World"
fi

printf "Hello\n%s\n\n" "$recipient"

You might save the above with the filename example-posix.sh or similar.

When you run it, does it behave the same as the noncompliant script? Hopefully not; try it out.

Satisfying.

Finding the bashisms with `checkbashisms`

There is a tool embedded in the Debian devscripts project, called checkbashisms. It is a simple but powerful Perl script that ferrets out any bashisms in a shell script that begins with the #!/bin/sh shebang line.

On Debian and Ubuntu, it can be installed with sudo apt install devscripts and on Fedora with sudo dnf install devscripts-checkbashisms while Alpine is sudo apk add checkbashisms. Other distros may have something similar. You might also try installing Perl, then downloading and running the checkbashisms Perl script itself.

What happens when you run it on example-noncompliant.sh or example-posix.sh? So telling...

Pursuing best practices with `shellcheck`

My new favorite shell scripting helper is Shellcheck. You can paste your shell script online and check it there, or install shellcheck in the usual way (Debian, Ubuntu, Fedora, Alpine, Archlinux, and others have it readily available in the standard package repositories.)

It does raise the POSIX-compliance flag on any lines that need it, but many other issues are checked as well. Your code might run just fine, but have gotchas that need some attention. Shellcheck will help you there. I integrate it into my editor, so that I can lint while I type.

Measure against the POSIX specification

Thankfully, the POSIX.1-2017 standard is openly documented. Consider this: when discovering and testing options for a given tool like read, grep, or sed, instead of going to the GNU pages, the distro man pages, or the Bash or Zsh docs, why not go to the POSIX spec itself? The list of utilities and their options is plainly explained.

In instances where you really need an enhancement provided by the extended tools, you can make that choice. With the POSIX spec in hand, it becomes an informed decision.

Often, I find that I don't need sed -E or grep -E as badly as I thought. A few extra escape characters, and I am there.

Consider other languages and configuration tools

Sometimes, when the extended syntax provided by GNU utilities is warranted, it may be a sign that the right tool for the job isn't the shell and its compatriots at all.

If your system has Python, Ruby, NodeJS, or other favorite language, might that be a more robust, flexible, and consistent option? Even in circumstances (embedded systems) in which those runtimes would be too bulky, perhaps remote scripting from another machine is in order. For instance, one could use Python to SSH to the remote machine, gain the information necessary, perform some logic, then send the appropriate commands back, without Python being necessary on the target machine.

This is the reason such tools as Ansible, Saltstack, Chef, and Puppet exist. These, too, can be quite bloated if the needs are simple. But they are unbeatable for flexibility and repeatability.

Other resources

In my research for this article, I encountered some resources you may find at least as interesting as this one:

The informative and well-written A Brief POSIX Advocacy: Shell Script Portability by Arnaud Tomeï
The Autoconf portable shell guide
Sven Mascheck's remarks on various Unix tools
How to make bash scripts work in dash by Greg Wooledge

Please feel free to share your tips, questions, corrections in the comments!

Install Docker on Windows (WSL) without Docker Desktop

Jonathan Bowman — Sun, 14 Feb 2021 21:18:59 +0000

Updated April 10, 2022, with current Alpine instructions, Debian/Ubuntu package signing tweaks (no more apt-key), and better guidance for handling iptables in Debian. A little more suggestion about TCP access, as well. And further emphasis on the optional nature of the /mnt/wsl/shared-docker socket directory.

Windows Subsystem for Linux 2 sports an actual Linux kernel, supporting real Linux containers and Docker. Docker works on WSL 2, and without requiring the robust but heavy Docker Desktop if that is undesirable. However, due to both WSL and Docker complexities, a little tender loving care is required to get Docker up and running. This article attempts to explore such a process and options along the way.

Contrary to what the length of this article might suggest, getting Docker working on WSL is fairly simple. In a nutshell:

Instead of using an init system such as systemd to launch the Docker daemon, launch it by calling dockerd manually.
If sharing the Docker daemon between WSL instances is desired, configure it to use a socket stored in the shared /mnt/wsl directory.
If sharing and privileged access without sudo are desired, configure the docker group to have the same group ID across all WSL instances
For simplicity, rather than launch a Windows-based Docker client, launch docker inside WSL.

Plenty more nuance and decisions below, of course. See details regarding the companion Github repo by scrolling to the bottom.

Are you sure you don't want Docker Desktop?

Before proceeding, let's note that Docker Desktop is amazing. If you want Docker to work on Windows and WSL 2, installing Docker Desktop is most likely the way to go. With Docker Desktop's WSL 2 backend, Docker integrates with Windows in a fairly elegant way, and the docker client can be launched from either Powershell or Linux. Reading about what goes on under the hood is an entertaining and informative endeavor, as well. Very clever.

If you came here looking how to get Docker running easily, or if you want Windows containers (still a rarity) out of the box, then Docker Desktop is your friend, and you can go install it now.

But if you, like me, feel that all the added complexity of Docker Desktop is unnecessary, you don't need Windows containers, or you are simply tired of that whale in the system tray taking... so... long... then perhaps you want to run the docker daemon (dockerd) in the WSL distro of your choice and be happy. You are at the right place.

Note that Docker Desktop is only free individuals or for small companies. If you are using it for work, and your company exceeds a certain size or revenue, then consider paying for a subscription. Of course, if you use Docker without Docker Desktop, as detailed in this article, then this does not apply. See more details about the Docker subscription model here.

Have you tried Podman?

Before we mosey along, though: are you aware of Podman? Podman is daemonless (no background service needed), modern (cgroups v2 out of the box), supports rootless, and serves as a drop-in replacement for Docker. Without needing to worry about sockets and ports, a lot of headaches go away.

I have written about getting Podman to work on WSL 2. Feel free to try it out. You may never look back. And, yes, VSCode can work with podman.

And yet... Sometimes, one just needs Docker to work. Maybe some tooling you use can't handle Podman, or you just want to put WSL through its paces. If so, read on.

Make sure that WSL is version 2

Please note that these steps require WSL 2 (not version 1). WSL 2 uses an actual Linux kernel that allows Linux containers. WSL 1 was genius with running Linux on the Windows kernel, but of course lacked some of the features, such as containers. Microsoft offers a more detailed comparison in the docs.

You will most certainly need WSL 2 to run the Docker service.

Does the command wsl --set-default-version 2 work? This will set the default version to WSL 2, or fail if you are still on the first version.

Microsoft's has step-by-step instructions on how to upgrade to WSL 2.

To run WSL 2, Windows version 1903 or higher is needed, with Build 18362 or higher. To tell what version you are running, run winver in Powershell or CMD, or just type Win key and R (⊞-r) to open the Run dialog and then enter winver. Hopefully you will see something like "Version 21H2. OS Build 19044.1586"

Install a Linux distro

If you do not yet have a running WSL instance with a distro of your choice, the next step is to pick one from the Microsoft Store. If you dislike the Windows Store, there are other options.

Custom installations are also a great option with WSL 2. For instance, install and configure Fedora, or any other distro for which you can obtain a rootfs in tar format and then wsl --import rootfs.tar.

This guide includes instructions for launching dockerd in Debian, Ubuntu, Alpine, and Fedora. If you think there is another obvious WSL distro that should be considered, feel free to let me know in the comments.

Configure a non-root user

Once you have installed the distro of your choice, launch it and set up a non-root user if you have not already. Debian and Ubuntu will configure this automatically at first launch, as should Alpine if you installed it from the Store. If the whoami command returnes "root", then you will want to add a non-root user. For Alpine or Fedora, use adduser myusername to create a new user. On Alpine, this should prompt for the new password. On Fedora, you will additionally need to passwd myusername and enter the password you want to use. (If your Fedora does not have passwd, then you will need to first dnf install passwd cracklib-dicts).

On later versions of Alpine from the Microsoft Store, while a non-root user is created as part of setup, this user is initially password-less. You can double check on any distro with:

cat /etc/shadow | grep myusername | cut -d: -f2

(If you are not root, you may need to su first).

If the result is "!" then that user has no password set. If the result is a random hash string, then you are good. If you need to set a password, you can use passwd myusername (of course, in all of the above, use your username in place of "myusername."

Configure admin (sudo) access for the non-root user

If you used Debian or Ubuntu from the Windows store and set up the default user on first launch, then sudo should already be configured on behalf of the default user. You can skip this step, and proceed to updating packages and testing network connectivity, below.

When signed in as the user you set up (try su myusername if you are still root), can you sudo -v without an error?

If not, first make sure that sudo is installed. On Alpine, that's apk add sudo and on Fedora, dnf install sudo. If this fails due to network connectivity, see below. You can follow the directions there in order to correct DNS, but of course eliminate any occurrence of sudo in those commands, as you do not have it yet, and you should still be root anyway.

Is your user a "sudoer"? Try the following to see if they are part of the sudo or wheel group:

grep -E 'sudo|wheel' /etc/group

On distros that have a sudo group, such as Ubuntu and Debian, you should see something like sudo:x:27:myusername and on distros that have a wheel group, such as Fedora and Alpine, you should see something like wheel:27:myusername.

If your username is missing from the group, take note of the group name (sudo or wheel) and add the user in question to that group:

Alpine: addgroup myusername wheel
Fedora: usermod -aG wheel myusername
Probably not necessary, but on Ubuntu/Debian: usermod -aG sudo myusername

Finally, as root, make sure that the admin group (whether sudo or wheel) is enabled for sudo:

grep -E '%sudo|%wheel' /etc/sudoers

If the line is there, but commented out with a #, then run visudo then make sure the line reads thus (use wheel or sudo as determined earlier):

%wheel ALL=(ALL) ALL

Then save.

Once these steps are complete, test again with:

su myusername
sudo -v

If you are prompted for the password, then all is well. If you instead received an error containing something like "Sorry, user myusername may not run sudo" then you may need to follow the steps again, from the beginning.

Set default user

If you obtained your Linux distro from the Store, you can likely skip this step, as the default user is already set up.

If, however, when you launch WSL, you are still root, then set your new user as the default.

Assuming you have Windows build 18980 or later: simply add a user section to /etc/wsl.conf.

Something like this will work well if you do not already have that file, or a [user] section in it:

printf "\n[user]\ndefault = myusername\n" | sudo tee -a /etc/wsl.conf

However, if on a version of Windows before build 18980, then you will instead need to edit the registry to set a default user. Before doing this, we will need two bits of information: the user id, and the name of the WSL distro. Chances are, you already know these. If not, you can obtain the user id with id -u myusername and check your list of WSL distros with (in Powershell) wsl -l

Then, use the following command in Powershell, but use your WSL distro name in place of "Alpine" and use your user id in place of "1000":

Get-ItemProperty Registry::HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Lxss\*\ DistributionName | Where-Object -Property DistributionName -eq Alpine  | Set-ItemProperty -Name DefaultUid -Value 1000

Whichever method you use, test by logging out of WSL, and then log back in. Confirm that whoami yields the correct username. Success.

Update/upgrade packages and test network connectivity

Let's make everything new and shiny with one of the following:

Debian/Ubuntu: sudo apt update && sudo apt upgrade
Fedora: sudo dnf upgrade
Alpine: sudo apk upgrade -U

Network issues?

Upgrading the packages also serves as a network test. For a variety of reasons, network connectivity issues can happen with WSL 2, and tweaking the DNS settings often resolves these problems in my experience. If the upgrade command succeeded, you can skip this section. But if the above commands fail to access the package servers, it may be something unique to your network, or your firewall or anti-malware software. I recommend the following:

echo -e "[network]\ngenerateResolvConf = false" | sudo tee -a /etc/wsl.conf
sudo unlink /etc/resolv.conf
echo nameserver 1.1.1.1 | sudo tee /etc/resolv.conf

The first line tells WSL to cease auto-configuring the /etc/resolv.conf file. Then we remove/unlink the old file, and create a new one.

With this newly-configured DNS resolver (in this case, pointing directly to Cloudflare's DNS server) you can try upgrading packages again. Success? Excellent.

Prepare for Docker installation

Thankfully, there are official guides for installing Docker on various Linux distributions. I have based these instructions on those, with some tweaks learned from real world testing.

Remove residue

If this is not a fresh install, and you may have experimented with docker before, then first clear out any residual docker installs:

Fedora: sudo dnf remove moby-engine docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-selinux docker-engine-selinux docker-engine
Debian/Ubuntu: sudo apt remove docker docker-engine docker.io containerd runc
Alpine (probably not necessary, but just in case): sudo apk del docker-cli docker-ce docker-openrc docker-compose docker

Install dependencies

Then, install pre-requisites:

Debian/Ubuntu: sudo apt install --no-install-recommends apt-transport-https ca-certificates curl gnupg2
Fedora: sudo dnf install dnf-plugins-core
Alpine: Nothing needed. Dependencies will be installed later, automatically.

Debian: switch to legacy iptables

Docker utilizes iptables to implement network isolation. For good reason, Debian uses the more modern nftables, but this means that Docker cannot automatically tweak the Linux firewall. Given this, you probably want to configure Debian to use the legacy iptables by default:

update-alternatives --config iptables

And select iptables-legacy.

If you are comfortable, instead, with nftables and want to configure nftables manually for Docker, then go for it. I suspect that most, however, will want to switch to iptables legacy.

Debian/Ubuntu package repository configuration

On Debian or Ubuntu, first temporarily set some OS-specific variables:

. /etc/os-release

Then, make sure that apt will trust the repo:

curl -fsSL https://download.docker.com/linux/${ID}/gpg | sudo tee /etc/apt/trusted.gpg.d/docker.asc

ID will be either "ubuntu" or "debian", as appropriate, depending on what is in /etc/os-release.

Then add and update the repo information so that apt will use it in the future:

echo "deb [arch=amd64] https://download.docker.com/linux/${ID} ${VERSION_CODENAME} stable" | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt update

Fedora package repository configuration

On Fedora, first add Docker's repo:

sudo dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo

Install Docker

Now we can install the official Docker Engine and client tools:

Debian/Ubuntu: sudo apt install docker-ce docker-ce-cli containerd.io
Fedora: sudo dnf install docker-ce docker-ce-cli containerd.io
Alpine (install the latest from edge): sudo apk add docker --repository=http://dl-cdn.alpinelinux.org/alpine/edge/community

Add user to `docker` group

The Docker daemon is a service that Docker requires to be running in the background. The service (dockerd) and client (docker) communicate over a socket and/or a network port. For communication over the socket, privileged access is required. Two ways to obtain this access:

Run docker as root (i.e. sudo docker)
Through group membership, grant specific users privileged access to the Docker socket

In other words, unless you want to utilize sudo or root access every time, add your user to the Docker group, named docker:

Fedora/Ubuntu/Debian: sudo usermod -aG docker $USER
Alpine: sudo addgroup $USER docker

Then close that WSL window, and launch WSL again. You should see docker when you run the command groups to list group memberships.

Sharing dockerd: choose a common ID for the `docker` group

If you only plan on using one WSL distro, this next step isn't strictly necessary. However, if you would like to have the option of sharing the Docker socket system-wide, across WSL distributions, then all will need to share a common group ID for the group docker. By default, they each may have a different ID, so a new one is in order.

First, let's pick one. It can be any group ID that is not in use. Choose a number greater than 1000 and less than 65534. To see what group IDs are already assigned that are 1000 or above:

getent group | cut -d: -f3 | grep -E '^[0-9]{4}' | sort -g

Can't decide what number to use? May I suggest 36257. (Just dial DOCKR on your telephone keypad...) Not likely to be already in use, but check anyway:

getent group | grep 36257 || echo "Yes, that ID is free"

If the above command returns a line from /etc/group (that does not include docker), then pick another number and try again. If it returns "Yes, that ID is free" then you are good to go, with the following:

sudo sed -i -e 's/^\(docker:x\):[^:]\+/\1:36257/' /etc/group

Or, if groupmod is available (which it is on Fedora, Ubuntu, and Debian, but not Alpine unless you sudo apk add shadow), this is safer:

sudo groupmod -g 36257 docker

Once the group id has been changed, close the terminal window and re-launch your WSL distro.

Note that the above steps involving the docker group will need to be run on any WSL distribution you currently have or install in the future, if you want to give it access to the shared Docker socket.

(Optionally) prepare a shared directory

As with the last step, if you only plan on using one WSL distro, this next step isn't strictly necessary. However, if you would like to have the option of sharing the Docker socket system-wide, across WSL distributions, then a shared directory accessible to all is needed.

Let's first make a shared directory for the docker socket, and set permissions so that the docker group can write to it.

DOCKER_DIR=/mnt/wsl/shared-docker
mkdir -pm o=,ug=rwx "$DOCKER_DIR"
chgrp docker "$DOCKER_DIR"

Configure `dockerd` to use the shared directory

Again, this step can be skipped if you opt against using a shared directory for the docker socket. However, you may have other settings you wish to put in daemon.json, so you may appreciate some familiarity with this topic.

I suggest using the configuration file /etc/docker/daemon.json to set dockerd launch parameters. If the /etc/docker directory does not exist yet, create it with sudo mkdir /etc/docker/ so it can contain the config file. Then the following, when placed in /etc/docker/daemon.json, will set the docker host to the shared socket:

{
  "hosts": ["unix:///mnt/wsl/shared-docker/docker.sock"]
}

Launch `dockerd`

Most Linux distributions use systemd or other init system, but WSL has its own init system. Rather than twist things to use the existing init system, we just launch dockerd directly:

sudo dockerd

There should be several lines of info, warnings related to cgroup blkio, and the like, with something like API listen on /mnt/wsl/shared-docker/docker.sock at the end. If so, you have success.

Open another wsl terminal.

If and only if you opted to use the shared docker socket in /mnt/wsl/shared-docker as detailed above, first set the DOCKER_HOST environment variable:

export DOCKER_HOST="unix:///mnt/wsl/shared-docker/docker.sock"

Then, try out the docker cli:

docker run --rm hello-world

You should see the "Hello from Docker!" message.

Launch script for `dockerd`

The following lines can be placed in .bashrc or .profile if autolaunching is desired, or in a separate shell script. For instance, you may want to create a script ~/bin/docker-service so that you can run docker-service only when you want, manually.

DOCKER_DISTRO="Debian"
DOCKER_DIR=/mnt/wsl/shared-docker
DOCKER_SOCK="$DOCKER_DIR/docker.sock"
export DOCKER_HOST="unix://$DOCKER_SOCK"
if [ ! -S "$DOCKER_SOCK" ]; then
    mkdir -pm o=,ug=rwx "$DOCKER_DIR"
    chgrp docker "$DOCKER_DIR"
    /mnt/c/Windows/System32/wsl.exe -d $DOCKER_DISTRO sh -c "nohup sudo -b dockerd < /dev/null > $DOCKER_DIR/dockerd.log 2>&1"
fi

If you went with the default docker socket location of /var/run/docker.sock instead of the shared socket directory of /mnt/wsl/shared-docker as detailed above, then the script can be something like this:

DOCKER_DISTRO="Debian"
DOCKER_LOG_DIR=$HOME/docker_logs
mkdir -pm o=,ug=rwx "$DOCKER_LOG_DIR"
/mnt/c/Windows/System32/wsl.exe -d $DOCKER_DISTRO sh -c "nohup sudo -b dockerd < /dev/null > $DOCKER_LOG_DIR/dockerd.log 2>&1"

You may choose whatever location you would like for your docker logs, of course. It just needs to be in a place that has permissions so that your user can write to it.

Note that DOCKER_DISTRO should be set to the distro you want to have running dockerd. If unsure of the name, simply run wsl -l -q from Powershell to see your list of WSL distributions. Pick the right one and set it to DOCKER_DISTRO.

The script above does the following:

Sets the environment variable $DOCKER_HOST to point to the shared socket. This isn't necessary for dockerd but it will allow running docker without needing to specify -H unix:///mnt/wsl/shared-docker/docker.sock each time.
Checks if the docker.sock file already exists and is a socket. If it is present, do nothing. If not present then the script does the remaining steps, as follows.
Creates the shared docker directory for the socket and dockerd logs, setting permissions appropriately
Runs dockerd from the specified distro. This is important, because it allows any WSL distro to launch dockerd if it isn't already running.
When dockerd is launched, pipe its output and errors to a shared log file.
When dockerd is launched, it runs in the background so you don't have to devote a terminal window to the process as we did earlier. The sudo -b flag gives us this, and we run with nohup so that it runs independent of the terminal, with an explicit null input to nohup to avoid extra warnings. Both standard output and errors are written to the logfile, hence the 2>&1 redirect.

Passwordless launch of `dockerd`

If the above script is placed in .bashrc (most Linux distros) or .profile (distros like Alpine that have Ash/Dash as the default shell), or other shell init script, then it has an unfortunate side effect: you will likely be prompted for a password most every time a new terminal window is launched.

To work around this, you can, if you choose, tell sudo to grant passwordless access to dockerd, as long as the user is a member of the docker group. To do so, enter sudo visudo and add the following line (if your visudo uses vi or vim, then be sure to press "i" to begin editing, and hit ESC when done editing):

%docker ALL=(ALL)  NOPASSWD: /usr/bin/dockerd

Save and exit (":wq" if the editor is vi, or Ctrl-x if it is nano), and then you can test if sudo dockerd prompts for a password or not. For peace of mind, you can double-check: something like sudo -k ls -a /root should still require a password, unless the password has been entered recently.

Make sure `$DOCKER_HOST` is always set

If using the script earlier to launch dockerd, then $DOCKER_HOST will be set, and future invocations of docker will not need an unwieldy -H unix:///mnt/wsl/shared-docker/docker.sock

If that script is already in your .bashrc or .profile, then the following is unnecessary. If, however, you manually invoke dockerd in some way, then the following may be desirable in your .bashrc or .profile, if you opted for the shared docker socket directory:

DOCKER_SOCK="/mnt/wsl/shared-docker/docker.sock"
test -S "$DOCKER_SOCK" && export DOCKER_HOST="unix://$DOCKER_SOCK"

The above checks for the docker socket in /mnt/wsl/shared-docker/docker.sock and, if present, sets the $DOCKER_HOST environment variable accordingly. If you want a more generalized "if this is wsl, then set the socket pro-actively" then you may prefer the following, which simply check for the existence of a /mnt/wsl directory and sets the docker socket if so:

[ -d /mnt/wsl ] && export DOCKER_HOST="unix:///mnt/wsl/shared-docker/docker.sock"

Running `docker` from Windows

If configured as above, I recommend always running docker from wsl. Do you want to run a container? Do so from a WSL window. (See my article on using Windows Terminal for a convenient way to use WSL and Powershell.)

This doesn't just apply to the terminal, either. For instance, VSCode supports docker in WSL 2.

But if you want the convenience and utility of running docker in a Powershell window, I have a couple suggestions.

One is to expose dockerd over a TCP Port, or, better yet, set up an SSH server in WSL and connect that way. docker context will likely be your friend. Such methods will be explored in a later article, but I encourage you, reader, to explore. A hint: ever tried scoop.sh? After setting it up, scoop install docker docker-compose will get you some familiar tools, then an SSH server such as Dropbear or OpenSSH on the WSL side...

A simplified method I recommend: a Powershell function that calls the WSL docker, passing along any arguments. This function can be placed in your Powershell profile, usually located at ~\Documents\WindowsPowerShell\Microsoft.PowerShell_profile.ps1

$DOCKER_DISTRO = "fedora"
function docker {
    wsl -d $DOCKER_DISTRO docker -H unix:///mnt/wsl/shared-docker/docker.sock @Args
}

Again, try wsl -l -q to see a list of your WSL distributions if you are unsure which one to use.

Make sure the Docker daemon is running, then launch a new Powershell window, and try the hello-world container again. Success?

You could also make a batch file with the appropriate command in it. For instance, name it docker.bat and place in C:\Windows\system32 or other location included in %PATH%. The following contents will work in such a script:

@echo off
set DOCKER_DISTRO=fedora
wsl -d %DOCKER_DISTRO% docker -H unix:///mnt/wsl/shared-docker/docker.sock %*

You could go a step further and ensure that dockerd is running whenever you start Powershell. Assuming that the dockerd start script detailed above is saved in a file in WSL as $HOME/bin/docker-service and is executable (try chmod a+x $HOME/bin/docker-service), then the following line in your Powershell profile will launch dockerd automatically:

wsl -d $distro ~/bin/docker-service

Not sure where your Powershell profile is located? Try entering $profile in a powershell window.

If you don't want to rely on a particular WSL shell script, you could implement a Powershell function to launch dockerd, such as this:

function Docker-Service {
  Param ([string]$distro)
  $DOCKER_DIR = "/mnt/wsl/shared-docker"
  $DOCKER_SOCK = "$DOCKER_DIR/docker.sock"
  wsl -d "$distro" sh -c "[ -S '$DOCKER_SOCK' ]"
  if ($LASTEXITCODE) {
    wsl -d "$distro" sh -c "mkdir -pm o=,ug=rwx $DOCKER_DIR ; chgrp docker $DOCKER_DIR"
    wsl -d "$distro" sh -c "nohup sudo -b dockerd < /dev/null > $DOCKER_DIR/dockerd.log 2>&1"
  }
}

This function takes one parameter: the distro name.

In all of the above, the principle is the same: you are launching Linux executables, using WSL interoperability.

A note on bind mounts: stay on the Linux filesystem

With docker, it is possible to mount a host system's directory or files in the container. The following often works, but is not advisable when launching WSL docker from Windows:

echo "<h1>Hello, World</h1>" > index.html
docker run -p "8080:80" -v "$PWD:/usr/share/nginx/html:ro" nginx

Instead of doing the above haphazardly, when launching WSL docker from Powershell, two recommendations:

For performance reasons, only bind mount from within the Linux filesystem. To get to a Linux directory while in Powershell, try something like cd (wsl wslpath -m ~/path/to/my/dir)
Instead of $PWD to get the current directory, try '$(wslpath -a .)' and, yeah, the single quotes are helpful for escaping.

An example:

cd (wsl wslpath -m ~)
mkdir html
cd html
echo "<h1>Hello, World</h1>" > index.html
docker run -p "8080:80" -v '$(wslpath -a .):/usr/share/nginx/html:ro' nginx

Then point your browser to http://localhost:8080, and happiness will result. (Depending on your network configuration, you may instead need to access this through http://[WSL IP Address]:8080 which should be obtainable with ifconfig or ip addr)

Try wsl wslpath from Powershell, or just wslpath from Linux, to see the options. This is a very useful tool, to say the least.

Example scripts

After walking through the steps in this article, you should now have a working and potentially auto-launched dockerd, shared Docker socket, and conveniently configured docker command.

Some of the code examples above have been placed in scripts in a companion Github repo. I summarize the files available here:

docker-service.sh is a Unix shell script to launch dockerd
docker-service.ps1 contains a Powershell function to launch dockerd in WSL
docker.bat is a Windows batch file for launching WSL docker from CMD
docker.ps1 contains a Powershell function for launching WSL docker from Powershell, if placed in your profile

No doubt there are ways these can be tweaked to be more useful and reliable; feel free to post in the comments.

Interested in further tinkering with WSL 2?

Bash Execution Tips for Shell Jockeys and Script Fabricators

Jonathan Bowman — Tue, 09 Feb 2021 18:24:20 +0000

Have you ever wanted to execute one command if another failed? Or one command only if the first one succeeded? What about background execution?

Bash and other popular shells such as Zsh and Ash/Dash have some useful but sometimes confusing operators for command execution.

What is the difference between &&, &, || and ;?

A very brief cheatsheet:

Use && to execute one command only when the previous one succeeds.
Use || to execute one command only when the previous one fails.
Combine the above for conditional branching.
Use ; to join two commands when you want the second to execute no matter the result of the first one.
Use & to run the first job in the background while the next executes. Follow all with wait for a clean return to the command prompt

Execute if previous command succeeds: `&&`

When running one command, sometimes it is desirable for another command to execute if and only if the first one was successful.

Let's say we want to create a user's home directory if and only if that user already exists. We can test for the user's existence with id, then, if successful, create the home directory, then, if that is successful, write a .bashrc file:

id fredflintstone && mkdir /home/fredflintstone && echo 'echo "Welcome, $USER"' > /home/fredflintstone/.bashrc

Note the use of the &&, a logical "and", for conditional execution. If this, then that.

The && operator should not be confused with the background execution operator & noted below.

Execute if previous command fails: `||`

Sometimes, the opposite of the above is desired: execute the next command if the previous one failed.

For instance, perhaps we want to add a DNS server to /etc/resolv.conf only if it doesn't already exist. We can use grep to test if the server name already exists, then write back to the file if it wasn't already there:

grep 1.1.1.1 /etc/resolv.conf || echo "nameserver 1.1.1.1" | sudo tee -a /etc/resolv.conf

Note the use of the ||, a logical "or", for conditional execution. If not this, then that.

Also note the pipe | operator. It should not be confused with the || operator. The pipe | means "send the output of this command to the input of the next." In this case, the echo command pipes output to the tee command, which appends (due to the -a option) the input to /etc/resolv.conf.

Branching logic with both `&&` and `||`

A pleasant combination of the above is indeed possible. Imagine that you want to execute a command, then, if it succeeds, execute one command, but if it fails, execute a different command. Some creative chaining is possible, as in this example:

id fredflintstone && mkdir -p /home/fredflintstone || id bettyrubble && mkdir /home/bettyrubble

In the above, if one user does not exist, another will be tried. However, it should be pointed out that this is not equivalent to an if/then/else statement. If the 2nd command mkdir -p /home/fredflintsone would fail (not likely, with the -p flag), id bettyrubble would still run. In other words, when executing true && false || true all three commands run, because the failure of the 2nd command triggers the 3rd.

Sometimes, I use this method to explicitly set human-readable variables in a script, so that later readers (i.e. me) will easily understand what is going on:

sudo passwd -S $USER && PWD_IS_SET=true || PWD_IS_SET=false

In effect, the above "caches" a test so that the script can later test for the value of $PWD_IS_SET numerous times, in a memorable and readable way.

Unconditional execution with `;`

To mash a couple commands together in one line, use the semicolon. In human language, a semicolon says, "Do this, wait until it completes, then do that." For instance:

echo Hello ; echo World

The execution is unconditional; no matter what the first command does, the second will also execute afterward:

cat filename_that_does_not_exist ; echo Continue anyway

Background execution with `&`

Try this:

echo Hello & echo World

Yes, both commands executed, but the behavior may well be quite different than the use of ; above. The above two commands were executed in parallel. In other words, at the same time. There is no guarantee regarding which will complete first. In fact using just & at the end of a long running command will allow it to run in the background indefinitely.

The main point in the context of this article: & is not && nor is it ; (the order and conditions for execution are different for each).

Testing teaser

A thorough exploration of the builtin shell command test is better left for a future article. That said, let's at least dabble a bit, as the concept is quite applicable to conditional execution.

Note: in POSIX-derived shells such as Bash, Zsh, and Ash/Dash, the test command and the [ command are the same command, with the exception that when [ is used, it should end with a ]. While [ works well in a pattern like "if [ -d some_directory ]; then" for multi-line readability (the last line should be "fi" to end the if statement), for succinct one-liners I prefer "test -d some_directory" and the like.

A quick cheatsheet with some commonly used tests using the test command:

test -d some_directory will be true if a directory exists
test -r some_file will be true if a regular file is readable
test -n "$SOME_STRING" will be true if a string (such as a variable) is non-empty
test -z "$SOME_NONEXISTENT_STRING" will be true if a string is empty

See the POSIX spec for test for many more options. You might also browse Bash Conditional Expressions or Zsh Conditional Expressions for shell-specific docs. When possible, I try to write shell scripts in POSIX-compliant ways for portability (scripts that work across a variety of shells). That said, sometimes you may prefer to use the full power of your shell's specific features. Browse the relevant docs to consider your options.

The above can be very useful for conditional execution. Something like this works well:

test -r /etc/resolv.conf || echo "nameserver 1.1.1.1" | sudo tee /etc/resolv.conf

In other words, if /etc/resolv.conf does not exist, create it with the appropriate contents. But if it already exists, do nothing.

The possibility of idempotence (alternate title: put a little Dev in your Ops)

Perhaps you can see a clear use case here for system configuration. If you have ever used Ansible and similar configuration tools, you may note that it is desirable for every task to be idempotent; in other words, even when run more than once, the desired result is the same.

I believe it is good for shell scripts to be as idempotent as possible, when they are intended to configure a system in a certain state. The above examples that reference /etc/resolv.conf do this well, so I will reference them again here:

test -f /etc/resolv.conf || echo "nameserver 1.1.1.1" | sudo tee /etc/resolv.conf
grep 1.1.1.1 /etc/resolv.conf || echo "nameserver 1.1.1.1" | sudo tee -a /etc/resolv.conf

These two lines are very similar, and only one is needed, depending on the desired outcome. The first is a great example of creating a new file with the desired initial state. It uses test -f to first determine if the file exists. Sure, you could always overwrite the file and ensure state that way, but that would update the file stats, such as timestamp, needlessly. In addition, this example may be useful on configuration files with a desired initial state, when future modifications are expected and should not be altered.

The second is instead an example of ensuring that a certain line exists in a file. It uses grep to test for a word or line in the file, then adds the line if and only if it isn't already there.

These methods achieve idempotency with the combination of testing and conditional execution. A worthy goal.

One-liners and robust scripts

The examples and concepts in this article are equally at home in a terminal window or in a substantial configuration script. Shell execution logic should be your friend. Feel free to post examples and tips in the comments below.

Dotfiles the easy way

Jonathan Bowman — Sun, 07 Feb 2021 13:42:21 +0000

In my ongoing quest to explore a variety of ways of managing config files, I believe I have found a way that is attractively simple.

Step one: git clone

cd ~
git clone -c status.showUntrackedFiles=no -n --separate-git-dir .git $REPO_URL tmpdir
rm -r tmpdir

Rationale:

status.showUntrackedFiles is set to "no" so that future git status requests only show files that were intentionally tracked with git add and git commit
-n means no checkout. We aren't ready for it yet.
--separate-git-dir .git takes some explanation. It is impossible to git clone into a non-empty directory without some extra steps. This trick does it in one step (two if you count the deletion of the throwaway directory). Tell Git to use a separate Git directory but then, sneaky miscreants that we are, we name the directory the default: .git
The undesirable side effect is an extra directory tmpdir that has a single .git file of no consequence. The entire directory can safely be removed.

Step two for non-empty repo: git checkout

git checkout

Deal with any file conflicts. For instance, you might backup and remove an existing .bashrc. Then run git checkout again. If you are sure that overwriting is OK, you can pass the force flag:

git checkout -f

Step two if new (empty) repo: add files and push

If this is a brand new setup, your repo is likely empty. Add some files:

git add .bashrc
git commit -m "initial commit of Bash config"

Repeat as necessary with additional files and directories. Then push:

git push

Step three: maintain

Keep your files up to date with git add, git commit, git push. Pull remote changes with git pull. In other words, manage your home directory as you would any other Git repo. Please avoid git add . as that will track every file in your home directory, an undesirable endeavor.

If this spawns other creative ideas or optimizations, feel free to post in the comments!

Using Multiple Git Repositories to Store Dotfiles in a Modular Fashion

Jonathan Bowman — Thu, 04 Feb 2021 12:22:54 +0000

In this article, I offer an approach for managing dotfiles in a modular way. I find a modular approach important because only some config files are useful in all contexts, while others are unique to a specific environment. For instance, my text editor configuration (.vimrc, in my case) is used on my Windows laptop, Linux laptop, FreeBSD server, and even my phone. On the other hand, files for configuring a Linux graphical environment, a developer's Macbook, Windows Subsystem for Linux (WSL), or Windows Powershell, may not make sense to clutter or confuse environments to which they do not apply.

It would be nice to use multiple Git repositories, or multiple branches of one Git repository, in order to customize various environments.

In a previous article, I outlined a simple approach to storing dotfiles that makes the entire home directory a git repository.

Let's build on that approach, exploring the possibility of using multiple repositories in a modular fashion.

Summary steps

Feel free to read the full article below for detailed explanation and options. As a quick summary, the following steps should get you started with three "modules": base, personal, and work:

Set up the base repo
Create two additional directories, such as ~/.config/custom/personal and ~/.config/custom/work and initialize a git repo in each, similar to the base instructions but using those directories instead of home.
Modularize your config files (see below) so that the main config includes related files in subdirectories of ~/.config/custom
Manage the files: cd into each module directory and add, commit, push and pull as necessary

Repository setup

You can continue to use the convenience functions from the first article (dtfnew and dtfrestore), just make sure you position yourself in the appropriate directory first, and specify the correct repo for each module. For example:

cd ~
dtfrestore $BASEREPO
mkdir ~/.config/custom
git clone $WORKREPO ~/.config/custom/work
git clone $LAPTOPREPO ~/.config/custom/laptop

In the above, we first create a base repository in the home directory. This is the repo in which are stored the main files like .bashrc, .profile, etc. As noted below, these files should be configured to load other files in other directories.

Then we clone the remote repositories to the given directories after creating them.

You can browse the companion Github repo for the basic functions used above, in both a Unix shell version and a Powershell version

One directory per "module" with a common parent directory

Choose a parent directory in which you will place each module directory. I use the term "module" here to refer to a repository that adds additional config files. I do not mean to refer to git submodules. Although that introduces possibilities worth exploring another day...

I use ~/.config/custom as the parent directory, but you can use any location that serves you well.

Underneath that parent directory, create a directory for each "module." The end result may look something like this:

~/.config/custom/
├── base
├── macbook
├── personal
├── server
├── work
└── wsl

Perhaps you don't need that many, but you get the idea.

Within each directory, you can place relevant config files. I suggest some advance planning to determine naming scheme, as follows.

Modularize your config files

An important strategy with a modular approach is to compartmentalize. Rather than imagining a single config file that changes per system, use that file to load other config files if they are present.

Here are some ideas, specific to various tools:

Unix shell configs

Shell configurations like .bashrc or .zsheenv or .profile can source files from other directories.

For instance, in .bashrc we might place something like the following:

for file in "$(find $HOME/.config/custom -name 'bashrc')"; do . $file; done

This would load any or all of the following files if they exist:

~/.config/custom/work/bashrc
~/.config/custom/personal/bashrc
~/.config/custom/wsl/bashrc
~/.config/custom/linuxui/bashrc

SSH configs

In .ssh/config, the Include keyword can be used like so

Include ~/.config/custom/*/*.ssh

This will include any files ending in .ssh (such as config.ssh) in any subdirectory of ~/.config/custom. For example, this will include any of the following files, if they are available:

~/.config/custom/work/config.ssh
~/.config/custom/personal/personal.ssh
~/.config/custom/server/hosts.ssh

And so on. Name the directories in ways that suit you.

Powershell

In Windows, the Powershell profile in ~\Documents\WindowsPowerShell\Microsoft.PowerShell_profile.ps1 is automatically loaded. Within that file, you could loop through files in a directory of your choice and load them:

Get-ChildItem -Recurse "$HOME\.config\dotfiles\*.ps1" | ForEach-Object {& $_.FullName}

For example, this will include the following files, if they are available:

~\.config\dotfiles\dc1\profile.ps1
~\.config\dotfiles\work\profile.ps1
~\.config\dotfiles\personal\extra.ps1

Vim

With Vim or Neovim, you can load multiple files from a directory of your choice, using the runtime command. For instance, add something like the following to ~/.vimrc:

runtime! ../.config/custom/**/*.vim

This will load any and all files ending with .vim in ~/.config/custom or any subdirectory thereof. Note that the directory path is relative to the vim runtime path ($VIMRUNTIME), hence the .. at the beginning of the path.

Maintaining config files in module directories

The above are examples to demonstrate a theme: create base config files that simply load an arbitrary number of other files, in a directory of your choosing. Once this is done, individual files as well as directories of files can be added to Git repo(s).

As noted, this requires thinking through directory and file structure carefully, because files are tracked in entirely separate git directories. But that careful organization pays off with simplicity: cd into the module directory, then use git as you like, no extra command line options needed. For instance, imagine that we have two modules: base and personal, with base being our main repo in $HOME and personal having an additional person Vim config in ~/.config/custom/personal/personal.vim. Initiating the tracking could look something like this:

cd ~
git add ~/.vimrc
git commit -m "New Vim config"
git push
cd ~/.config/custom/personal
git add personal.vim
git commit -m "Addition Vim config for personal laptop"
git push

Other hints and recipes?

I hope this offers you some ideas and inspiration for your own configurations. Please feel free to comment below with suggestions and experiences.

Get Git Default Branch from the Command Line (Powershell or Bash/Zsh)

Jonathan Bowman — Fri, 29 Jan 2021 12:40:42 +0000

On occasion, one needs to know the default branch for a given Git repo. Below I have compiled methods that fit a variety of use cases, and cover specific platforms (Github and Gitlab) as well as methods that work universally regardless of remote platform.

Github API method

Given a repo username/reponame, the following script will work in Bash, Dash/Ash, or Zsh, provided you have the wget tool, and sed:

wget -q -O - "https://api.github.com/repos/username/reponame" | sed -nE 's/^\s+"default_branch": "([^"]+).+$/\1/p'

If you have jq it is even simpler:

wget -q -O - "https://api.github.com/repos/username/reponame" | jq -r '.default_branch'

Or for those who prefer curl:

curl -s "https://api.github.com/repos/username/reponame" | jq -r '.default_branch'

This is pretty easy in Powershell, as well:

iwr -useb https://api.github.com/repos/username/reponame | ConvertFrom-Json | select -exp default_branch

In summary, we first query information about the given repo, then parse the JSON that is returned, extracting the value of the default_branch key.

Is this something you might use repeatedly? Consider building a function and placing the following in your shell config (such as .bashrc):

defbranch () {
  REPO=$1
  wget -q -O - "https://api.github.com/repos/$REPO" | sed -nE 's/^\s+"default_branch": "([^"]+).+$/\1/p'
}

Feel free to use this gist

Or, for Windows Powershell, place the following in your Documents\WindowsPowerShell\Microsoft.PowerShell_profile.ps1:

function defbranch {
  Param ([string]$repo)
  iwr -useb https://api.github.com/repos/$repo | ConvertFrom-Json | select -exp default_branch
}

Feel free to use this gist

Similar functionality on Gitlab

The above is specific to Github, because it is specifically a Github setting. In Github, even an empty repo can have a default branch that is different than master.

Gitlab, another popular "social coding" service, has different but similar functionality. A populated repo can have a default branch. An empty one will not.

Here is an example function that uses the Gitlab API (see gist):

gldefbranch () {
  REPO=$(echo $1 | sed "s/\//%2F/g")
  BRANCH=$(wget -q -O - "https://gitlab.com/api/v4/projects/$REPO" | sed -nE 's/.*"default_branch":"([^"]+).*/\1/p')
  [ "$BRANCH" == null ] && echo master || echo $BRANCH
}

and the Powershell equivalent:

function gldefbranch {
  Param ([string]$repo)
  $repo = $repo.replace("/","%2F")
  $branch = (iwr -useb "https://gitlab.com/api/v4/projects/$repo" | ConvertFrom-Json | select -exp default_branch)
  if ($branch) {
    echo $branch
  } else {
    echo "master"
  }
}

A better, universal approach if repo is non-empty

My original intent was to read a Github setting on a repo, new or old, empty or not. However, if desiring to simply get the default branch on an existing repo, an API call is unnecessary. We can use git ls-remote.

Thanks to u/Cyberbeni in this Reddit discussion for these suggestions.

To query the default branch on a remote repository, try this, with your repository URL assigned to or substituted for $REPO_URL:

git ls-remote --symref "$REPO_URL" HEAD | sed -nE 's|^ref: refs/heads/(\S+)\s+HEAD|\1|p'

Or Powershell equivalent:

"$(git ls-remote --symref 'https://gitlab.com/bowmanjd/dotfiles1.git' HEAD)" -replace '.*?^ref: refs/heads/(\S+).+','$1'

Get local origin/HEAD reference if available

If the repo was downloaded with git clone, then origin/HEAD is set automatically. In such cases, the following should work quickly, without needing to query the remote:

git symbolic-ref refs/remotes/origin/HEAD | cut -d '/' -f4

In Powershell:

(git symbolic-ref refs/remotes/origin/HEAD) -split '/' | select -Last 1

Please note the above is unreliable, as it is easily possible to work locally without origin/HEAD being set.

Again, thanks to u/Cyberbeni for the tip on Reddit.

Get locally-set Git preference

Of course, if you just want to know the current user's preferred default branch, the following will yield the results, or blank if none has been set up:

git config init.defaultBranch

One can set this value to, for instance, main with something like

git config --global init.defaultBranch main

Happy scripting!

DEV Community: Jonathan Bowman

Linux on the Dell XPS: Fixing AX201 Wi-Fi performance

The NetworkManager solution

wifi.powersave config options

Detecting and setting the power-save setting on the wireless card

Awaiting other solutions...

Other resources

How to Upgrade to Fedora 37 In Place on Windows Subsystem for Linux (WSL)

Backup first

Begin the upgrade: freshen up

Install the upgrade system

Download new release packages

Reboot?

Moment of truth: launch the upgrade

Finalization

Reference: all the steps in one place

Keep upgrading

Two Methods for Testing HTTPS API Calls with Python and pytest and Also Communicating with the in-Laws

A potentially awkward social situation, solved by HTTPS

The tests

pytest-httpserver

Adding SSL with trustme

VCR.py

Improved family dynamics

Alternatives

Also of interest

Hardening and Simplifying Python's urlopen

Security problems with the default urlopen()

A suggested solution

The handlers, defined

Flexibility

HTTP Calls in Python Without Requests or Other External Dependencies

Consider installing a library

Summary code

An introduction to urllib.request.urlopen()

HTTP errors

A versatile Response object

Requests with data

The query string

Sending data in the request body

Sending JSON in the request body

Parsing JSON in the response body

Other tricks or suggestions?

A "POSIX Playground" Container for Shell Script Testing

Using the container

Posh: an obscure shell with great usefulness

Writing Bash Scripts that are not only Bash: Checking for Bashisms and testing with Dash

A summary checklist

An example

Dash, a POSIX compliant shell

Testing the example

Avoid double-bracket tests [[ ]]

Use vanilla read

Use printf when newlines are at issue

Refactoring the example

Finding the bashisms with checkbashisms

Pursuing best practices with shellcheck

Measure against the POSIX specification

Consider other languages and configuration tools

Other resources

Install Docker on Windows (WSL) without Docker Desktop

Are you sure you don't want Docker Desktop?

Have you tried Podman?

Make sure that WSL is version 2

Install a Linux distro

Configure a non-root user

Configure admin (sudo) access for the non-root user

Set default user

Update/upgrade packages and test network connectivity

Network issues?

Prepare for Docker installation

Remove residue

Install dependencies

Debian: switch to legacy iptables

Debian/Ubuntu package repository configuration

Fedora package repository configuration

Install Docker

Add user to docker group

Sharing dockerd: choose a common ID for the docker group

(Optionally) prepare a shared directory

Security problems with the default `urlopen()`

An introduction to `urllib.request.urlopen()`

A versatile `Response` object

Avoid double-bracket tests `[[ ]]`

Use vanilla `read`

Use `printf` when newlines are at issue

Finding the bashisms with `checkbashisms`

Pursuing best practices with `shellcheck`

Add user to `docker` group

Sharing dockerd: choose a common ID for the `docker` group

Configure `dockerd` to use the shared directory

Launch `dockerd`

Launch script for `dockerd`

Passwordless launch of `dockerd`

Make sure `$DOCKER_HOST` is always set

Running `docker` from Windows

Execute if previous command succeeds: `&&`

Execute if previous command fails: `||`

Branching logic with both `&&` and `||`

Unconditional execution with `;`

Background execution with `&`