DEV Community: Anson VanDoren

Solving the 'Broken Calculator' problem the hard way

Anson VanDoren — Sun, 28 Feb 2021 18:37:32 +0000

I don't normally do LeetCode problems...

Although I really enjoy coding, and frequently make good use of those skills as part of my 9-5 job, I've never really spent much time solving challenges on LeetCode or similar sites. I enjoy the thought exercises, but haven't "needed" the experience to get an edge in software interviews (although whether or not this should actually give anyone an edge is highly debatable), and when I do have time to sit down and write some code it's usually to solve an immediate, practical problem. None of this is to say that I don't or wouldn't enjoy being able to do this, it's just a fact of life for me.

So I almost ignored an opportunity last weekend when my brother pointed me toward an interesting entry and challenged me to submit a solution.

Note: you will likely have to sign up for an account with LeetCode to see the problem on their website. If you don't feel like doing that, just keep reading - I'll summarize it below.

...but when I do...

The puzzle was, as seems to often be the case, one that looked easy at first, until you go to write code to solve it. There are several variations on this problem, but here's the version in this particular challenge:

You have a calculator that is somehow broken so badly that all that's left functional is the screen (which can display numbers between 1 and 1 billion), and two other buttons. Oddly, these buttons are two that don't show up on any calculator I've seen, namely a double and a decrement (by one) operation.

The challenge is to get from a starting number on the calculator to a given second number using only those two functions.

The example given is:

Input: x = 2, y = 3
Output: 2
Explanation: Use double operation, and then decrement operation [2 -> 4 -> 3]

Anyway, calculator vagaries aside, the code needed to solve this puzzle should:

Given any starting number x
Using only the double (x = x * 2) and/or decrement (x = x - 1) operations
Determine the minimum operations needed to arrive at arbitrarily chosen y

The immediately obvious, and (as far as I can tell) wrong way to look at this problem is to do what's stated, and try to get from x to y. Having done a few coding challenges before, I've learned that it's often helpful to start by working backwards from the solution, and that turned out to be useful in this case as well.

Once I got to this backwards mindset, the rest fell into place pretty quickly:

If y is less than x:
- Decrement x until it equals y and return the number of operations. There is no doubling in this case.
- A quicker way to look this is to just return x - y
If y equals x, then return 0
If y is greater than x:
- If y is an odd number, increment y by 1 (since we're going backwards, incrementing here is equivalent to decrementing if starting from x)
- If y is an even number:
- If y is greater than x, divide y by 2 (equivalent to doubling x, one of the two permitted operations).
- If y is less than x, then increment y until you arrive at x
Loop through this sequence until y is equal to x

This ends up looking like some variation of the below Python code:

# first pass algorithm
def min_ops(x: int, y: int) -> int:
    if y < x:
        return x - y
    ops = 0
    while y != x:
        if y > x and y % 2 == 0:
            y /= 2
        else:
            y += 1
        ops += 1
    return ops

# testing code
if __name__ == '__main__':
    test_cases = [(2, 3, 2)]

    for case in test_cases:
        x, y, expected = case
        result = min_ops(x, y)
        if result != expected:
            print(f"{x}->{y} should have been {expected}, but got {result}")
        else:
            print(f"Correctly solved {x}->{y} in {result} operations")

...I prefer a more complicated answer...

There's absolutely nothing wrong with this algorithm. It gives the correct answer, it meets LeetCode's criteria for memory usage and runtime, and it's pretty easy to understand.

However... it really bugged me that I had to iterate through each state of the answer to arrive at the result. For really, really big numbers, this would take a long time. I know, I know...

It doesn't really matter
The answer is correct
It meets spec
Perfect is the enemy of good enough
Premature optimization is the root of all evil

... and all that.

In real life, I almost certainly would've stopped here unless there was a business reason for optimizing further. But it was the weekend, and although the weather was perfect for spending outside, and I had a ton of other side projects I wanted to get to, it couldn't take too much longer to arrive at a solution I actually liked, right? ... right??

Many hours and 18 sheets of scratch paper later...

Patterns are fun (if you're a nerd)

Here we go.

Let's start with the case where x is 1, and see what it takes to get to various y values:

A few things are definitely going on here:

There is an even/odd component, since the last operation for any odd number must be a decrement, and the last operation for any even number may not be a decrement.

There is a "powers of 2" element. Notice that (with x = 1, at least) the set y = [2, 4, 8, 16, ... 2^n] is composed solely of double operations, with no decrements.

In addition to the above, one can also see that there are the same number of double operations for all numbers in the set y = [2^n + 1...2^(n+1)]. For example, each of y = [9, 10 ... 15, 16] has exactly 4 double operations, with varying numbers of decrement operations. The same is true (but with exactly 3 double operations) for y = [5, 6, 7, 8].

Comparing the y = [9, 10 ... 15, 16] set again, it appears that the "upper" half is a repeat of the "lower half", but instead of each row having at least 3 double operations at the start, each has instead two doubles, a decrement, and then another double. Along with this, the "right side" of the pattern from y = [13, 14, 15, 16] is repeated, but one operation "rightward".

Finally, the same pattern at the trailing edge of the y = [9 ... 16] set is also seen in the y = [5 ... 8] set, and looks like it might be starting in the y = [3, 4] set, but isn't complete.

How does this make an algorithm?

So glad you asked...

Let's start with same condition to handle the case when y is less than x. There's not much fun going on in that portion, since it's just a matter of decrementing until you arrive at the result.

def min_ops_magic(x: int, y: int) -> int:
    if y < x:
        return x - y

Next, let's take advantage of the fact that all values in y = [2^n + 1...2^(n+1)] have the same number of double operations. This number of double operations is the same as the power of 2 for the upper bound of that group, or n+1 in the equation above.

In our example, the set y = [9, 10, ... 15, 16] contains all numbers such that y > x * 2^3 and y ≤ x * 2^4. So I'll refer to these numbers as being in the "n == 4" group. If I use up those 4 operations, I'm left with this:

So what I need to determine is:

Given inputs x and y, what is the smallest n such that y ≤ x * 2^n

There are a few different ways to code this. I'm also calculating the 2^n at the same time here, since we'll need that later

Option 1

import math
from typing import Tuple

def next_pow2(x: int, y: int) -> Tuple[int, int]:
    n = math.ceil(math.log2(y/x))
    pow_of_two = 2 ** n

    return n, pow_of_two

This option works because:

y = x * 2^n

y/x = 2^n

log(y/x) = log(2^n)

log(y/x) = n * log(2)

log(y/x) / log(2) = n

log2(y/x) = n

Rounding n up to the next highest integer means we can solve for n anywhere in the given range.

Option 2

from typing import Tuple

def next_pow2(x: int, y: int) -> Tuple[int, int]:
    n = 0
    pow_of_two = 1
    while x * pow_of_two < y:
        pow_of_two <<= 1
        n += 1

    return n, pow_of_two

This option uses bit-shifting to count how many times we need to shift a bit left (starting from binary 1), which is the same operation as incrementing n:

0001 == 1

0010 == 2

0100 == 4

1000 == 8

Initially I thought that Option 2 would've been faster, but after some timing checks it seems I was wrong, and the log2() operation was noticeably quicker.

Option 3

from typing import Tuple

def next_pow2(x: int, y: int) -> Tuple[int, int]:
    n = int((y - 1) / x).bit_length()
    pow_of_two = 2 ** n

This option is similar to Option 2, but it uses a Python built-in method on the integer type to return how many bits are needed to represent the number. The upper limit (16, here) has one more bit than all the other numbers in the group, so I subtract that out first.

Option 3 was the one I discovered last, and is the quickest of the three. If you have a faster way to calculate this number in Python, I'd love to hear it!

Let's add Option 3 to our algorithm. I'll inline it since I don't need to call from anywhere else:

def min_ops_magic(x: int, y: int) -> int:
    if y < x:
        return x - y

    n = int((y - 1) / x).bit_length()
    pow_of_two = 2 ** n

The "magic", Part 1 - LSBs

OK, this is where I thought things started to get a little weird. I'm sure there's a great mathematical explanation for it, but I can't exactly see it yet. Let me show you how I got to the answer, though.

If you look back up at the previous diagram, you'll see I grouped the remaining operations in a specific way. There is a J-tetromino (rotated 180°) that's repeated in both the upper and lower halves of the group, and an I-tetromino that only occurs on the top half, and pushes the upper J out by one. And no, the fact that there's Tetris pieces in this puzzle isn't the weird part.

Based on the work from the previous section, I thought it made sense to number the rows counting upwards and starting at the upper bound. It's probably not clear what I mean, so here's another diagram.

After fiddling around for rather a long time trying to figure out how to calculate the number of operations expressed by each row, I happened to write out the index value out in binary:

See the pattern? I didn't at first, either. Try now:

There's a very tempting, obvious, and generally incorrect solution that involves adding all the "1" bits of the index number to get the remaining operations needed. In fact, it does work for the case I'm showing here, but it doesn't work for all cases. I'll get to that part in a second. Hopefully for now you can just take my word for it, at least for a couple more steps.

Instead of adding all of the "1" bits together, I only want to add n least significant bits. I know, I know... in this case that means all of the bits. But I promise that's not the case for all xs and ys.

Let's get some more code going. This is only an additional two lines, so I'll just add it to the algorithm we already have and highlight the changes below:

def min_ops_magic(x: int, y: int) -> int:
    if y < x:
        return x - y

    n = int((y - 1) / x).bit_length()
    pow_of_two = 2 ** n

    # count down from top of 'bin'
    idx = x * pow_of_two - y

    # sum the '1's of the `n` least significant bits
    lsb_ones = bin(idx)[-n:].count("1")

Getting the index is fairly straightforward: just subtract Y from the highest number in this 'bin'.

Summing up the "1"s in the n least significant bits is pretty Python-specific, and there's several other ways to do it as well. In Python 3.10 (which should be released this year), there's actually a new int.bit_count() method which will do exactly this, and much faster. At any rate, this line is just converting the integer idx to a binary string representation, slicing only the last n characters of that string, and then counting how many of those characters are "1".

Again, note that even though in this particular case the number of bits in these numbers happens to be 3 or fewer (for n == 3), that isn't always the case, so we can't count all of the bits.

We're very close now. As promised, here's a case where we we need to add one final term to get our total operations:

The "magic", Part 2 - MSBs

I chose a somewhat-but-not-quite random x value, and set of y values for this example. Feel free to verify this works for other values, though.

Spoiler alert: it does

Let's check with x = 10 and the bucket of y = [21, 22 ... 39, 40]

Again, I've blocked out the n=2 operations, and am left with a few tetronimos, and some big boxes that would be a Tetris nightmare. You can see I've also filled in the idx values for each row and written them out in binary.

Remember how I said we can only use the n least significant bits in the previous calculation? Take a look in this example what would happen if we used all of them. Wrong answers all over the place...

OK, so if we can only use the n LSBs for that calculation, why did I write out all the MSBs anyway?

Only because this is the final piece of the puzzle.

For the LSBs, we just summed the number of "1" bits to get our J-tetronimos (tetronima? idk...) For the MSBs, though, we're going to keep treating them as real numbers, but only after we've gotten rid of those already-used LSBs.

You're probably way ahead of me on this, but you can see that if you treat the bits to the left of the vertical line as their own numbers, you get the number of extra columns needed to complete the total operations. Here it is a little clearer:

All that's left is a bit of coding:

def min_ops_magic(x: int, y: int) -> int:
    if y < x:
        return x - y

    n = int((y - 1) / x).bit_length()
    pow_of_two = 2 ** n

    # count down from top of 'bin'
    idx = x * pow_of_two - y

    # sum the '1's of the `n` least significant bits
    lsb_ones = bin(idx)[-n:].count("1")

    # bit-shift `n` to the right to get only the MSBs
    msb_vals = idx >> n

    # the grand finale
    return n + lsb_ones + msb_vals

Since we've already gotten what we needed from the n LSBs of the index, just shift it right by n bits to get at the MSBs that we care about. No special conversion needed after that - we're still just treating it as an integer.

Finally, the answer to the question of life, the universe, and everything (or at least this LeetCode puzzle) is simply the sum of the 2's exponent, the sum of the n LSB "1" values, and the number represented by the leftover MSBs.

Crazy, right?

But was it worth it?

tldr; yes

Besides everything I learned from this (including a lot of brushing up on bitshifting math that I ultimately didn't need for the solution anyway), I also ended up with a more satisfying (to me), faster (for most cases) solution. Your results may vary, but on my machine for an arbitrary set of ~80 different input conditions, the "magic" solution is ~20% faster.

Bonus points if you can see how this explanation relates to Hamming weight and/or to this unrelated coding challenge (sorry, I can't find a link in English, but Google Translate works well enough).

As always, please do let me know if you find any mistakes in the code or explanation, or if you have questions.

Captive Web Portal for ESP8266 with MicroPython - Part 4

Anson VanDoren — Thu, 02 Jan 2020 14:21:35 +0000

At the end of Part 3 of this series, I had both a DNS and HTTP server up and running on my Wemos D1 Mini MCU to make a captive portal that will let me set up the D1 Mini to join my home WiFi without needing to hardcode the SSID and password on each new project I make.

All HTTP requests from any client connected to the D1 Mini's WiFi AP are redirected to a web page asking for the home WiFi credentials, but so far nothing happens when you tap the "Connect" button other than being redirected back to the form. In this part, I'll wire up the code to take the submitted SSID and password, and have the MCU try to connect with them.

Parsing the form submission

I implemented the HTML form action as a GET request instead of POST. This is really the "wrong" way to do it, but since this is a very small, special-purpose web server, I didn't see any reason to go to the trouble of parsing POST requests as well. So the new WiFi SSID and password will come in as query parameters in a GET request to the /login path.

The way I chose to do this is to implement the /login route as a function instead of a static file. When that route is requested, the function will try to get the SSID and password from the query parameters and save them. Since we also have the opportunity to modify the headers, I'll send a redirect back to the / path for now. Later in this post, we'll have the HTTP server monitor whether we're connected to my home WiFi or not, and if we are, then it will start serving a different page at the / route instead of the no-longer-needed login form.

First, let's modify the get_body() function so that it knows how to handle a route that's a function instead of a bytestring pointing to a static file.

# captive_http.py
...
class HTTPServer(Server):
    ...
    def get_response(self, req):
        """generate a response body and headers, given a route"""

        headers = b"HTTP/1.1 200 OK\r\n"
        route = self.routes.get(req.path, None)

        if type(route) is bytes:
            # expect a filename, so return contents of file
            return open(route, "rb"), headers

        if callable(route):
            # call a function, which may or may not return a response
            response = route(req.params)
            body = response[0] or b""
            headers = response[1] or headers
            return uio.BytesIO(body), headers

        headers = b"HTTP/1.1 404 Not Found\r\n"
        return uio.BytesIO(b""), headers
    ...

Next, I'll create a method called login() to pull the credentials and save them. I'll also add this to my routes dictionary:

# captive_http.py
...
class HTTPServer(Server):
    def __init__(self, poller, local_ip):
        ...
        self.routes = {b"/": b"./index.html", b"/login": self.login}
        self.saved_credentials = (None, None)
        ...

    def login(self, params):
        ssid = params.get(b"ssid", None)
        password = params.get(b"password", None)
        if all([ssid, password]):
            self.saved_credentials = (ssid, password)

        headers = (
            b"HTTP/1.1 307 Temporary Redirect\r\n"
            b"Location: http://{:s}\r\n".format(self.local_ip)
        )

        return b"", headers
    ...

Now that we have some (hopefully) valid WiFi credentials, we need get the captive portal to try to log in to my home WiFi using those.

Attempting a WiFi connection

Turning our attention back to the event loop in the CaptivePortal class, we need to start checking whether the HTTP server has gotten any login credentials. If it does, we should try to log in to the new WiFi.

I'll add a method that tries to connect to my home WiFi with the provided credentials, if we're not already connected.

# captive_portal.py
...
class CaptivePortal:
    ...
    def check_valid_wifi(self):
        if not self.sta_if.isconnected():
            if self.has_creds():
                # have credentials to connect, but not yet connected
                # return value based on whether the connection was successful
                return self.connect_to_wifi()
            # not connected, and no credentials to connect yet
            return False
        return False

    def has_creds(self):
        self.ssid, self.password = self.http_server.saved_credentials
        return None not in self.http_server.saved_credentials
    ...

Then, call this method on each event loop:

# captive_portal.py
...
class CaptivePortal:
    ...
    def captive_portal(self):
        ...
        try:
            while True:
                gc.collect()
                # check for socket events and handle them
                for response in self.poller.ipoll(1000):
                    sock, event, *others = response
                    is_handled = self.handle_dns(sock, event, others)
                    if not is_handled:
                        self.handle_http(sock, event, others)

                if self.check_valid_wifi():
                    print("Connected to WiFi!")
        except KeyboardInterrupt:
            print("Captive portal stopped")

Go ahead and try this out. If you enter your home WiFi SSID and password correctly, you should see these lines somewhere in the MCU output:

Entering REPL. Use Control-X to exit.
>
MicroPython v1.12 on 2019-12-20; ESP module with ESP8266
Type "help()" for more information.
>>>
>>> import main
...
Trying to connect to SSID 'MyHomeWifi' with password notmyrealpassword
#15 ets_task(4020f4d8, 28, 3fffa410, 10)
Connection in progress
Connection in progress
Connection in progress
Connected to MyHomeWifi
Wrote credentials to ./wifi.creds
Connected to WiFi!
Captive portal stopped
Cleaning up
DNS Server stopped

Updating the servers after connection

Note: now that the MCU has successfully connected once, it's saved the credentials to its flash memory and next time you restart the code, it will automatically connect without needing to start up the DNS and HTTP servers. To continue to test the rest of this project, you'll need to delete the /pyboard/wifi.creds file to actually get the captive portal to start up.

Now that the MCU is connected to my home WiFi, I need to update the HTTP server with its new IP address on the other network so that redirections continue to work. I can safely shut down the DNS server at this point since I've already redirected to an IP address instead of domain name. Eventually, I'll want to shut down the HTTP server as well, now that the MCU is configured, but I do want to show a "connection successful" page that displays the MCU's new IP address on my home network.

Let's turn off the DNS server first, since it's no longer doing anything. This is a one-line addition to the captive portal event loop, but we'll also add a line in the same spot to update the HTTP server with its new address and SSID:

# captive_portal.py
...
class CaptivePortal:
    ...
    def captive_portal(self):
        ...
        try:
            while True:
                gc.collect()
                # check for socket events and handle them
                for response in self.poller.ipoll(1000):
                    sock, event, *others = response
                    is_handled = self.handle_dns(sock, event, others)
                    if not is_handled:
                        self.handle_http(sock, event, others)

                if self.check_valid_wifi():
                    print("Connected to WiFi!")
                    self.dns_server.stop(self.poller)
                    self.http_server.set_ip(self.local_ip, self.ssid)
        ...

For the HTTP server, we need to do a few things in the set_ip() function we're about to write:

Change the existing attribute local_ip to the new IP address. This will cause all redirects to point to the new IP address, which is on my home WiFi network instead of the MCU's access point. This way, when I turn off the MCU's access point (which I'll code up soon), I can still see the status page when my phone reconnects to my home network.
Create a new property called ssid that is initially set to None, and updated to the new SSID after we connect to it.
Change the routes dictionary so that after we're connected, the root path (/) points to a different location, which we'll write as a callable route.

# captive_http.py
...
class HTTPServer(Server):
    def __init__(self, poller, local_ip):
        ...
        self.ssid = None

    def set_ip(self, new_ip, new_ssid):
        """update settings after connected to local WiFi"""

        self.local_ip = new_ip.encode()
        self.ssid = new_ssid
        self.routes = {b"/": self.connected}
    ...

In the future, I may want a more complicated "connected" page, but for now I just want to display the new SSID the MCU connected to as well as its IP address. To do this, I'm going to make a rudimentary template HTML file, and have the connected() function fill in those values before serving it. Let's start with a new HTML page:

<!-- connected.html -->
<html>
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>Connected!</title>
  </head>
  <body>
    <h1>Connected!</h1>
    <p>
      Device is connected to WiFi access point <strong>'%s'</strong> with IP
      address <strong>%s</strong>
    </p>
  </body>
</html>

Note the %s in two places where I want to fill in the dynamic data: I'll use Python's built-in string formatting methods to convert these into the actual strings I'm looking for after reading in the contents of the HTML file like this:

# captive_http.py
...
class HTTPServer(Server):
    ...
    def connected(self, params):
        headers = b"HTTP/1.1 200 OK\r\n"
        body = open("./connected.html", "rb").read() % (self.ssid, self.local_ip)
        return body, headers
    ...

Go ahead and test it out. After logging in, you should be redirected to a page showing your home WiFi SSID, and the MCU's new IP address on that network. Additionally, if you look at the address bar in your browser, you should see that it's been updated to the new IP. If you disconnect from the MCU's access point now, and reconnect to your home WiFi, you should see that the page remains available, showing that you are in fact connected.

Gracefully exiting

We've basically accomplished what we want to now, and have no more need for the captive portal. I do want to keep the HTTP server up at least temporarily, since I may want to add some configuration options the user can select after the MCU is connected to home WiFi. We can, however, shut down the MCU's access point interface. To make sure the transition is complete, I'll add some time delay before shutting it down.

The check_valid_wifi() method seems like a good place to put this so that it can coordinate only having one network interface active at a time:

# captive_portal.py
...
class CaptivePortal:
    ...
    AP_OFF_DELAY = const(10 * 1000)

    def __init__(self, essid=None):
        ...
        self.conn_time_start = None
    ...
    def check_valid_wifi(self):
        if not self.sta_if.isconnected():
            if self.has_creds():
                # have credentials to connect, but not yet connected
                # return value based on whether the connection was successful
                return self.connect_to_wifi()
            # not connected, and no credentials to connect yet
            return False

        if not self.ap_if.active():
            # access point is already off; do nothing
            return False

        # already connected to WiFi, so turn off Access Point after a delay
        if self.conn_time_start is None:
            self.conn_time_start = time.ticks_ms()
            remaining = self.AP_OFF_DELAY
        else:
            remaining = self.AP_OFF_DELAY - time.ticks_diff(
                time.ticks_ms(), self.conn_time_start
            )
            if remaining <= 0:
                self.ap_if.active(False)
                print("Turned off access point")
        return False
    ...

Odds and ends

This project is just about finished. A few final notes that may help you get this set up on your own:

Sometimes either the server, or my phone would screw up the socket connection and I'd get ECONNRESET errors. Using the built-in function machine.reset() on the MCU usually fixed this, but sometimes I'd need to completely close the browser app on my phone and start it again to clear out the connection.
Occasionally, I'd find that the HTTP server was taking forever to respond to requests. I added the @micropython.native annotation to the HTTPServer.handle() function which seemed to improve this.
You may find that the 10 second delay for shutting off the MCU's access point may not be long enough for the redirect to go through. It's easy enough to bump up until you find what works for you.

I'd love to hear your feedback on this project! If made some cool modifications/additions to the code, integrated it with your project, or found some issues, please let me know in the comments, or on the GitHub repo!

Code: GitHub project repo

Captive Web Portal for ESP8266 with MicroPython - Part 3

Anson VanDoren — Wed, 01 Jan 2020 15:13:31 +0000

In Part 2 of this series, I finished implementing the "captive portal" DNS server so that after connecting to my Wemos D1 Mini MCU's WiFi access point, all DNS queries pointed toward the MCU's IP address instead of the actual IP address for the requested domain. I'm implementing this captive portal as a way to be able to configure the MCU to be able to log into my home WiFi without hardcoding the SSID and password, so what I need to do next is set up a HTTP server that will present a form where I can fill out these details.

So far, the CaptivePortal class creates the DNS Server and registers its socket with a poller that listens for data on the socket stream. This was fairly straightforward since UDP (which DNS uses) is connectionless, and I didn't need to keep track of whether a connection was fully closed or not. I'll do something similar for the HTTP server, but will need to keep track of which connections are incoming and outgoing, and which are closed. This makes things a little more complicated, but I'll start with my base server class, and proceed step by step from there.

Basic HTTP server with sockets

Create a new file called captive_http.py and add the code below:

# captive_http.py
import usocket as socket

from server import Server

class HTTPServer(Server):
    def __init__(self, poller, local_ip):
        super().__init__(poller, 80, socket.SOCK_STREAM, "HTTP Server")
        if type(local_ip) is bytes:
            self.local_ip = local_ip
        else:
            self.local_ip = local_ip.encode()
        self.request = dict()
        self.conns = dict()

        # queue up to 5 connection requests before refusing
        self.sock.listen(5)
        self.sock.setblocking(False)

    def handle(self, sock, event, others):
        if sock is self.sock:
            # client connecting on port 80, so spawn off a new
            # socket to handle this connection
            print("- Accepting new HTTP connection")
            self.accept(sock)
         elif event & select.POLLIN:
            # socket has data to read in
            print("- Reading incoming HTTP data")
            self.read(sock)
       elif event & select.POLLOUT:
            # existing connection has space to send more data
            print("- Sending outgoing HTTP data")
            self.write_to(sock)

Here's the basis for how the HTTP server will look. It has a handle() method just like the DNS server did, but in this case we need to consider three different cases:

The client wants to initiate a new request from the HTTP server. In this case the event will come in over the original "server" socket, on port 80. I'll handle this in the accept() method by spawning off a new "client" socket to handle that particular request/response.
Note: "client socket" in this case refers to a socket belonging to the server, but which is spawned to handle a specific client request. By contrast, the "server socket" in this discussion is only responsible for accepting new connections and creating a "client socket" for them, but doesn't itself respond to any request.
An existing connection (on a client socket) has sent us more data. This is signified by a POLLIN event on the client socket, and calls the read() method to read in the data from the socket until we get to the end of the full message.
An existing connection (on a client socket) is ready for us to send more data to the client. This is signified with a POLLOUT event on the client socket, and I'll handle it in the write_to() method by sending the next set of data for that connection.

One point to keep in mind here is that regardless of the actual size of the incoming or outgoing messages, we'll only be sending segments of 536 bytes at a time; this is the default maximum segment size for TCP/IP.

For reading data in from the socket, this is relatively easy: just read all data that's sent until we get to the end. For this server, we only need to handle HTTP GET requests, so we can assume that if we find a blank line in the data (\r\n\r\n), it is the end of the message. If we needed to handle POST requests as well, we'd need to read in the Content-Length or Transfer-Encoding: Chunked headers to determine how to know when the request was finished.
For writing data out to the socket, we need to break the response up into chunks of 536 bytes each. After sending each chunk, we'll advance a pointer to where the next chunk of data will be, then return and wait for the poller to let us know the socket is available to send more data. Once there's no more data left to write, we close the socket.

Accepting a new socket connection

Let's start with the easiest case: accepting a new connection. Here, we already know what socket to expect the connection request on, since it's the same socket we created at initialization ("server socket"). The socket.accept() method will spawn off a new socket (the "client socket") to use for this connection and return the new client socket as well as the destination address to which it's paired.

Once I have the new client socket, I set it to non-blocking and reusable (like the server socket), and then add it to the list of open streams on which poller will listen for events.

At this point, the server socket goes back to listening for new incoming connections, and the newly-created client socket will do the work of reading in the request and writing out a response.

# captive_http.py
...
import uerrno
import uselect as select
...
class HTTPServer(Server):
    ...
    def accept(self, server_sock):
        """accept a new client request socket and register it for polling"""

        try:
            client_sock, addr = server_sock.accept()
        except OSError as e:
            if e.args[0] == uerrno.EAGAIN:
                return

        client_sock.setblocking(False)
        client_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        self.poller.register(client_sock, select.POLLIN)
    ...

Reading from a client socket

Once we're spawned off a client socket for an incoming connection request, and set it up for polling, we should expect a subsequent POLLIN event (or several) where the client is sending us data that we need to read in. The way we're building this project, we can assume that any POLLIN event that doesn't belong to the DNS socket is going to be a client socket for the HTTP server, so we can just pass them all to our new read() method below.

Calling the incoming socket's read() method gives us some amount of data, but we don't know upfront whether it's the full message or a partial message. If there's no data, it's a good sign that we already hit the end of the message, and we should just close the socket now.

Otherwise, let's check if we're already partway through an incoming message for this client socket. If we are, add the new data to the existing partial message, otherwise create an empty bytestring and append the data.

Here, we're going to cheat a little bit since we only need to handle GET requests, and we can assume that any blank line signifies the end of the message. If the last four bytes of the incoming data is not \r\n\r\n, we expect there's more to come, so return early and wait for the next POLLIN event.

If we did find a blank line at the end, get the full request and print it out so we can take a look. The client will be expecting a message back, so for now let's just queue up a 404 Not Found message to get them off our backs until we can code up a way to send real HTML.

# captive_http.py
...
import uio
...
class HTTPServer(Server):
    ...
    def read(self, s):
        """read in client request from socket"""

        data = s.read()
        if not data:
            # no data in the TCP stream, so close the socket
            self.close(s)
            return

        # add new data to the full request
        sid = id(s)
        self.request[sid] = self.request.get(sid, b"") + data

        # check if additional data expected
        if data[-4:] != b"\r\n\r\n":
            # HTTP request is not finished if no blank line at the end
            # wait for next read event on this socket instead
            return

        # get the completed request
        req = self.request.pop(sid)

        print("Client raw request:\n", req)

        # send a 404 response for now
        headers = b"HTTP/1.1 404 Not Found\r\n"
        body = uio.BytesIO(b"")
        self.prepare_write(s, body, headers)
    ...

Sending to a client socket

Now all that's left to complete the HTTP transaction is to write a response back to the client. Remember that we're limited to writing 536 bytes at a time, but our response may be longer than that. In the previous section, after reading in the request, we called prepare_write() with the headers and body we wanted to send back to the client. We'll need a way to keep track of how much data we've written out for each response, and where we should start writing next time the same socket is ready to accept more of our response.

To help keep track of all that, I made a namedtuple for the writer connection that contains:

The full response body we need to send
A buffer that will contain only the next 536 (maximum) bytes we plan to send
A memoryview of the buffer to help with writing in and out of the buffer without needless copying
A range for starting and ending bytes of the buffer that we want to send next.

Notice that I'm not keeping a copy of the headers, since they'll always (in our case) be less than 536 bytes and will always go out with the first message segment.

With all that in place, I'm adding an extra newline to the headers to get a blank line (required between headers and body), and then writing the headers into the buffer (padded to make the buffer size 536 bytes). Then I use the memoryview to read from the body into the buffer, up to a maximum total of 536 bytes in the buffer. The readinto() call tells me how many bytes were actually read from body into the buffer, and I use that number to set the end point of the write_range. I save the WriteConn information into the self.conns dictionary with a key of the socket's ID so I can look it up later when I need to write the remainder of the message.

Lastly, I let the poller know that I'm prepared to write out to the socket, so it can let me know when the socket is available for outgoing data. This means I'll start getting POLLOUT events for this socket, which I'll need to handle to actually start writing this data.

# captive_http.py
...
from collections import namedtuple
WriteConn = namedtuple("WriteConn", ["body", "buff", "buffmv", "write_range"])
...
class HTTPServer(Server):
    ...
    def prepare_write(self, s, body, headers):
        # add newline to headers to signify transition to body
        headers += "\r\n"
        # TCP/IP MSS is 536 bytes, so create buffer of this size and
        # initially populate with header data
        buff = bytearray(headers + "\x00" * (536 - len(headers)))
        # use memoryview to read directly into the buffer without copying
        buffmv = memoryview(buff)
        # start reading body data into the memoryview starting after
        # the headers, and writing at most the remaining space of the buffer
        # return the number of bytes written into the memoryview from the body
        bw = body.readinto(buffmv[len(headers) :], 536 - len(headers))
        # save place for next write event
        c = WriteConn(body, buff, buffmv, [0, len(headers) + bw])
        self.conns[id(s)] = c
        # let the poller know we want to know when it's OK to write
        self.poller.modify(s, select.POLLOUT)
    ...

Now that the poller is set to pass POLLOUT events to this client socket, we need to handle those events, which will trigger a call to our new write_to() method. First, we need to get the writer connection back, which will have everything we need to determine which bytes should be written out. Remember that the write_range value of this tuple is a list of start and end positions of the buffer that we should write:

If the buffer is full, this would normally be [0, 536], and we'd write the entire contents of the buffer.
If the buffer is only partially full, the second value in the list would be something less than 536.
If we got interrupted while writing last time, the first value may be something higher than 0, and we'e start writing from that position instead of 0.

After writing out to the socket, we check how much was written. If we didn't write any bytes, or we had less than 536 bytes to write, we're done sending this response, and we can close the socket.

Otherwise, we wrote all the bytes in the buffer, so we need to keep the socket open, advance the buffer to the next bytes of the response body so that when the next POLLOUT event comes in, we'll be prepared to continue writing; this part happens in the buff_advance() method.

# captive_http.py
...
class HTTPServer(Server):
    ...
    def write_to(self, sock):
        """write the next message to an open socket"""

        # get the data that needs to be written to this socket
        c = self.conns[id(sock)]
        if c:
            # write next 536 bytes (max) into the socket
            bytes_written = sock.write(
                c.buffmv[c.write_range[0] : c.write_range[1]]
            )
            if not bytes_written or c.write_range[1] < 536:
                # either we wrote no bytes, or we wrote < TCP MSS of bytes
                # so we're done with this connection
                self.close(sock)
            else:
                # more to write, so read the next portion of the data into
                # the memoryview for the next send event
                self.buff_advance(c, bytes_written)

    def buff_advance(self, c, bytes_written):
        """advance the writer buffer for this connection to next outgoing bytes"""

        if bytes_written == c.write_range[1] - c.write_range[0]:
            # wrote all the bytes we had buffered into the memoryview
            # set next write start on the memoryview to the beginning
            c.write_range[0] = 0
            # set next write end on the memoryview to length of bytes
            # read in from remainder of the body, up to TCP MSS
            c.write_range[1] = c.body.readinto(c.buff, 536)
        else:
            # didn't read in all the bytes that were in the memoryview
            # so just set next write start to where we ended the write
            c.write_range[0] += bytes_written
    ...

Closing a client socket

There were a few cases above where we needed to close a client socket, either because we were done writing to it, or else done reading from it. Note that we never close the server socket (until we stop our program), since we want to be able to continue listening for incoming connection requests.

To cleanly close a client socket, we also need to unregister it from the poller, and delete any remaining request or connection data we had stored for it. Since we have very limited RAM on the ESP8266, we do this manually and then call the garbage collector to make sure we're conserving every single byte of RAM as early as possible to avoid memory errors.

# captive_http.py
...
import gc
...
class HTTPServer(Server):
    ...
    def close(self, s):
        """close the socket, unregister from poller, and delete connection"""

        s.close()
        self.poller.unregister(s)
        sid = id(s)
        if sid in self.request:
            del self.request[sid]
        if sid in self.conns:
            del self.conns[sid]
        gc.collect()
    ...

Adding the HTTP server to our CaptivePortal class

Now that we have the skeleton of the HTTP server set up, let's instantiate one in the CaptivePortal class and add it to the polling loop. We can test that we're actually reading in data correctly before we move on to actually correctly handling requests.

We previously wrote our handle_dns() method so that it returned False if it didn't handle the stream event, and True if it did. We'll make use of this here to make sure the HTTP server doesn't try to handle events that don't belong to it or one of its child client sockets.

# captive_portal.py
...
from captive_dns import DNSServer
from captive_http import HTTPServer

class CaptivePortal:
    ...
    def __init__(self, essid=None):
        ...
        self.dns_server = None
        self.http_server = None
        ...

    def captive_portal(self):
        print("Starting captive portal")
        self.start_access_point()

        if self.http_server is None:
            self.http_server = HTTPServer(self.poller, self.local_ip)
            print("Configured HTTP server")
        if self.dns_server is None:
            self.dns_server = DNSServer(self.poller, self.local_ip)
            print("Configured DNS server")

        try:
            while True:
                gc.collect()
                # check for socket events and handle them
                for response in self.poller.ipoll(1000):
                    sock, event, *others = response
                    is_handled = self.handle_dns(sock, event, others)
                    if not is_handled:
                        self.handle_http(sock, event, others)
        except KeyboardInterrupt:
            print("Captive portal stopped")
        self.cleanup()

    def handle_http(self, sock, event, others):
        self.http_server.handle(sock, event, others)

    ...

Time to test it out. Copy the code to the MCU and start it up. Connect to the MCU's access point with your phone, and then open a browser and try to navigate to a non-HTTPS page (http://neverssl.com, http://example.com, for example).

Entering REPL. Use Control-X to exit.
>
MicroPython v1.12 on 2019-12-20; ESP module with ESP8266
Type "help()" for more information.
>>>
>>> import main
Trying to load WiFi credentials from ./wifi.creds
./wifi.creds does not exist
Starting captive portal
Waiting for access point to turn on
#37 ets_task(4020f510, 29, 3fff8f88, 10)
AP mode configured: ('192.168.4.1', '255.255.255.0', '192.168.4.1', '192.168.4.1')
HTTP Server listening on ('0.0.0.0', 80)
Configured HTTP server
DNS Server listening on ('0.0.0.0', 53)
Configured DNS server
Sending connectivitycheck.gstatic.com. -> 192.168.4.1
- Accepting new HTTP connection
- Reading incoming HTTP data
Client raw request:
 b'GET /generate_204 HTTP/1.1\r\nUser-Agent: Dalvik/2.1.0 (Linux; U; Android 6.0.1;
 Nexus 7 Build/MOB30X)\r\nHost: connectivitycheck.gstatic.com\r\nConnection:
 Keep-Alive\r\nAccept-Encoding: gzip\r\n\r\n'
- Sending outgoing HTTP data
Captive portal stopped
Cleaning up
DNS Server stopped

This looks good. We can see that the DNS server is redirecting all domains to the MCU's IP address, and then the HTTP server is accepting the connection and reading in the request data. This particular request is my Android device checking whether it has active internet or not. iOS devices do something similar, but with a different endpoint. If you wanted to trick the device into thinking it actually had an internet connection, all you'd need to do is respond to this request with a 204 No Content response. If you're curious how this works, this article has some additional detail.

Setting up routes and serving actual responses

So far, we're responding to all requests with a 404 Not Found header and an empty body. Obviously we're going to need to send real responses to make this work. There's two things we need to get started with this:

Some actual content in the form of HTML files.
A way to examine the request, get the path requested, and map that to a specific HTML file.

Captive portal landing HTML page

Since the goal of this project is to let me enter my home WiFi credentials so the MCU can connect to it, I'll need a simple HTML page with a form on it where I can enter those details. I created a file called index.html with a basic form and a bit of inline styling. Since this is an extremely simple page on a very basic server, I'm OK with using inline CSS instead of a separate file to make serving things simpler for me.

<!-- index.html -->
<html>
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>WiFi Login</title>
    <style>
      body {
        font-family: sans-serif;
        background: #3498db;
        width: 100%;
        text-align: center;
        margin: 20px 0px 20px 0px;
      }
      p {
        font-size: 12px;
        text-decoration: none;
        color: #fff;
      }
      h1 {
        font-size: 1.5em;
        color: #525252;
      }
      .box {
        background: white;
        width: 50%;
        border-radius: 6px;
        margin: 0 auto 0 auto;
        padding: 10px 0px 10px 0px;
      }
      input[type='text'],
      input[type='password'] {
        background: #ecf0f1;
        border: #ccc 1px solid;
        border-bottom: #ccc 2px solid;
        padding: 8px;
        width: 60%;
        color: #aaa;
        margin-top: 10px;
        font-size: 1em;
        border-radius: 4px;
      }
      .btn {
        background: #2ecc71;
        width: 40ch;
        padding-top: 5px;
        padding-bottom: 5px;
        color: white;
        border-radius: 4px;
        border: #27ae60 1px solid;
        margin: 20 auto;
        font-weight: 800;
        font-size: 0.8em;
      }
    </style>
  </head>
  <body>
    <form action="/login" method="get" class="box">
      <h1>WiFi login credentials</h1>
      <input type="text" placeholder="My WiFi SSID" name="ssid" required />

      <br />

      <input
        type="password"
        placeholder="WiFi Password"
        name="password"
        required
      />

      <br />

      <button type="submit" class="btn">Connect</button>
    </form>
  </body>
</html>

Mapping files to HTTP paths

Now that I have a HTML file, I need to tell my server which HTTP paths should serve that file. Let's add a routes dictionary to the HTTPServer class. If I was making a more general-purpose HTTP server, I'd probably want to create a method for adding routes, and let the parent create the routes instead of having the server do it directly. In this case, however, I'll have a very limited number of routes that will never change, so I'm just going to create them through a dictionary literal in the HTTPServer constructor.

# captive_http.py
...
class HTTPServer(Server):
    def __init__(self, poller, local_ip):
        ...
        self.request = dict()
        self.conns = dict()
        self.routes = {b"/": b"./index.html"}
        ...

HTTP routing based on requested path

Now that we have a file created, and know what route should serve that file, we just need to examine each incoming request to see what path is being requested, and respond appropriately.

First, let's parse the HTTP request and pull out the details we may be interested in:

# captive_http.py
...
ReqInfo = namedtuple("ReqInfo", ["type", "path", "params", "host"])
...

class HTTPServer(Server):
    ...
    def parse_request(self, req):
        """parse a raw HTTP request to get items of interest"""

        req_lines = req.split(b"\r\n")
        req_type, full_path, http_ver = req_lines[0].split(b" ")
        path = full_path.split(b"?")
        base_path = path[0]
        query = path[1] if len(path) > 1 else None
        query_params = (
            {
                key: val
                for key, val in [param.split(b"=") for param in query.split(b"&")]
            }
            if query
            else {}
        )
        host = [line.split(b": ")[1] for line in req_lines if b"Host:" in line][0]

        return ReqInfo(req_type, base_path, query_params, host)

    def read(self, s):
        ...
        # get the completed request
        req = self.parse_request(self.request.pop(sid))

        # send a 404 response for now
        headers = b"HTTP/1.1 404 Not Found\r\n"
        body = uio.BytesIO(b"")
        self.prepare_write(s, body, headers)
    ...

Then, I'll create a helper function to take the parsed request and either return either the contents of the HTML file if the route matches, or else an empty byte string. Add the following to captive_html.py:

# captive_http.py
...
class HTTPServer(Server):
    ...
    def get_response(self, req):
        """generate a response body and headers, given a route"""

        headers = b"HTTP/1.1 200 OK\r\n"
        route = self.routes.get(req.path, None)

        if type(route) is bytes:
            # expect a filename, so return contents of file
            return open(route, "rb"), headers

        headers = b"HTTP/1.1 404 Not Found\r\n"
        return uio.BytesIO(b""), headers
    ...

If the route is found, I'll get a bytes string with the path to the HTML file, and I'll open that file (in binary mode) and return the file stream object.

If the route is not found, I'll get a None from the dictionary, and instead I'll return an empty bytes stream object. In both cases, I'll return appropriate headers along with the body.

Let's modify the read() function to try to get a body depending on the route, and also generate a correct header depending on whether the route was matched or not:

# captive_http.py
...
class HTTPServer(Server):
    ...
    def read(self, s):
        ...
        # get the completed request
        req = self.parse_request(self.request.pop(sid))

        body, headers = self.get_response(req)

        self.prepare_write(s, body, headers)
    ...

Time to test it out. Copy all the new/changed files over to the MCU, and fire it up.

Note: don't forget to copy your new index.html file onto the MCU as well, along with the .py files. Place the HTML files in the same folder (/pyboard) as the others.

Once the captive portal is running, connect to the MCU's WiFi access point and navigate to http://192.168.4.1/

Actually, if you navigate to any HTTP-only site, at the root path, you'll get the same page since our DNS server is telling the client that all domains point to the MCU's IP address. You can test this by navigating to any site that doesn't use HTTPS (try http://neverssl.com to test). This is definitely progress, but still not quite what we want. If (for example), I tried navigating to http://neverssl.com/online, I'd get a 404 Not Found back from the server instead of it redirecting me to the login page I want. Additionally, it kind of bugs me that the domain I tried navigating to still shows up in the browser address bar instead of the MCU's IP address.

Redirecting to the captive portal

Both of these issues can be basically solved in the same way: if the requested host or path from the client are not the ones I want to serve, I can send a redirect response back to the client to point them where I want them to go. This response will just be an empty body with a 307 Temporary Redirect header.

Let's modify the HTTPServer.read() method to check for a valid request. If the requested host matches the MCU's IP address, and the route is known, then serve the page for that route. Otherwise, return a redirect response pointing to the root path of the MCU's IP address.

# captive_http.py
...
class HTTPServer(Server):
    ...
    def read(self, s):
        ...
        # get the completed request
        req = self.parse_request(self.request.pop(sid))

        if not self.is_valid_req(req):
            headers = (
                b"HTTP/1.1 307 Temporary Redirect\r\n"
                b"Location: http://{:s}/\r\n".format(self.local_ip)
            )
            body = uio.BytesIO(b"")
            self.prepare_write(s, body, headers)
            return

        # by this point, we know the request has the correct
        # host and a valid route
        body, headers = self.get_response(req)

        self.prepare_write(s, body, headers)
    ...

Then we can write the is_valid_req() method like this:

# captive_http.py
...
class HTTPServer(Server):
    ...
    def is_valid_req(self, req):
        if req.host != self.local_ip:
            # force a redirect to the MCU's IP address
            return False
        # redirect if we don't have a route for the requested path
        return req.path in self.routes
    ...

Go ahead and test that out. Any non-HTTPS domain you try, with any path, should redirect you back to http://192.168.4.1, which matches the route for the index page with the login form.

Recap

We've made progress, but still don't have a functioning product yet. The D1 Mini has a working DNS server to point all domain requests to the local IP address, and a HTTP server that will redirect all unknown hosts and paths to the root path of the local IP address, which will now serve a form asking for the WiFI SSID and password where we want the D1 Mini to connect in the future.

All that's left is to actually parse the form submission, have the D1 Mini try to connect to the new WiFi, and then let the user know if it was successful. This is stretching into a longer writeup than I was anticipating, but I didn't want to skimp too much on the details, or make any post too long to comfortably follow.

Captive Web Portal for ESP8266 with MicroPython - Part 2

Anson VanDoren — Tue, 31 Dec 2019 12:18:36 +0000

In Part 1 of this series, I had set up the beginnings of a "captive portal" DNS server. The intent of this server is to redirect all DNS requests to point to the IP address of the Wemos D1 Mini MCU that's running the WiFi access point. See the intro to that post for why I want to do this.

I'd progressed to the point where I had a working socket that could display the raw datagram coming in over port 53 for DNS queries when I connected my phone to the MCU's AP. Now I need to understand how to interpret those requests, and how to construct a DNS answer I can send back to the client that will redirect them to the IP address I want.

Understanding DNS requests

I don't want to go into excruciating detail here about how DNS messages are constructed, but it is pretty important to understand in order to make this work. I relied pretty heavily on a great article by James Routley that you can find here. I'd suggest reading it for the details, and then come back to see how I implemented it for my captive portal. The Wikipedia article also has some good amplifying information that was pretty useful. If you just can't get enough of DNS nitty-gritty, you could also try reading RFC 2929 which is the internet standard for DNS messages.

DNS request example

Here's an example of the request my DNS server logged as incoming from my phone when it first connected:

b'\x99\xa5\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03ssl\x07gstatic\x03com\x00\x00\x01\x00\x01'

Python is showing me the message in byte format (b''), which converts to ASCII where it can, and displays the raw hexadecimal value of each byte where it can't.

Request Header

The message header is 12 bytes long; in this example the header is:

\x99\xa5\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00

The first two bytes are the request ID generated by the client. As the server, I don't really care what this ID is, except that I need to remember it and include it in the response back to the client.
- In this example, the ID is \x99\xa5
The second two bytes (16 bits) are the Flags section:
- In this example, the Flags are \x01\x00, or in binary 0000000100000000
- See Wikipedia article linked above for a full explanation of what each bit means.
- In this case, the binary representation of these two bytes (in big-endian ("network byte") format) means all header flags are set to 0 except for the RD bit, or "Recursion Desired".
- Cloudflare has a good article explaining recursive vs. iterative DNS requests if you're interested. For our part, we don't really care about this since we're only redirecting all DNS requests to our captive portal anyway.
The rest of the header (\x00\x01\x00\x00\x00\x00\x00\x00) gives the number of questions, answers, authority records, and additional records in the message. I'll ignore them for my purposes, and assume the client is only sending one question (which is true in this case).

Request Question

The question section has a variable length for the first part (QNAME) and a fixed two bytes for each of the remaining two parts (QTYPE and QCLASS). All I really care about for this project is the QNAME, which is the domain the client wants an IP address for, as I'll need that to construct the answer response. This section contains one or more "labels" where the first byte tells how many bytes the full label is, and the rest of the label is part of the domain being queried.

In this example:

First byte tells how long the upcoming label is (\x03 for the first label)
The next 3 bytes (ssl for the first label) is the label name. Since these bytes are valid ASCII characters, Python has converted it for me automatically.
After each label in this section, there is an implicit ., which I'll need to add.
Repeat until label length is \x00, which marks end of the QNAME section

This QNAME fully resolves to ssl.gstatic.com, after which there is a zero-byte (\x00) telling me the QNAME section is done.

The next byte pair is QTYPE, which in this case is \x00\x01 for an A record. This Wikipedia article lists other possible options for record types.
The final byte pair is QCLASS, which in this case is \x00\x01 for internet (IN).

Parsing a DNS question

Let's create a new class in our captive_dns.py file to handle DNS queries:

# captive_dns.py
...
class DNSQuery:
    def __init__(self, data):
        self.data = data
        self.domain = ""
        # header is bytes 0-11, so question starts on byte 12
        head = 12
        # length of this label defined in first byte
        length = data[head]
        while length != 0:
            label = head + 1
            # add the label to the requested domain and insert a dot after
            self.domain += data[label : label + length].decode("utf-8") + "."
            # check if there is another label after this one
            head += length + 1
            length = data[head]
...

When we create a new DNSQuery, we'll pass in the raw datagram we got from the socket and store it. What we really are trying to get at to start with is the requested domain, so we don't need to parse the header information yet (although we will need to for the response).

To get the domain out of the message, we start on byte 12, read the length of that label, and then read that many bytes into the domain, appending a . to each label. When we get to a length-byte of 0, we know we have the full requested domain.

Now let's update our DNSServer class to parse the incoming requests:

# captive_dns.py
...
class DNSServer(Server):
    ...
    def handle(self, sock, event, others):
        # server doesn't spawn other sockets, so only respond to its own socket
        if sock is not self.sock:
            return

        # check the DNS question, and respond with an answer
        try:
            data, sender = sock.recvfrom(1024)
            request = DNSQuery(data)
            print("Client requested IP address for domain:", request.domain)

            # help MicroPython with memory management
            del request
            gc.collect()
        except Exception as e:
            print("DNS server exception:", e)

If I run this now, and reconnect my phone to the MCU access point, I'll see something like this with the resolved domain name questions:

MicroPython v1.12 on 2019-12-20; ESP module with ESP8266
Type "help()" for more information.
>>>
>>> import main
Trying to load WiFi credentials from ./wifi.creds
./wifi.creds does not exist
Starting captive portal
Waiting for access point to turn on
#57 ets_task(4020f510, 29, 3fff8f88, 10)
AP mode configured: ('192.168.4.1', '255.255.255.0', '192.168.4.1', '192.168.4.1')
DNS Server listening on ('0.0.0.0', 53)
Configured DNS server
Client requested IP address for domain: connectivitycheck.gstatic.com.
Client requested IP address for domain: encrypted-tbn0.gstatic.com.
Client requested IP address for domain: www.google.com.
Client requested IP address for domain: android.googleapis.com.
Client requested IP address for domain: zowkbggnyjedl.
Client requested IP address for domain: uxfwdospee.
Client requested IP address for domain: dwcccklrboesuuf.
Client requested IP address for domain: encrypted-tbn0.gstatic.com.
Client requested IP address for domain: www.google.com.
Client requested IP address for domain: update.googleapis.com.
Client requested IP address for domain: inbox.google.com.
Client requested IP address for domain: encrypted-tbn0.gstatic.com.
Client requested IP address for domain: www.google.com.
Client requested IP address for domain: connectivitycheck.gstatic.com.
Client requested IP address for domain: android.googleapis.com.
Client requested IP address for domain: mtalk.google.com.
Client requested IP address for domain: zowkbggnyjedl.
Client requested IP address for domain: uxfwdospee.
Client requested IP address for domain: dwcccklrboesuuf.

Notice groups of 3 "garbage" domain requests interspersed between the legitimate ones. It took me quite a while to figure out some google terms that would lead me to an answer as to what those were (winning answer was "gobbledygook dns queries"), but it turns out those are Chrome trying to prevent DNS hijacking by an ISP. There's a pretty good StackExchange answer if you're curious for more details. In our case it is pretty irrelevant since we aren't trying to actually correct DNS results, and we don't care that Chrome/Chromium knows about it.

Understanding DNS responses

Now that we know what domain the client is asking for, it's time to generate a response. Since the point of the captive portal is to redirect all DNS requests back to the MCU's IP address, we don't care about what the correct answer is. This makes it a lot simpler to finish out this section of the project.

If you need to, refer back to the links at the beginning of the post to see the details about the response format, since I'll just gloss over it here. The response contains all the same sections as the request (Header, Question), but it will also contain an Answer section that we'll need to construct here. The Question will be identical, but we'll also need to modify the Header section a bit.

Response Header

Again, we're not trying to construct a necessarily correct response, just a valid one so the client can process it. What we need in the header section is:

Set first two bytes to the ID that came with the request, so the client knows which request this response belongs to.
Cheat a little, and set the flags section without actual regard for request. For every answer, we'll set the flags to \x81\x80, which translates to 1000000110000000 in binary and means:
- QR=1 (this is a response, not a query)
- Opcode=0 (this is a response to a standard query)
- TC=0 (this response is not truncated)
- RD=1 (makes assumption client also set this bit on for a recursive lookup, which is normal)
- RA=1 (telling client recursive lookup is available)
Copy the question count from request and set answer count to the same
Set authority records and additional records to 0

Response Question and Answer

The body of our response will repeat back the question that was asked and then add an answer on the end. We already have the question stored in our DNSQuery class (raw data starting on byte 12 and going to the end), so we can copy that over easily.

To construct the answer, we need a few parts:

A pointer back to the requested domain name. This pointer is two bytes long, with the first two bits being 1s, and the remaining 14 bits being an unsigned integer that specifies the number of bytes from the beginning of the message where the prior occurrence of the name can be found. In our case, that's byte 12.
The next two byte pairs are TYPE and CLASS (similar to QTYPE and QCLASS in the question section). We'll set these for an A record and IN class, regardless of the question, since that's all our simple server knows how to handle (and also by far the most common request we'll get).
The next byte pair is TTL in seconds, as a 32-bit number. This tells the client how long this response is valid for, and I chose 60sec, or \x00\x00\x00\x3C.
Then comes the length (in bytes) of the response body. Since I'm returning an IPv4 address, this fits into 4 bytes, so I used \x00\x04.
Finally, I break down the IP address into 4 bytes and send them as the final bytes in the packet.

Using the example from the request section above, here's what my response looks like, broken down:

Header: \x99\xa5\x81\x80\x00\x01\x00\x01\x00\x00\x00\x00

Question: \x03ssl\x07gstatic\x03com\x00

Answer: \x00\x01\x00\x01\xc0\x0c\x00\x01\x00\x01\x00<\x00\x04\xc0\xa8\x04\x01

Note the odd-looking behavior in our TTL octets: we actually sent \x00\x00\x00\x3C, but when printed to the screen it shows as \x00\x00\x00< instead. This is because Python automatically tries to convert ASCII characters it knows about into the ASCII equivalent. \x3c is decimal 60, which is ASCII < character. This is perfectly normal, and the response is still correct in terms of bytes; it took me a while to figure out that this is just a Python internal representation when I was looking at the output of my program here.

Creating a DNS response

Now that we know the format of a response, let's update our DNSQuery class to generate one when asked:

# captive_dns.py
...
class DNSQuery:
    ...
    def answer(self, ip_addr):
        # ** create the answer header **
        # copy the ID from incoming request
        packet = self.data[:2]
        # set response flags (assume RD=1 from request)
        packet += b"\x81\x80"
        # copy over QDCOUNT and set ANCOUNT equal
        packet += self.data[4:6] + self.data[4:6]
        # set NSCOUNT and ARCOUNT to 0
        packet += b"\x00\x00\x00\x00"

        # ** create the answer body **
        # respond with original domain name question
        packet += self.data[12:]
        # pointer back to domain name (at byte 12)
        packet += b"\xC0\x0C"
        # set TYPE and CLASS (A record and IN class)
        packet += b"\x00\x01\x00\x01"
        # set TTL to 60sec
        packet += b"\x00\x00\x00\x3C"
        # set response length to 4 bytes (to hold one IPv4 address)
        packet += b"\x00\x04"
        # now actually send the IP address as 4 bytes (without the "."s)
        packet += bytes(map(int, ip_addr.split(".")))

        gc.collect()

        return packet
...

Here, I'm passing in an IP address (the MCU's address in this case), and getting back a (hopefully) valid DNS answer pointing all domains to that IP.

With that in place, it's trivial to have the DNS server send back the answer to each question. Make the following addition to the DNSServer class:

# captive_dns.py
class DNSServer(Server):
    ...
    def handle(self, sock, event, others):
        # server doesn't spawn other sockets, so only respond to its own socket
        if sock is not self.sock:
            return

        # check the DNS question, and respond with an answer
        try:
            data, sender = sock.recvfrom(1024)
            request = DNSQuery(data)

            print("Sending {:s} -> {:s}".format(request.domain, self.ip_addr))
            sock.sendto(request.answer(self.ip_addr), sender)

            # help MicroPython with memory managament
            del request
            gc.collect()
        except Exception as e:
            print("DNS server exception:", e)

Fire it up and test everything out:

Entering REPL. Use Control-X to exit.
>
MicroPython v1.12 on 2019-12-20; ESP module with ESP8266
Type "help()" for more information.
>>>
>>> import main
Trying to load WiFi credentials from ./wifi.creds
./wifi.creds does not exist
Starting captive portal
Waiting for access point to turn on
#73 ets_task(4020f510, 29, 3fff8f88, 10)
AP mode configured: ('192.168.4.1', '255.255.255.0', '192.168.4.1', '192.168.4.1')
DNS Server listening on ('0.0.0.0', 53)
Configured DNS server
Sending connectivitycheck.gstatic.com. -> 192.168.4.1
Sending connectivitycheck.gstatic.com. -> 192.168.4.1
Sending mtalk.google.com. -> 192.168.4.1
Sending alt5-mtalk.google.com. -> 192.168.4.1
Sending android.googleapis.com. -> 192.168.4.1

The output looks like things are working, but we don't have a real way to test right now. Next up, we'll write a HTTP server to see that the client is actually getting redirected where we want.

Recap

We can now parse a DNS request to figure out what domain name the client is asking about, and then create a (fake) response that will direct them to the D1 Mini's IP address. In the next section, we'll create a HTTP server on the MCU to make use of the HTTP requests that are currently being routed there by our DNS server.

Captive Web Portal for ESP8266 with MicroPython - Part 1

Anson VanDoren — Mon, 30 Dec 2019 21:09:19 +0000

Project idea

I've worked on a few small projects recently using an ESP8266 SoC, and one thing that sort of bothered me is that I needed to hard-code my home WiFi access point SSID and password into either the Python file or a separate config file. It's not really that big of a deal for small personal projects, but when I buy "smart" gadgets from anywhere else, they always come with the option to set up the device by connecting to a temporary WiFi access point on the device itself, and then typing my home WiFi SSID and password into a HTML form.

This aim of this project is to program the ESP8266 MCU to:

On startup, first check if it already knows how to log into my home WiFi. Once I set up a device, I'd expect it to remember the WiFi credentials I told it even if it was rebooted.
If it doesn't have any known credentials, or the previously saved credentials don't work, then start an unsecured WiFi access point on bootup instead.
With my phone, connect to the MCU WiFi AP, and then enter the SSID and password for my actual home WiFI.
The MCU then tries to connect to my home WiFi using the newly provided credentials, and display a basic status page showing what network it connected to, and what is its IP address on my home network.

This idea isn't really new, and it's referred to as a "captive portal" in other contexts. It's something you may have seen when logging into public WiFi at your favorite coffee shop, where you're redirected to a sign-in page before being allowed online. There's a few different ways to accomplish that, but the way I'm going to do it will require a HTTP server (to serve some HTML to ask for WiFi SSID and password), and also a DNS server to redirect all DNS questions to the IP address of the MCU so that I don't need to know the board's IP address. I'll be able to just connect to the board's access point, and then navigate to any (non-HTTPS) website, and it will redirect me to the login page I want.

The non-HTTPS part is important; for the sake of simplicity, the HTTP server is only listening on port 80, and not 443. HTTPS requests will still get redirected to the board, but there isn't a HTTPS socket listening for them there, so they'll time out. It's possible to add HTTPS redirection like this, but there's a few problems:

Many browsers will display an warning, or refuse to show the page, if a HTTPS site is redirected. This is obviously good for security, but makes it a pain for this particular use-case.
I would need to set up SSL certificates, start a third socket server on the MCU, and be able to handle TLS/SSL connections, which is more involved than plain HTTP. I think the ESP8266 is probably capable of this, but it's a tiny device with limited resources available, and this would be way overkill for what I actually need.

For my Android phone (and I think for iOS devices as well), the OS detects that it doesn't have internet access and will automatically ask me to sign in anyway, which goes over HTTP and does the redirection I need. If this doesn't work, I could just intentionally navigate to a site that I know uses HTTP only (like http://example.com or http://neverssl.com) to trigger the redirection.

Basic hardware setup

I wrote a previous article on setting up a NodeMCU ESP8266 with MicroPython a few months ago, so instead of repeating all the instructions here you can check out that article instead. I'll assume that you can already rshell into your MCU for the rest of this post.

The only difference from last time is that I'm using a different board this time, a Wemos D1 Mini I ordered from Amazon. It has the same ESP8266 chipset and mostly the same features, but fewer GPIO pins than the other one. The price is about the same either way, but I was looking for a smaller form factor for my next project, and this is about half the size of the NodeMCU development board.

The one additional step I did need to get the D1 Mini to work was to install an additional driver. All of my search results for this ended up with kind of sketchy-looking websites, but I think the "official" driver from the manufacturer should be safe. The link above is for the macOS version of the driver, but there's also drivers for Windows and Linux on the same site.

Aside from this, everything was basically the same except that the Wemos D1 Mini shows up on a different serial port, in my case /dev/cu.wchusbserial1430. As a reminder, you can check this yourself using the esptool.py tool from Espressif:

$ esptool.py read_mac
esptool.py v2.8
Found 3 serial ports
Serial port /dev/cu.wchusbserial1430
Connecting....
Detecting chip type... ESP8266
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: 2c:f4:32:78:d0:c1
Uploading stub...
Running stub...
Stub running...
MAC: 2c:f4:32:78:d0:c1
Hard resetting via RTS pin...

ESP8266 filesystem

Connecting to the board with rshell after initially flashing the latest MicroPython binary from here (I used v1.12) shows a single file (boot.py) in the /pyboard folder.

I didn't touch the boot.py file, and mine looked like this:

# /pyboard/boot.py
# This file is executed on every boot (including wake-boot from deepsleep)
#import esp
#esp.osdebug(None)
import uos, machine
#uos.dupterm(None, 1) # disable REPL on UART(0)
import gc
#import webrepl
#webrepl.start()
gc.collect()

The boot.py file is the first file run each time the board resets. It's not doing much in my case other than a garbage collection, but I'll leave it as is.

The other file that's run on each boot is called main.py, but it doesn't exist yet. Since this project is intended to just be the bootstrap code for a new device, I don't want to clutter up main.py with the captive portal code. I'll write most of this code in separate files, and just import and call it from a couple of lines in main.py.

One other important fact I learned on this project is that since MicroPython needs to import and "compile" (into frozen bytecode) each file in one chunk, the raw Python file sizes need to be limited to what's available in RAM on the MCU, minus what the interpreter is using up. In practice, I found this meant that I couldn't import a source file much larger than ~8Kb. In some cases (for this and other projects in MicroPython), I've needed to run garbage collection between imports as well. The documentation talks a bit about it if you want more details. If you get (cryptic, seemingly incomplete) errors like the example below, it may mean your file size is too large, and you should break it up into multiple files:

>>> import captive_http
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError:

>>>

Getting started

It's possible to use rshell to edit files directly (sort of) on the MCU, but I prefer to edit from my host machine and then copy the files in from there during testing. This way I keep the latest updates inside my git repo on my desktop and never need to worry about which is the latest version. My workflow looks something like this:

Edit files (e.g., main.py) on my host PC
rshell to MCU, then cp main.py /pyboard/
On the MCU, start the repl and then import main

To start this project off, I'm going to create two files:

main.py, which will kick off the code I want to run and otherwise be available for future project code.
captive_portal.py, which will coordinate the HTTP and DNS servers to make this WiFi bootstrapping code work.

I'll start with main.py since it's only a couple of lines:

# main.py
from captive_portal import CaptivePortal

portal = CaptivePortal()

portal.start()

As you can see, not much going on here other than importing my CaptivePortal class and running its start() function. I'll write that class and method in a new file like this:

# captive_portal.py
import network
import uerrno
import uos as os
import utime as time

class CaptivePortal:
    CRED_FILE = "./wifi.creds"
    MAX_CONN_ATTEMPTS = 10

    def __init__(self):
        self.sta_if = network.WLAN(network.STA_IF)

        self.ssid = None
        self.password = None

    def start(self):
        # turn off station interface to force a reconnect
        self.sta_if.active(False)
        if not self.try_connect_from_file():
            self.captive_portal()

    def connect_to_wifi(self):
        print(
            "Trying to connect to SSID '{:s}' with password {:s}".format(
                self.ssid, self.password
            )
        )
        # initiate the connection
        self.sta_if.active(True)
        self.sta_if.connect(self.ssid, self.password)

        attempts = 0
        while attempts < self.MAX_CONN_ATTEMPTS:
            if not self.sta_if.isconnected():
                print("Connection in progress")
                time.sleep(2)
                attempts += 1
            else:
                print("Connected to {:s}".format(self.ssid))
                self.local_ip = self.sta_if.ifconfig()[0]
                self.write_creds(self.ssid, self.password)
                return True

        print("Failed to connect to {:s} with {:s}. WLAN status={:d}".format(
            self.ssid, self.password, self.sta_if.status()
        ))
        # forget the credentials since they didn't work, and turn off station mode
        self.ssid = self.password = None
        self.sta_if.active(False)
        return False

    def write_creds(self, ssid, password):
        open(self.CRED_FILE, 'wb').write(b','.join([ssid, password]))
        print("Wrote credentials to {:s}".format(self.CRED_FILE))

    def captive_portal(self):
        print("Starting captive portal")

    def try_connect_from_file(self):
        print("Trying to load WiFi credentials from {:s}".format(self.CRED_FILE))
        try:
            os.stat(self.CRED_FILE)
        except OSError as e:
            if e.args[0] == uerrno.ENOENT:
                print("{:s} does not exist".format(self.CRED_FILE))
                return False

        contents = open(self.CRED_FILE, 'rb').read().split(b',')
        if len(contents) == 2:
            self.ssid, self.password = contents
        else:
            print("Invalid credentials file:", contents)
            return False

        if not self.connect_to_wifi():
            print("Connect with saved credentials failed, starting captive portal")
            os.remove(self.CRED_FILE)
            return False

        return True

There's a few things that may look strange here if you're coming from CPython ("normal" Python) and haven't used MicroPython before. MicroPython is fairly full-featured, but it is still only a subset of CPython, and as such some of the standard library is missing. In most of these cases, I'm importing the "micro" version of the library instead (like uos instead of os). The PyCopy docs are really excellent when trying to figure out similarities and differences between the two Python versions and has been a huge help throughout this project.

Note: PyCopy is a fork of MicroPython, but the documentation is much better written and more complete. Generally they are feature equivalent, so I tend to use the docs from PyCopy instead.

The class __init__() method sets up variables for eventual SSID and password, and also a reference to the MCU's station interface. The station interface is for the MCU to connect to another WiFi hotspot. Later on, we'll also configure the MCU's access point interface.

For now, the start() method is just trying to connect to my home WiFi point from previously-saved credentials if it can, and if not, it will (eventually) start the captive portal itself.

The try_connect_from_file() method is fairly straightforward:

Check if the file exists where we expect WiFi credentials
If it does, open it and check that it has two comma separated values
If it does, try to connect with those values assuming the first is my home WiFi SSID, and the second is the WiFi password
If any step fails, return False so that we can start up the captive portal instead to prompt the user to enter these credentials.

connect_to_wifi() turns on the station interface and tries to connect with the credentials we have (which don't exist yet, since we didn't read them from the file and haven't prompted the user). It waits up to 20 seconds for the connection to be established before bailing out. If it fails to connect, it will print the interface status, which will be one of the constants listed in the MicroPython network documentation.

Copy both of these files into the /pyboard/ folder on the MCU, then run repl and import main:

/pyboard> repl
Entering REPL. Use Control-X to exit.
>
MicroPython v1.12 on 2019-12-20; ESP module with ESP8266
Type "help()" for more information.
>>>
>>> import main
Trying to load WiFi credentials from ./wifi.creds
./wifi.creds does not exist
Starting captive portal

Base server class

Since I'll need both a DNS and HTTP server, it makes sense to extract out any common parts into a superclass. The basic functionality of this class will be to create a socket on a specified port and then register it with a stream poller that will notify the server when a new event occurs on the socket. MicroPython has the uselect module to help deal with streams like this, which is a subset of CPython select module. Using this poller, we can run both the HTTP and DNS servers at the same time without either one blocking waiting to listen to its socket.

Here's the Server class I created in a new file:

# server.py
import usocket as socket
import uselect as select


class Server:
    def __init__(self, poller, port, sock_type, name):
        self.name = name
        # create socket with correct type: stream (TCP) or datagram (UDP)
        self.sock = socket.socket(socket.AF_INET, sock_type)

        # register to get event updates for this socket
        self.poller = poller
        self.poller.register(self.sock, select.POLLIN)

        addr = socket.getaddrinfo("0.0.0.0", port)[0][-1]
        # allow new requests while still sending last response
        self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        self.sock.bind(addr)

        print(self.name, "listening on", addr)

    def stop(self, poller):
        poller.unregister(self.sock)
        self.sock.close()
        print(self.name, "stopped")

The HTTP server will use a TCP socket on port 80, while the DNS server will use a UDP socket on port 53. Once the socket is created, I'll register it with a poller passed in from the caller (so I can reuse the same poller) and sign up for the POLLIN event, which will notify whenever the socket has new incoming data to read. Even though I didn't register for them in the event bitmask, I'll also potentially get POLLHUP (hangup) and POLLERR (error) events.

Next I set the socket options to reuse its address. This is mostly only important for the TCP socket since there may still be outgoing data for a recently-closed socket while I want to send new data out on a new connection. For a better in-depth discussion on the SO_REUSEADDR flag, here's a great StackOverflow answer.

Finally, I bind the new socket to the shortcut address "0.0.0.0" which will listen for all IPv4 addresses on the MCU and the given port. The MicroPython documentation for usocket is helpful in getting this setup correctly.

Other than the constructor, the only method I pulled up into this superclass is one to gracefully halt the server when I'm done by unregistering it from the poller and closing the socket.

DNS server class

Now that I have the skeleton in place, I'll create a DNS server subclass in a new file called captive_dns.py:

# captive_dns.py
import usocket as socket
import gc

from server import Server

class DNSServer(Server):
    def __init__(self, poller, ip_addr):
        super().__init__(poller, 53, socket.SOCK_DGRAM, "DNS Server")
        self.ip_addr = ip_addr

    def handle(self, sock, event, others):
        # server doesn't spawn other sockets, so only respond to its own socket
        if sock is not self.sock:
            return

        # check the DNS question, and respond with an answer
        try:
            data, sender = sock.recvfrom(1024)
            print("Got data:", data, "from sender:", sender)
        except Exception as e:
            print("DNS server exception:", e)

This DNS server doesn't really do much yet. Whenever its handle() method is called with a socket, event type, and possibly additional data, it will check to make sure the socket is what it expects, and then try to read in 1024 bytes of data from the socket. Obviously we'll add more here soon, but this should be enough to test the basic functionality.

Setting up the access point

Now, back in the CaptivePortal class, I'll instantiate a DNS server and set up an event poller loop to start listening to the socket stream. Before we can do that, though, I need to set up the MCU to turn on its access point interface so it can accept incoming WiFi connections:

# captive_portal.py
import network
import ubinascii as binascii
...

class CaptivePortal:
    AP_IP = "192.168.4.1"
    CRED_FILE = "./wifi.creds"
    MAX_CONN_ATTEMPTS = 10

    def __init__(self, essid=None):
        self.local_ip = self.AP_IP
        self.sta_if = network.WLAN(network.STA_IF)
        self.ap_if = network.WLAN(network.AP_IF)

        if essid is None:
            essid = b"ESP8266-%s" % binascii.hexlify(self.ap_if.config("mac")[-3:])
        self.essid = essid

        self.ssid = None
        self.password = None
    ...

Here, I'm setting up a local IP address that I'll use for the access point (which happens to be the default ESP8266 address, but doesn't need to be), and then getting access to the Access Point interface. I also added a constructor parameter for the access point SSID in case I wanted to name it differently. If I don't pass in a special name, it will use part of the MCU's MAC address to create a unique SSID for the access point.

Next, I'll create a method to actually turn on and configure the access point interface, and then call that at the start of the captive_portal() method:

# captive_portal.py
...
class CaptivePortal:
    ...
    def start_access_point(self):
        # sometimes need to turn off AP before it will come up properly
        self.ap_if.active(False)
        while not self.ap_if.active():
            print("Waiting for access point to turn on")
            self.ap_if.active(True)
            time.sleep(1)
        # IP address, netmask, gateway, DNS
        self.ap_if.ifconfig(
            (self.local_ip, "255.255.255.0", self.local_ip, self.local_ip)
        )
        self.ap_if.config(essid=self.essid, authmode=network.AUTH_OPEN)
        print("AP mode configured:", self.ap_if.ifconfig())

    def captive_portal(self):
        print("Starting captive portal")
        self.start_access_point()
    ...

I ran into issues occasionally where if the AP interface was already turned on when I tried to reconfigure it, it would throw an error or else just never configure itself properly. To get around that, I explicitly restart it every time I want it on, and wait for it to report it's active before proceeding.

The ifconfig() function call sets (in order):

IP address
Netmask
Gateway
DNS server

The values I'm using for IP address, netmask, and gateway are all the default for the interface, but I'm changing the DNS server to point to the MCU itself, instead of a "real" DNS service so that we can have the MCU respond to all DNS queries for devices that connect to it in order to redirect them how we want.

Lastly, I set the network SSID (the name I'll see when I try to connect from my phone), and set it to an open access point so I don't need to enter a password to connect. Then it's just a matter of calling this function at the beginning of my captive_portal() method to kick things off.

To test everything out so far, I copied my new/changed files onto the MCU and then imported main from the REPL:

MicroPython v1.12 on 2019-12-20; ESP module with ESP8266
Type "help()" for more information.
>>>
>>> import main
Trying to load WiFi credentials from ./wifi.creds
./wifi.creds does not exist
Starting captive portal
Waiting for access point to turn on
#33 ets_task(4020f510, 29, 3fff8f88, 10)
#34 ets_task(4020f510, 29, 3fff8f88, 10)
AP mode configured: ('192.168.4.1', '255.255.255.0', '192.168.4.1', '192.168.4.1')

Looks good here. On my phone, I searched for available WiFi access points, and there it is!

Polling for socket events

Now that I have an access point that my phone can connect to, it's time to start actually polling with the DNS server. Remember that the access point configuration is telling all its client that it is the DNS server, so when my phone connects and tries to look up a domain name, it will be asking the MCU itself what IP address that URL points to.

Make the following additions to the CaptivePortal class:

# captive_portal.py
...
import gc
import uselect as select

from captive_dns import DNSServer

class CaptivePortal:
    AP_IP = "192.168.4.1"
    CRED_FILE = "./wifi.creds"
    MAX_CONN_ATTEMPTS = 10

    def __init__(self, essid=None):
        ...
        self.dns_server = None
        self.poller = select.poll()
        ...

    def captive_portal(self):
        print("Starting captive portal")
        self.start_access_point()

        if self.dns_server is None:
            self.dns_server = DNSServer(self.poller, self.local_ip)
            print("Configured DNS server")

        try:
            while True:
                gc.collect()
                # check for socket events and handle them
                for response in self.poller.ipoll(1000):
                    sock, event, *others = response
                    self.handle_dns(sock, event, others)
        except KeyboardInterrupt:
            print("Captive portal stopped")
        self.cleanup()

    def handle_dns(self, sock, event, others):
        if sock is self.dns_server.sock:
            # ignore UDP socket hangups
            if event == select.POLLHUP:
                return True
            self.dns_server.handle(sock, event, others)
            return True
        return False

    def cleanup(self):
        print("Cleaning up")
        if self.dns_server:
            self.dns_server.stop(self.poller)
        gc.collect()
    ...

Here I'm using the uselect module to create a Poll object. When I instantiate the DNSServer class, it registers its socket with the poller, and any events on that socket stream will show up as a response that I can iterate through. I'm calling the ipoll method with a timeout of 1000msec, and if there's any event, I check whether the DNSServer wants to handle it.

Since UDP is a connectionless protocol (and possibly due to a bug in the ESP8266 port of MicroPython), I found I was getting dozens of POLLHUP (stream hang-up) events per second after the first, which are useless to me in this application. I ignore those for now, and otherwise send the event to the DNSServer to handle itself.

To test it all out, I saved, copied, and ran from REPL again. After I connected my phone to the access point, I can see some DNS requests coming in. After confirming it worked, I used Ctrl-C to exit, and verified the cleanup code was also called.

MicroPython v1.12 on 2019-12-20; ESP module with ESP8266
Type "help()" for more information.
>>>
>>> import main
Trying to load WiFi credentials from ./wifi.creds
./wifi.creds does not exist
Starting captive portal
Waiting for access point to turn on
#50 ets_task(4020f510, 29, 3fff9428, 10)
AP mode configured: ('192.168.4.1', '255.255.255.0', '192.168.4.1', '192.168.4.1')
DNS Server listening on ('0.0.0.0', 53)
Configured DNS server
Got data: b'\xa7A\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03www\x06google\x03com
\x00\x00\x01\x00\x01' from sender: ('192.168.4.2', 42434)
Got data: b'Eo\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x11connectivitycheck\x07gs
tatic\x03com\x00\x00\x01\x00\x01' from sender: ('192.168.4.2', 13197)
Got data: b'\xa7A\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03www\x06google\x03com
\x00\x00\x01\x00\x01' from sender: ('192.168.4.2', 42434)
Got data: b'Eo\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x11connectivitycheck\x07gs
tatic\x03com\x00\x00\x01\x00\x01' from sender: ('192.168.4.2', 13197)
Captive portal stopped
Cleaning up
DNS Server stopped

Recap

So far, I've accomplished the following:

Configured the Wemos D1 Mini MCU with MicroPython firmware and established a serial connection.
Made the MCU try to connect to an already-known WiFi point if it has one, on bootup.
Written a base server class that can bind to a socket and register to receive events on that socket stream.
Written the skeleton of a DNS server that listens to UDP datagrams on port 53 and displays them on screen.
Set up the basic event loop of the captive server to poll for events and handle them appropriately.

In the next part, I'll tackle how to understand DNS questions and generate an answer that will point all DNS queries back to the MCU's IP address so that the user will always be presented with a login page when they connect to the MCU's WiFi access point.

Note: if you purchase an item from Amazon using any of the links in this article, I may receive a small commission.

HMAC signatures for Insomnia requests

Anson VanDoren — Fri, 27 Dec 2019 21:44:08 +0000

I've been working with the Binance API recently, and found I was spending too much time making manual curl or requests calls just to see what sort of data I was getting back in order to appropriately process it. It never occurred to me that there might be a better way until I stumbled across the Insomnia REST client last night.

Working with APIs with Insomnia

If you haven't worked with Insomnia before, but you do spend a decent amount of time either producing or consuming APIs, go check it out now. If you've used Postman, it's probably not going to blow your socks off, but if you've ever found yourself trying to remember all the curl options just to make a simple request and check out the response, you'll probably love this.

A few minutes after installing it, I had this set up for my Binance work:

I added a few environment variables that work across all calls (base API URL, API key, API secret, etc.), and then created new endpoints for each of the API calls I need to refer to. I won't go into much more detail here since it's pretty easy to set up and use, but if this looks interesting for your work, definately go check it out!

Signed requests

This was all well and good until I got the part that I really wanted to do, which was executing API operations like listing account balances and placing trades. Binance uses HMAC signing to confirm sensitive requests come from authorized individuals only. The Binance API docs have a reasonable amount of information on how to do this, but in my actual code I'm using the python-binance library to take care of it so I hadn't looked into it much.

Insomnia comes with a few auth options out-of-the-box, but none of them worked for my case. There are a handful of Insomnia plugins available on the NPM registry, but none that worked for what I needed.

Insomnia plugins

That brings me to the real reason for writing this post, which is so that future-me will remember how I solved this and not re-invent the wheel when I run across this again down the road. The Insomnia docs have a page on writing plugins, but it's not as well documented as it could be.

Basically I had two choices:

A template tag, which I could reference like an environment variable inside of the client
A request/response hook that is triggered either just before the request is sent out, or upon receiving a response to a previous request.

My first thought was to write a template tag that I could just put into the signature query parameter which would look at the other parameters, compute the HMAC, and write it out before sending. I would still like to implement it this way, but I ran into a problem relating to how the timestamp tag (an Insomnia built-in) was updating after I computed the hash but before sending the request, which rendered the signature invalid before it was sent off.

Since this didn't seem to be an easy option, I chose instead to write a request hook that looks at all requests, checks whether they are going to Binance, and if so, whether they need to be signed. In the particular case of Binance, I chose to make this second part trigger off whether there was a timestamp parameter already included in the query. All of the Binance REST endpoints that need to be signed also require a timestamp, and the other endpoints do not accept one.

The basic structure of my request hook looks like this:

// insomnia_plugin_binance_signing.js

module.exports.requestHooks = [
    (context) => {
        // Validate context object
        // Check the URL points to the Binance API
        // Check if a timestamp parameter exists
        // Make sure there is not already a signature parameter
        // Check the API Key is present in the environment variables
        // Set a time window to prevent time-sensitive request from executing late
        // Compose the query string from the request
        // Generate the signature
        // Set the signature
    }
]

The first couple of validations are boring, so I won't include them here, but the whole thing is on GitHub if you're curious. Basically I'm just ensuring the context object exists, it has a request property, and that request property has a getUrl() method. If any check fails, just return early and do nothing.

Below is the basic implementation, skipping redundant parts. Again, check out the full code if you want more detail.

        // Validate context object
        // ... boring stuff...

        const req = context.request;

        // Check the URL points to the Binance API
        if (!req.getUrl().startsWith("https://api.binance.com")) {
            console.log("Not a Binance API URL.");
            return;
        }

        // Check if a timestamp parameter exists
        if (!req.hasParameter("timestamp")) {
            console.log("No timestamp parameter, not signing.");
            return;
        }

        // Check the API Key is present in the environment variables
        const key = req.getEnvironmentVariable('api_secret');
        if (key == null) {
            throw new Error("Cannot find environment variable 'api_secret'");
        }

        console.log('Generating request signature...');

        // The message to be signed for Binance API is the query string
        const message = computeSigningBase(req);
        // Use crypto-js library to compute the hash
        const signature = computeHttpSignature(message, key);

        // Set the signature parameter on the outgoing request
        req.setParameter('signature', signature);

        console.log('Signature appended to outgoing request');

The context object doesn't provide the query string directly, but it can be generated easily:

function computeSigningBase(req) {
    const paramObj = req.getParameters();

    var params = [];
    for (const p of paramObj) {
        params.push(`${p.name}=${p.value}`);
    }
    return params.join("&");
}

The hash function is straightforward from the crypto-js library:

const CryptoJS = require('crypto-js');

function encodeURL(str) {
    return str.replace(/\+/g, '-').replace(/\//g, '_');
}

function computeHttpSignature(msg, key) {
    const hash = CryptoJS.HmacSHA256(msg, key);
    return encodeUrl(CryptoJS.enc.Hex.stringify(hash));

Using the plugin

Once I was happy with how things worked, I wrote up a basic package.json file and published to the NPM registry as insomnia_plugin_binance_signing. Insomnia has a plugin manager that will search NPM packages and automatically install from there. Once it loads the plugin (for a response/request hook), it will automatically apply that plugin to all incoming/outgoing messages, so there's nothing special I need to do in my setup after this.

Had I went with a template tag, the only additional step would have been to add the tag into the correct spot in the request GUI.

I don't have the rest of the signed endpoints set up yet, but the ones I have tried work perfectly now. If any request includes the timestamp parameter already (using the Insomnia built-in), it will be signed on its way out.

Telegram notifications from website deployment

Anson VanDoren — Tue, 24 Dec 2019 14:53:19 +0000

I recently set up webhooks on my remote machine to receive POST requests from GitHub whenever I pushed a change to ansonvandoren.com. It works well, but since it's running as a background service on a remote server, I don't get a lot of detail about its status.

About a year ago, when I first started using Telegram as my daily-driver chat app (thanks, Google, for cancelling or ruining every chat service you've ever invented), I noticed they also offered an API. I immediately thought of several things that could be useful for, but lacked the time and motivation to explore it further.

Today I had both a good project, and some time to implement it, so I started fiddling around.

What I want to accomplish:

Current process

Make website changes locally, commit them to a git repo, and push to GitHub.
GitHub sends a POST request to my server, which then:
- Clones the repo locally on the server.
- Builds the static site using Hugo.
- Copies the built content to the public WWW folder.
If any part of 2. fails, it will roll back to the previous version and log a simple failure message.

What I want to add

Integrate with Telegram so that I will get a notification when a build/deploy succeeds, and a failure message otherwise with some indication of what went wrong.

Where I'm starting from

Here's my current build script:

#!/bin/bash -e
# Note the '-e' in the line above. This is required for the
# error trapping implemented below.

# Repo name on GitHub
REMOTE_REPO=https://github.com/anson-vandoren/ansonvandoren.com.git
# A place to clone the remote repo so Hugo can build from it
WORKING_DIRECTORY=$HOME/staging_area
# Location (server block) where Nginx looks for content to serve
PUBLIC_WWW=/var/www/ansonvandoren.com/html
# Backup folder in case something goes wrong during this script
BACKUP_WWW=$HOME/backup_html
# Domain name so Hugo can generate links correctly
MY_DOMAIN=ansonvandoren.com

# For future notifications on Telegram
commit_message=$1
pusher_name=$2
commit_id=$3

# If something goes wrong, put the previous version back in place
function cleanup {
    echo "A problem occurred. Reverting to backup."
    rsync -aqz --del $BACKUP_WWW/ $PUBLIC_WWW
    rm -rf $WORKING_DIRECTORY
    # **Placeholder for Telegram notification**
}

# Call the cleanup function if this script exits abnormally. The -e flag
# in the shebang line ensures an immediate abnormal exit on any error
trap cleanup EXIT

# Clear out the working directory
rm -rf $WORKING_DIRECTORY
# Make a backup copy of current website version
rsync -aqz $PUBLIC_WWW/ $BACKUP_WWW

# Clone the new version from GitHub
git clone $REMOTE_REPO $WORKING_DIRECTORY

# **Placeholder for Telegram notification**

# Delete the old version
rm -rf $PUBLIC_WWW/*
# Have Hugo generate the new static HTML directly into the public WWW folder
/usr/local/bin/hugo -s $WORKING_DIRECTORY -d $PUBLIC_WWW -b "https://${MY_DOMAIN}"

# **Placeholder for Telegram notification**

# Clear out working directory
rm -rf $WORKING_DIRECTORY
# Exit without trapping, since everything went well
trap - EXIT

You can see where I stubbed in the places I thought I could hook into Telegram, but so far there's no code or infrastructure behind it.

Creating a Telegram bot

After some initial research on the Telegram API documentation, the first step seems to be to create a new bot using the "BotFather". This is a painless process done entirely through the Telegram app, and within a few seconds I was the proud owner of ansonvandoren_bot, and I had my API token (hereafter denoted <TOKEN>)

After obtaining the token, I started a chat with the bot so I could get a conversation ID. I don't think it's possible for the bot to initiate a chat (probably a good thing), so I need to take the first step. The BotFather conversation will have a link to start talking with your robotic spawn, or else you can just go to https://t.me/your_new_bot to get redirected.

Getting the chat ID can be done from a browser, or from the command line via a tool like curl. If you're not familiar with the curl tool, there is a very readable guide here. In this first case, I'm using it with no arguments, which defaults to sending a HTTP GET request to the URL specified, and displaying the output.

$ curl https://api.telegram.org/bot<TOKEN>/getUpdates
{"ok":true,"result":[{"update_id":424724792,
..."chat":{"id":123456184,"first_name":"Anson","last_name":"VanDoren",...

The "id":123456184 field here is what I was looking for, and is the identifier for my conversation with the bot. Since my particular use-case only involves a single conversation (me vs. The Bot), I don't need to get any fancier than this. I'll refer to this field as <CHAT_ID> for the rest of this post.

Now that I've got the two key pieces of information, I can try sending a message. I'm using curl again, but this time with the -s (silent) and -X POST use the HTTP POST verb instead of GET. I'm specifying the parameters I want to send with the -d options, one at a time. Scroll right to see the whole thing:

$ curl -s -X POST https://api.telegram.org/bot<TOKEN>/sendMessage -d chat_id=<CHAT_ID> -d text="The bot speaks!"

It worked! That should really be most of the API functionality I'll need for this little project. The API is actually very robust, and geared toward richer content and interactivity. I will probably come back and explore it later, but this project has a pretty narrow scope and I just wanted to get it finished up.

Integrating the bot with my build/deploy script

All that's left is to figure out how to work this into my build script. This part can be as basic or complicated as you want. Below is how I decided to get notified:

#!/bin/bash -e
# Note the '-e' in the line above. This is required for
# error trapping implemented below.

# Repo name on GitHub
REMOTE_REPO=https://github.com/anson-vandoren/ansonvandoren.com.git
# A place to clone the remote repo so Hugo can build from it
WORKING_DIRECTORY=$HOME/staging_area
# Location (server block) where Nginx looks for content to serve
PUBLIC_WWW=/var/www/ansonvandoren.com/html
# Backup folder in case something goes wrong during this script
BACKUP_WWW=$HOME/backup_html
# Domain name so Hugo can generate links correctly
MY_DOMAIN=ansonvandoren.com

# Set up Telegram
TOKEN=INSERT_YOUR_TOKEN_HERE
CHAT_ID=INSERT_YOUR_CHAT_ID_HERE
BOT_URL="https://api.telegram.org/bot$TOKEN/sendMessage"

# Send messages to Telegram bot
function send_msg () {
    # Use "$1" to get the first argument (desired message) passed to this function
    # Set parsing mode to HTML because Markdown tags don't play nice in a bash script
    # Redirect curl output to /dev/null since we don't need to see it
    # (it just replays the message from the bot API)
    # Redirect stderr to stdout so we can still see an error message in curl if it occurs
    curl -s -X POST $BOT_URL -d chat_id=$CHAT_ID -d text="$1" -d parse_mode="HTML" > /dev/null 2>&1
}

commit_message=$1
pusher_name=$2
commit_id=$3

# If something goes wrong, put the previous verison back in place
function cleanup {
    echo "A problem occurred. Reverting to backup."
    rsync -aqz --del $BACKUP_WWW/ $PUBLIC_WWW
    rm -rf $WORKING_DIRECTORY

    # Use $? to get the error message that caused the failure
    send_msg "<b>Deployment of ansonvandoren.com failed:</b> $?"
}

# Call the cleanup function if this script exits abnormally. The -e flag
# in the shebang line ensures an immediate abnormal exit on any error
trap cleanup EXIT

# Clear out the working directory
rm -rf $WORKING_DIRECTORY
# Make a backup copy of current website version
rsync -aqz $PUBLIC_WWW/ $BACKUP_WWW

# Clone the new version from GitHub
git clone $REMOTE_REPO $WORKING_DIRECTORY

send_msg "<i>Successfully cloned GitHub repo for ansonvandoren.com</i>
<code>Message: $commit_message</code>
<code>Pushed by: $pusher_name</code>"

# Delete old version
rm -rf $PUBLIC_WWW/*
# Have Hugo generate the new static HTML directly into the public WWW folder
# Save the output of Hugo to send to Telegram
hugo_response=$(/usr/local/bin/hugo -s $WORKING_DIRECTORY -d $PUBLIC_WWW -b "https://${MY_DOMAIN}")
# Send Hugo response to bot as a fenced code block to preserve formatting
send_msg "<pre>$hugo_response</pre>"

# All done!
send_msg "<b>Deployment successful!</b>"

# Clear out working directory
rm -rf $WORKING_DIRECTORY
# Exit without trapping, since everything went well
trap - EXIT

A few things I learned:

How to pass arguments to a bash function: this is different from 'normal' progrmaming languages in that you don't define what the arguments are, but simply refer to them in sequential order with $1, $2, etc. within the function.
How to hide the output of a command: curl... > /dev/null
How to redirect error output to stdout: 2>&1
How to get the last error message raised by the script: $? Note: I have not been able to test this yet because I've not seen any errors on deployment. StackOverflow seems pretty convinced it should work, though.
A limited subset of HTML can be used to format Telegram bot messages. Markdown (an alternative formatting option allowed by the API) does not work well inside a bash script because the back-ticks mess up the formatting of the script.
You can save the output of a command to a variable using variable_name=$(command)
To use the full (multi-line) response stored from a command output as above, you need to put it inside double quotes. $hugo_response in my example would only show the first line, but "$hugo_response" does multi-line.

Testing it all out

As with previous examples, I created a dummy commit and pushed to GitHub to try it out:

$ git commit --allow-empty -m "Test commit"
$ git push

Configuring Nginx to Proxy Webhooks

Anson VanDoren — Mon, 23 Dec 2019 14:21:03 +0000

In my last post, I set up a workflow that allows me to make changes to my website in a local git repository, push it to GitHub, and have my remote server build the static content with Hugo and serve up the changes. On the remote server, I handled the GitHub hooks through a small Golang server called webhook, but wasn't really happy with needing to open up a separate port (it defaults to 9000) on my server firewall. In the grand scheme of things, it probably doesn't matter much from a security standpoint, but I thought I could do a bit better and learn something in the process of configuring Nginx to proxy the requests through HTTPS.

When I set up my website, I had already used Let's Encrypt to enable HTTPS by default. As part of that process, I used the certbot tool they provide to automatically modify my Nginx server block configuration, but now I needed to go back and modify them by hand.

Starting configuration

Firewall status

At the end of the last post, I had port 9000 open through ufw to enable the webhooks server:

$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
Nginx Full                 ALLOW       Anywhere
9000                       ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
Nginx Full (v6)            ALLOW       Anywhere (v6)
9000 (v6)                  ALLOW       Anywhere (v6)

Nginx configuration

My starting Nginx configuration looked like this, based on these instructions and what Certbot changed:

server {
    # Redirect www to non-www. Hugo doesn't like subdomains
    if ($host = www.ansonvandoren.com) {
        return 301 https://ansonvandoren.com$request_uri;
    }

    root /var/www/ansonvandoren.com/html;
    index index.html index.htm index.nginx-debian.html;
    server_name www.ansonvandoren.com ansonvandoren.com;

    location / {
        try_files $uri $uri/ =404;
    }

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/ansonvandoren.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/ansonvandoren.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
    # Redirect to HTTPS (and also redirect www to non-www)
    if ($host = www.ansonvandoren.com) {
        return 301 https://ansonvandoren.com$request_uri;
    } # managed by Certbot

    # Redirect to HTTPS
    if ($host = ansonvandoren.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
    listen [::]:80;
    server_name ansonvandoren.com www.ansonvandoren.com

    return 404; # managed by Certbot
}

Actually, it was quite a bit messier than that when I first looked, but I did some cleanup and rearranging through trial and error until I got things looking the way I wanted.

webhooks server

My webhooks server was listening on 0.0.0.0, port 9000

GitHub configuration

My website repo on GitHub had a webhook set up with:

Payload URL: http://ansonvandoren.com:9000/hooks/redeploy
Content type: application/json
Enable SSL verification turned on
Triggering event set to 'Just the push event'

What I wanted instead

I had two main things I wanted to accomplish with these changes:

Close port 9000 on ufw
Change my Payload URL on GitHub to use HTTPS (and not require port 9000)

How I did it

In hindsight, this was actually not a difficult problem to solve, but it did take quite a bit of trial and error, and I couldn't find any examples that accomplished exactly what I was looking for.

Close the firewall port

Since I'm the only one pushing commits to the website repo, I won't be interrupting anything if I close the port now.

$ sudo ufw delete allow 9000
Rule deleted
Rule deleted (v6)
$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
Nginx Full                 ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
Nginx Full (v6)            ALLOW       Anywhere (v6)

So far, so good...

Add Nginx proxy

This took me the most time to sort out, but it's relatively simple.

$ sudo vim /etc/nginx/sites-available/ansonvandoren.com

In my first server block (the one listening on 443), I added the lines shown below for the /hooks/ location without changing anything else in the server blocks.

server {
    ...
    server_name www.ansonvandoren.com ansonvandoren.com;

    # Forward webhook requests
    location /hooks/ {
        proxy_pass http://127.0.0.1:9000/hooks/;
    }

    location / {
        try_files $uri $uri/ =404;
    }
    ...
}

Simple, right? If any web requests come in over HTTPS on the /hooks path, they get proxied to localhost:9000, which is where the webhooks server is already listening.

Update GitHub

Almost done, and just need to let my GitHub webhook know about the changes. Back on the same settings page, I just needed to change the Payload URL to:

Payload URL: https://ansonvandoren.com/hooks/redeploy

To prevent you needing to scroll back up, I just removed the ':9000' port identifier, and changed the http:// to https://

Testing it out

To make sure everything worked after my changes, I made some edits to my site locally, committed them, and pushed to GitHub.

The webhook service looks good:

$ systemctl --user status webhook
● webhook.service - Simple Golang webhook server
   Loaded: loaded (/home/bloguser/.config/systemd/user/webhook.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-01-21 00:40:37 UTC; 1h 40min ago
 Main PID: 851 (webhook)
   CGroup: /user.slice/user-1000.slice/user@1000.service/webhook.service
           └─851 /usr/local/bin/webhook -ip 127.0.0.1 -hooks /opt/hooks/hooks.json -verbose

Jan 21 01:22:00 ansonvandoren webhook[851]:   Pages            | 20
Jan 21 01:22:00 ansonvandoren webhook[851]:   Paginator pages  |  0
Jan 21 01:22:00 ansonvandoren webhook[851]:   Non-page files   |  0
Jan 21 01:22:00 ansonvandoren webhook[851]:   Static files     |  5
Jan 21 01:22:00 ansonvandoren webhook[851]:   Processed images |  0
Jan 21 01:22:00 ansonvandoren webhook[851]:   Aliases          |  7
Jan 21 01:22:00 ansonvandoren webhook[851]:   Sitemaps         |  1
Jan 21 01:22:00 ansonvandoren webhook[851]:   Cleaned          |  0
Jan 21 01:22:00 ansonvandoren webhook[851]: Total in 99 ms
Jan 21 01:22:00 ansonvandoren webhook[851]: [webhook] 2019/01/21 01:22:00 [0fa3ba] finished redeploy

A quick check of the webpage shows the changes propagated correctly. Success!

Deploying a Hugo site on a GitHub commit

Anson VanDoren — Sun, 22 Dec 2019 16:44:14 +0000

Doing things the hard way

I recently helped my brother get his first website up and running. After a bit of research, we decided on static HTML, served via Caddy on a Digital Ocean droplet. The choice of Caddy was predominantly due to its built-in hooks that would allow him to maintain his website as a GitHub repo, and have it automatically update every time he pushed a commit.

When redesigning my website yesterday, I should have done the same thing, since our use-cases are pretty similar. But I didn't... Instead, I ended up with:

Digital Ocean droplet
Nginx serving content
Let's Encrypt to avoid the 'Not Secure' message in the URL bar
Hugo static site generator (so I can use Markdown!)

So now I'm left needing to figure out a way to implement the following workflow:

Make a change, or add a new blog post locally. Commit the change locally and push to GitHub.
Magically have my website update to reflect the new content.

Ideally, I want it to be as simple as that. My first implementation (based on this guide) was more complicated, and involved setting a remote repo on my server that I could push to (in addition to GitHub). I don't like this solution because:

It involves an extra step, and means I need to maintain two separate remote repos.
It makes it more likely that I'll forget to push to GitHub as well, so when my server inevitably dies due to neglect or freak accident, I may not have a solid backup.

This is still a workable solution, but I want it to be as streamlined as possible. In hindsight, I should have just used Caddy (with this awesome tutorial), but sometimes I like doing things the hard way.

What I need

A GitHub webhook set up to monitor a PushEvent and send that along to my server.
Nginx to receive the POST event from GitHub and forward it to an internal listener of some sort.
Internal listener to run a shell script that will:
- Fetch the latest commit from GitHub to a working directory
- Run Hugo to build static content based on the new commit
- Copy Hugo output to the appropriate folder that Nginx is serving

Easy, right?

Setting up webhook

The go-to (pun intended) server for this seems to be webhook, written in Go by Adnan Hajdarević. I don't have a Golang environment running on the remote and don't really need one for anything else, so the go get option wasn't optimal, nor was building from source. The Ubuntu package via apt-get install webhook was several versions behind the latest release, so I opted to download the latest released binary instead.

First I had to figure out what architecture I need:

$ uname -i
x86_64

Next, download the correct release (x86_64 version) from here and unzip it:

$ wget https://github.com/adnanh/webhook/releases/download/2.6.9/webhook-linux-amd64.tar.gz
$ tar -xvf webhook*.tar.gz

Make the binary available in my environment, and clean up:

$ sudo mv webhook-linux-amd64/webhook /usr/local/bin
$ rm -rf webhook-linux-amd64*

Make sure everything worked:

$ webhook -version
webhook version 2.6.9

webhooks needs a hook file (which tells it how to handle incoming requests), and a script that it should run if it matches an incoming request. I made a place for those to live, and transferred ownership to my (non-root) user:

$ sudo mkdir /opt/scripts
$ sudo mkdir /opt/hooks
$ sudo chown -R $USER:$USER /opt/scripts
$ sudo chown -R $USER:$USER /opt/hooks

Creating a hook

Next I set up a hooks.json file to allow webhook to trigger on an incoming GitHub POST. I like vim, but you can use nano or whatever other editor you prefer:

$ vim /opt/hooks/hooks.json

The documentation has more details about available properties, but I'll just show the configuration I ended up using.

Before doing that, though, I need a secret I can share with GitHub to ensure the hook is only triggered appropriately (i.e., not by someone randomly sending a POST request to the correct endpoint). You can use anything you want as the secret, but one easy way to generate a secret is to run uuidgen:

 $ uuidgen
 81febb4d-4483-4fc8-a2dc-8ece300bc5f4

This will output a nice long random value that is expected to be unique (see here for more detail). Copy this value, and use it both in the JSON file below, and also when you do the next steps on GitHub. Note that the example below shows "my-github-secret" instead; you'll want to replace this in your version.

[
    {
        "id": "redeploy",
        "execute-command": "/opt/scripts/redeploy.sh",
        "command-working-directory": "/opt/scripts",
        "pass-arguments-to-command":
        [
            {
                "source": "payload",
                "name": "head_commit.message"
            },
            {
                "source": "payload",
                "name": "pusher.name"
            },
            {
                "source": "payload",
                "name": "head_commit.id"
            }
        ],
        "trigger-rule":
        {
            "and":
            [
                {
                    "match":
                    {
                        "type": "payload-hash-sha1",
                        "secret": "my-github-secret",
                        "parameter":
                        {
                            "source": "header",
                            "name": "X-Hub-Signature"
                        }
                    }
                },
                {
                    "match":
                    {
                        "type": "value",
                        "value": "refs/heads/master",
                        "parameter":
                        {
                            "source": "payload",
                            "name": "ref"
                        }
                    }
                }
            ]
        }
    }
]

The GitHub documentation has more details about the different payload parameters for each event type. Basically, the above hook will listen on an endpoint called redeploy, and if the trigger-rule is satisfied, will run the redeploy.sh shell script with arguments from the POST message. This trigger rule will guarantee that it was sent from GitHub (only they have my secret), and the push happened on master branch in my GitHub repo (I don't want to rebuild if the commit was only to a feature/testing branch).

Configuring the firewall

Before moving on, I need to make sure my server firewall will allow the hooks through on port 9000:

$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
Nginx Full                 ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
Nginx Full (v6)            ALLOW       Anywhere (v6)

Right now, I'm only allowing ssh and normal web traffic, so I need to open up port 9000:

$ sudo ufw allow 9000
Rule added
Rule added (v6)
$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
Nginx Full                 ALLOW       Anywhere
9000                       ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
Nginx Full (v6)            ALLOW       Anywhere (v6)
9000 (v6)                  ALLOW       Anywhere (v6)

DigitalOcean has a good primer on ufw if you need more details on configuring a firewall.

Edit #1: In the end, I decided to configure Nginx to proxy requests for https://ansonvandoren.com/hooks/ through to my webhooks server instead, and closed port 9000 again. I may write a separate post on that since it wasn't as trivial as I wanted it to be, but the above method works just as well. The basics of what I did were from hints on this article.

Edit #2: I did actually write the subsequent post, and you can read it here on my blog

Setting up GitHub

On the GitHub page for my static site, I navigated to Settings > Webhooks > Add Webhook.
For the Payload URL, I entered: http://ansonvandoren.com:9000/hooks/redeploy (where redeploy is the id I set up in my hooks.json file).
I chose application/json for Content type, and pasted the secret I generated with uuidgen into the Secret field.
I left Enable SSL verification selected, and for Which events would you like to trigger this webhook?, I chose Just push event, and then clicked Add webhook.

Right away I see a notification icon showing that it failed to deliver, since I haven't actually started the webhook server yet on my remote machine. Once I do, though, GitHub will start sending POST requests every time a commit is pushed to my website repo.

It is possible to set up webhook to use HTTPS if you need to, but it's not as straightforward and I don't have a real need to in my case. If you need to do this, check out this section of the docs.

Edit: Using Nginx to proxy requests through to webhooks as I did later actually makes it easier to use HTTPS for this, since I already had it set up for my web page thanks to Let's Encrypt.

Writing the redeploy script

When I created the hooks.json file, I pointed it to a script called /opt/scripts/redeploy.sh, so now I need to actually write that script.

$ vim /opt/scripts/redeploy.sh

Your needs may vary and you may need to customize this script in different ways than I did, but below should give you some ideas about how to proceed.

#!/bin/bash -e
# Note the '-e' in the line above. This is required for error trapping implemented below.

# Repo name on GitHub
REMOTE_REPO=https://github.com/anson-vandoren/ansonvandoren.com.git
# A place to clone the remote repo so Hugo can build from it
WORKING_DIRECTORY=$HOME/staging_area
# Location (server block) where Nginx looks for content to serve
PUBLIC_WWW=/var/www/ansonvandoren.com/html
# Backup folder in case something goes wrong during this script
BACKUP_WWW=$HOME/backup_html
# Domain name so Hugo can generate links correctly
MY_DOMAIN=ansonvandoren.com

# For future notifications on Telegram
commit_message=$1
pusher_name=$2
commit_id=$3

# If something goes wrong, put the previous version back in place
function cleanup {
    echo "A problem occurred. Reverting to backup."
    rsync -aqz --del $BACKUP_WWW/ $PUBLIC_WWW
    rm -rf $WORKING_DIRECTORY
    # !!Placeholder for Telegram notification
}

# Call the cleanup function if this script exits abnormally. The -e flag
# in the shebang line ensures an immediate abnormal exit on any error
trap cleanup EXIT

# Clear out the working directory
rm -rf $WORKING_DIRECTORY
# Make a backup copy of current website version
rsync -aqz $PUBLIC_WWW/ $BACKUP_WWW

# Clone the new version from GitHub
git clone $REMOTE_REPO $WORKING_DIRECTORY

# !!Placeholder for Telegram notification

# Delete old version
rm -rf $PUBLIC_WWW/*
# Have Hugo generate the new static HTML directly into the public WWW folder
/usr/local/bin/hugo -s $WORKING_DIRECTORY -d $PUBLIC_WWW -b "https://${MY_DOMAIN}"

# !!Placeholder for Telegram notification

# Clear out working directory
rm -rf $WORKING_DIRECTORY
# Exit without trapping, since everything went well
trap - EXIT

With the comments, I think most of that should be clear enough. There's some additional material on bash error trapping (it was brand new to me) that can be found here.

The script needs to be executable so the hook can run it:

$ chmod +x /opt/scripts/redeploy.sh

I wanted to make sure the script ran OK before moving on:

$ bash /opt/scripts/redeploy.sh
Cloning into '/home/blog/staging_area'...
remote: Enumerating objects: 155, done.
remote: Counting objects: 100% (155/155), done.
remote: Compressing objects: 100% (125/125), done.
remote: Total 155 (delta 21), reused 145 (delta 11), pack-reused 0
Receiving objects: 100% (155/155), 620.23 KiB | 7.13 MiB/s, done.
Resolving deltas: 100% (21/21), done.

                   | EN
+------------------+----+
  Pages            | 15
  Paginator pages  |  0
  Non-page files   |  0
  Static files     |  5
  Processed images |  0
  Aliases          |  5
  Sitemaps         |  1
  Cleaned          |  0

Total in 47 ms

Testing with webhooks

Everything looks good, and the website is still working. Let's try a real test by running webhooks:

$ webhook -hooks /opt/hooks/hooks.json -verbose
[webhook] 2019/01/20 21:46:23 version 2.6.9 starting
[webhook] 2019/01/20 21:46:23 setting up os signal watcher
[webhook] 2019/01/20 21:46:23 attempting to load hooks from /opt/hooks/hooks.json
[webhook] 2019/01/20 21:46:23 found 1 hook(s) in file
[webhook] 2019/01/20 21:46:23   loaded: redeploy
[webhook] 2019/01/20 21:46:23 serving hooks on http://0.0.0.0:9000/hooks/{id}
[webhook] 2019/01/20 21:46:23 os signal watcher ready

Now, from my local machine, I'll create a test commit and push to GitHub

$ git commit --allow-empty -m "Trigger test"
[master eb3fdc6] Trigger test
$ git push origin master
Enumerating objects: 1, done.
Counting objects: 100% (1/1), done.
Writing objects: 100% (1/1), 197 bytes | 197.00 KiB/s, done.
Total 1 (delta 0), reused 0 (delta 0)
To github.com:anson-vandoren/ansonvandoren.com.git
   c12a30d..eb3fdc6  master -> master

It worked! Here's the output from webhooks back on the remote machine:

[webhook] 2019/01/20 21:46:34 [dfa235] incoming HTTP request from 192.30.252.37:39834
[webhook] 2019/01/20 21:46:34 [dfa235] redeploy got matched
[webhook] 2019/01/20 21:46:34 [dfa235] redeploy hook triggered successfully
[webhook] 2019/01/20 21:46:34 200 | 1.222118ms | ansonvandoren.com:9000 | POST /hooks/redeploy
[webhook] 2019/01/20 21:46:34 [dfa235] executing /opt/scripts/redeploy.sh (/opt/scripts/redeploy.sh) with arguments ["/opt/scripts/redeploy.sh" "Trigger notification5" "anson-vandoren" "4a1...7a2"] and environment [] using /opt/scripts as cwd
[webhook] 2019/01/20 21:46:35 [dfa235] command output: Cloning into '/home/blog/staging_area'...
Building sites …
                   | EN
+------------------+----+
  Pages            | 15
  Paginator pages  |  0
  Non-page files   |  0
  Static files     |  5
  Processed images |  0
  Aliases          |  5
  Sitemaps         |  1
  Cleaned          |  0

Total in 40 ms

[webhook] 2019/01/20 21:46:35 [dfa235] finished handling redeploy

Adding webhooks as a service

Since I'll be running this on a remote machine, I want to set it up as a systemd service so it will come back up if the server reboots for any reason. It took me a while to settle on the "right" way to do this one, but I think this should work well.

First, I need to create a service file as my non-root user, in the home directory as shown below. Again, you can use nano if you're not comfortable with vim.

$ vim ~/.config/systemd/user/webhook.service

The contents of this file should look similar to this:

[Unit]
Description=Simple Golang webhook server
ConditionPathExists=/usr/local/bin/webhook
After=network.target

[Service]
Type=simple
WorkingDirectory=/home/blog
ExecStart=/usr/local/bin/webhook -ip 127.0.0.1 -hooks /opt/hooks/hooks.json -verbose
Restart=on-failure
PrivateTmp=true

[Install]
WantedBy=default.target

For a complete descriptions of the sections of a service file, you can check out this documentation from RedHat.

Next I needed to ensure that my non-root user (shown below as 'myuser', but substitute whatever login you are using for this) could keep a process running even when not logged in. After enabling 'lingering', I enabled my new service and started it.

$ sudo loginctl enable-linger myuser
$ systemctl --user enable webhook
$ systemctl --user start webhook

Everything should be working correctly now, so I ran another empty commit/push from my local machine, and then checked the status of the service:

$ systemctl --user status webhook
● webhook.service - Simple Golang webhook server
   Loaded: loaded (/home/blog/.config/systemd/user/webhook.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-01-21 00:34:03 UTC; 50s ago
 Main PID: 20422 (webhook)
   CGroup: /user.slice/user-1000.slice/user@1000.service/webhook.service
           └─20422 /usr/local/bin/webhook -ip 127.0.0.1 -hooks /opt/hooks/hooks.json -verbose

Jan 21 00:34:49 ansonvandoren webhook[20422]:   Pages            | 15
Jan 21 00:34:49 ansonvandoren webhook[20422]:   Paginator pages  |  0
Jan 21 00:34:49 ansonvandoren webhook[20422]:   Non-page files   |  0
Jan 21 00:34:49 ansonvandoren webhook[20422]:   Static files     |  5
Jan 21 00:34:49 ansonvandoren webhook[20422]:   Processed images |  0
Jan 21 00:34:49 ansonvandoren webhook[20422]:   Aliases          |  5
Jan 21 00:34:49 ansonvandoren webhook[20422]:   Sitemaps         |  1
Jan 21 00:34:49 ansonvandoren webhook[20422]:   Cleaned          |  0
Jan 21 00:34:49 ansonvandoren webhook[20422]: Total in 53 ms
Jan 21 00:34:49 ansonvandoren webhook[20422]: [webhook] 2019/01/21 00:34:49 [67d5a4] finished handling

It worked! Just to make sure (I'm a little paranoid), I rebooted my remote machine, and checked again when it came back up. Everything worked as advertised, and webhook is running again, waiting for my next git commit.

The only thing I still need to set up is the Telegram notifications for build/deploy status. This post is long enough as it is, and that's almost an entirely separate topic, so I'll leave that for a separate post that you can find here.

This post was first originally published on my own blog