DEV Community: Euan T

Speech To Text With JavaScript

Euan T — Sun, 21 Jan 2024 16:45:41 +0000

I’ve recently been playing with some options to automatically convert speech into text. My original plan was to do it on the back-end in the ASP.NET core server, but my experiments with the System.Speech.Recognition.SpeechRecognitionEngine turned up a few key issues:

The API is only available on Windows.
The accuracy seemed to be pretty poor.

I therefore went looking for alternative options and stumbled across the Web Speech API. This API provides support for both speech recognition (speech to text) and speech synthesis (text to speech) in the web browser. My thinking was that I could convert the speech to text on the client side, write it into a hidden input and post that back to the back-end.

I’m most interested in the SpeechRecognition class in this post, which has quite an easy to use API that supports custom grammars, specifying the language to use for recognition, interim results and final results including degrees of certainty.

It’s probably best to demonstrate with a quick code sample. You can preview it on CodePen too. Unfortunately I can’t embed the CodePen here as the API is blocked when embedded in an iframe. This code sample uses React for state and event handling:

import React from "https://esm.sh/react";
import ReactDOM from "https://esm.sh/react-dom";

const SpeechRecognition = window.SpeechRecognition || window.webkitSpeechRecognition || null;

(function(document, undefined) {
   const targetElement = document.getElementById("root");

   if (!targetElement) {
      console.error('failed to find root');

      return;
   }

   const root = ReactDOM.createRoot(targetElement);

   root.render(
      <React.StrictMode>
         <SpeechRecognizerDemo />
      </React.StrictMode>
   );
})(document);

function SpeechRecognizerDemo({}) {
   const [recognizeSpeech, setRecognizeSpeech] = React.useState(false);
   const [recognizedSpeech, setRecognizedSpeech] = React.useState('');

   React.useEffect(() => {
      let speechRecognizer = null;

      if (recognizeSpeech) {
         setRecognizedSpeech('');

         speechRecognizer = new SpeechRecognition();

         speechRecognizer.continuous = true;
         speechRecognizer.interimResults = false;
         speechRecognizer.maxAlternatives = 1;

         speechRecognizer.onerror = (err) => {
           console.error(err); 
         };

         speechRecognizer.onresult = (event) => {
            setRecognizedSpeech('');

            for (const result of event.results) {
               if (result.isFinal) {
                  setRecognizedSpeech(oldValue => (oldValue + ' ' + result[0].transcript).trim());
               }
            }

            setRecognizedSpeech(oldValue => {
               return oldValue.replace(
                  /\S/,
                  c => c.toUpperCase()
               );
            })
         };

         speechRecognizer.start();
      }

      return () => {
         if (speechRecognizer) {
            speechRecognizer.stop();
         }
      };
   }, [recognizeSpeech]);

   if (!SpeechRecognition) {
      return <div className="alert alert-danger">
         Your web browser doesn't support speech recognition. Please try in a Chromium based browser or Safari.   
      </div>
   }

   return <div className="container-fluid">
      <div className="row">
         <div className="col">
            <button type="button" 
               className={`btn btn-sm btn-${recognizeSpeech ? 'danger' : 'success'}`}
               onClick={evt => {
                  evt.preventDefault();

                  setRecognizeSpeech(oldValue => !oldValue);

                  return false;
               }}>
               {recognizeSpeech ? 'Stop' : 'Start'} </button>
         </div>
      </div>

      <div className="row mt-2">
         <div className="col">
            <textarea className="form-control"
               readonly
               value={recognizedSpeech}></textarea>
         </div>
      </div>
   </div>
}

My initial testing looked pretty positive, but there are some (fairly serious) downsides:

Browser support is fairly limited – it’ll work in most Chromium based browsers and Safari. It won’t work in Firefox at all though. Where it is supported, it’s supported with a vendor prefix.
- I’ve tested on Google Chrome, Safari, and Microsoft Edge and it’s worked on all 3.
I’ve found some references saying that Chromium based browsers send the captured speech to a Google web service to perform the speech recognition, meaning you have to rely on an external 3rd party.

I’m hoping that Firefox will add support fairly soon. For now I’m looking at providing this as an option hidden behind a setting labeled as a preview so that end-users can opt-in if they wish to.

Monitoring Windows Services With C#

Euan T — Tue, 04 Apr 2023 14:33:03 +0000

Ever wanted to monitor the state of Windows Services in a .NET project? This is a problem I recently had to solve and after some experimentation I settled on using WMI to monitor changes.

This approach allows monitoring the start mode, state, process ID, and much more for any Windows Services on either a local or remote server.

To get started, you’ll need to install the System.Management NuGet package:

dotnet add package System.Management --version 7.0.0

Then it’s simply a case of using the ManagementEventWatcher class to monitor for intrinsic events where the target instance is of the type Win32_Service.

Note that there are a lot of other event types and classesthat you can monitor with WMI – these are worth experimenting with too!

So how does this work? Let’s see an example:

using System.Management;

EventQuery query = new("SELECT * FROM __InstanceOperationEvent WITHIN 1 WHERE TargetInstance isa \"Win32_Service\"");

using ManagementEventWatcher eventWatcher = new(query);
eventWatcher.EventArrived += EventArrived;

try
{
    eventWatcher.Start();

    Console.ReadLine();

    eventWatcher.Stop();
}
finally
{
    eventWatcher.EventArrived -= EventArrived;
}

static void EventArrived(object? sender, EventArrivedEventArgs e)
{
    if (e.NewEvent.GetPropertyValue("TargetInstance") is ManagementBaseObject instanceDescription)
    {
        using (instanceDescription)
        {
            foreach (PropertyData property in instanceDescription.Properties)
            {
                Console.WriteLine("{0}:: {1}", property.Name, property.Value);
            }

            Console.WriteLine();
        }
    }
}

This is a console app which starts watching for events, and whenever any instance event occurs it prints each property for the service that changed to the standard output. If you run this and then start/stop some services, you should see some output like the following:

AcceptPause:: False
AcceptStop:: False
Caption:: Background Intelligent Transfer Service
CheckPoint:: 0
CreationClassName:: Win32_Service
DelayedAutoStart:: True
Description:: Transfers files in the background using idle network bandwidth. If the service is disabled, then any applications that depend on BITS, such as Windows Update or MSN Explorer, will be unable to automatically download programs and other information.
DesktopInteract:: False
DisplayName:: Background Intelligent Transfer Service
ErrorControl:: Normal
ExitCode:: 0
InstallDate::
Name:: BITS
PathName:: C:\WINDOWS\System32\svchost.exe -k netsvcs -p
ProcessId:: 0
ServiceSpecificExitCode:: 0
ServiceType:: Share Process
Started:: False
StartMode:: Auto
StartName:: LocalSystem
State:: Stopped
Status:: OK
SystemCreationClassName:: Win32_ComputerSystem
SystemName:: COMPUTER-NAME
TagId:: 0
WaitHint:: 0

Actions such as changing the service start mode or the display name will also cause events to be raised too.

You can find all of the available properties by looking at the documentation for the Win32_Service class.

Logging with Source Generators in .NET

Euan T — Sun, 19 Feb 2023 10:24:13 +0000

.NET 6 added a new feature called Source Generators, allowing projects to generate code at compile time. This opens up a whole load of cool new opportunities for libraries to implement new functionality or improve existing functionality by eliminating boilerplate code.

One such library is Microsoft.Extensions.Logging, the standard logging library supplied by many of the built in templates for apps like web apps and worker apps.

The logging library supports compile-time logging source generation via the LoggerMessageattribute. It generates code which includes calls to LoggerMessage.Define and includes checks for whether the log level is enabled or not (though this check can be omitted by passing a parameter to the attribute).

So, what does this look like?

The definition for logging methods is pretty simple:

using Microsoft.Extensions.Logging;

internal static partial class Logging
{
    [LoggerMessage(
        EventId = 0,
        EventName = "Main",
        Level = LogLevel.Information,
        Message = "Program is starting"
    )]
    internal static partial void ProgramStarting(ILogger logger);

    [LoggerMessage(
        EventId = 1,
        EventName = "Main",
        Level = LogLevel.Error,
        Message = "Error dividing by divisor: {divisor}"
    )]
    internal static partial void DivisionError(ILogger logger, Exception ex, decimal divisor);
}

So far, pretty simple. You define a static partial class to contain your logging methods, which are in turn defined as static partial void. You then decorate each logging method with the LoggerMessageattribute and provide an event ID, an optional event name, a log level and a message. Your methods can take parameters to be substituted within the log message, along with an exception to be logged for error logging purposes.

At compile time, this then generates code like the following:

partial class Logging
{
    [global::System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.Extensions.Logging.Generators", "7.0.7.1805")]
    private static readonly global::System.Action<global::Microsoft.Extensions.Logging.ILogger, global::System.Exception?> __ProgramStartingCallback =
        global::Microsoft.Extensions.Logging.LoggerMessage.Define(global::Microsoft.Extensions.Logging.LogLevel.Information, new global::Microsoft.Extensions.Logging.EventId(0, "Main"), "Program is starting", new global::Microsoft.Extensions.Logging.LogDefineOptions() { SkipEnabledCheck = true }); 

    [global::System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.Extensions.Logging.Generators", "7.0.7.1805")]
    internal static partial void ProgramStarting(global::Microsoft.Extensions.Logging.ILogger logger)
    {
        if (logger.IsEnabled(global::Microsoft.Extensions.Logging.LogLevel.Information))
        {
            __ProgramStartingCallback(logger, null);
        }
    }
    [global::System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.Extensions.Logging.Generators", "7.0.7.1805")]
    private static readonly global::System.Action<global::Microsoft.Extensions.Logging.ILogger, global::System.Decimal, global::System.Exception?> __DivisionErrorCallback =
        global::Microsoft.Extensions.Logging.LoggerMessage.Define<global::System.Decimal>(global::Microsoft.Extensions.Logging.LogLevel.Error, new global::Microsoft.Extensions.Logging.EventId(1, "Main"), "Error dividing by divisor: {divisor}", new global::Microsoft.Extensions.Logging.LogDefineOptions() { SkipEnabledCheck = true }); 

    [global::System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.Extensions.Logging.Generators", "7.0.7.1805")]
    internal static partial void DivisionError(global::Microsoft.Extensions.Logging.ILogger logger, global::System.Exception ex, global::System.Decimal divisor)
    {
        if (logger.IsEnabled(global::Microsoft.Extensions.Logging.LogLevel.Error))
        {
            __DivisionErrorCallback(logger, divisor, ex);
        }
    }
}

Usage is simple too:

using ILoggerFactory loggerFactory = LoggerFactory.Create(b =>
{
    b.AddConsole();
});

ILogger logger = loggerFactory.CreateLogger<Program>();

Logging.ProgramStarting(logger);

// some time later when an error occurs...

Logging.DivisionError(logger, ex, divisor);

So, what are the advantages?

There are a couple of advantages to this as I see them:

Your log messages are presence checked at compile time. This means if you define a parameter in your logger message but forget to pass it within the method definition, the project will fail to compile with an error such as: error SYSLIB1014: Template 'divisor' is not provided as argument to the logging method.
In addition to the above, parameters are checked to ensure that parameters are named the same in both the logger message definition and the method parameter. Failure to do so also results in a compile error such as: SYSLIB1015: Argument 'd' is not referenced from the logging message.
As message parameter names are checked at compile time and used by name, you can provide parameters in the method signature in any order. This means that if you decide to change the log message in the future to add a parameter, you can simply add it to the end of the method signature rather than having to update every usage.
Provides compile warnings for duplicated event IDs. If you define two log methods with the same event ID you get a compile warning such as: warning SYSLIB1006: Multiple logging methods are using event id 0 in class Logging.

Setting up PHP with FastCGI on Windows using Apache

Euan T — Thu, 05 Jan 2023 16:57:18 +0000

I’ve recently been updating a few Windows machines to PHP 8.2, and decided that as I was doing the update I’d move from mod_phpto using FastCGI instead.

On a Linux server I would tend to use PHP-FPM, which unfortunately isn’t available for Windows. Instead, I ended up opting for mod_fcgid.

The Apache Haus distribution doesn’t ship with mod_fcgidincluded by default though, and you have to install it as an extra step which I thought was worth documenting for future reference.

Install PHP

I use Chocolateyto manage packages, so installing PHP is as easy as:

choco install php

The manual installation process is pretty simple too though:

Download the current either x86 or x64 (depending on your OS) “Non Thread Safe” distribution from PHP for Windows.
Extract the downloaded ZIP to somewhere on your filesystem. Chocolatey installs to C:\tools\php82 by default, and the rest of this post will assume the same path.
Add C:\tools\php82 as an entry in your PATHenvironment variable.

Copy the C:\tools\php82\php.ini-development or C:\tools\php82\php.ini-production file to C:\tools\php82\php.ini and make any required changes.

Check that PHP is working by checking its version:

php -v

You should expect some output such as:

PHP 8.2.0 (cli) (built: Dec 6 2022 15:25:41) (NTS Visual C++ 2019 x64)
Copyright (c) The PHP Group
Zend Engine v4.2.0, Copyright (c) Zend Technologies

Install Apache

Again, using Chocolatey this step is easy:

choco install apache-httpd --params '/port:80 /installLocation:C:\Apache24'

This will install Apache as a service and will install to the directory C:\Apache24.

Manual installation is similarly easy:

Download the current Apache release from Apache Haus, making sure that if you chose an x64 version of PHP that you also choose an x64 version of Apache.
Extract the downloaded folder to C:\Apache24.
Run the following command to install Apache as a service:

C:\Apache24\Apache24\bin\httpd.exe -k install -n Apache

Add C:\Apache24\Apache24\bin as an entry in your PATHenvironment variable.

Make any required changes to C:\Apache24\Apache24\conf\httpd.conf to configure the server, such as setting the document root, enabling modules, etc.

Check the syntax of the server configuration file:

httpd -t

You should expect some output such as:

Syntax OK

Install `mod_fcgid`

We have to install mod_fcgidmanually, as it is not included when installed either through Chocolatey or manually.

Head to the Apache Haus downloads page.
Scroll down to the heading “Modules for Apache 2.4.x VS17” (assuming you installed the version of Apache compiled with VS17).
Download the “Mod FCGID … for Apache 2.4.x” module, making sure that if you chose an x64 version of Apache that you also choose an x64 version of mod_fcgid.
Open the downloaded ZIP file, which should contain the following items (and several others that we’re not interested in):
- conf
- mod_fcgid.so
Copy these items as follows:
- Copy the content of confto C:\Apache24\Apache24\conf
- Copy mod_fcgid.so to C:\Apache24\Apache24\modules\mod_fcgid.so

Now that we have the module installed, we need to load and configure it:

Open C:\Apache24\Apache24\conf\httpd.conf in a text editor.
Add or uncomment the following to the end of the file (or put it next to the other Include directives):

LoadModule fcgid_module modules/mod_fcgid.so
Include conf/extra/httpd-fcgid.conf

Open C:\Apache24\Apache24\conf\extra\httpd-fcid.conf in a text editor.
Replace the content of the file with the following:

<IfModule fcgid_module>
    FcgidInitialEnv PHPRC "C:\\tools\\php82"
    FcgidInitialEnv PATH "C:\\tools\\php82;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;"
    FcgidInitialEnv SystemRoot "C:\\Windows"
    FcgidInitialEnv SystemDrive "C:"
    FcgidInitialEnv TEMP "C:\\WINDOWS\\TEMP"
    FcgidInitialEnv TMP "C:\\WINDOWS\\TEMP"
    FcgidInitialEnv windir "C:\\WINDOWS"

    FcgidIOTimeout 40
    FcgidConnectTimeout 10
    FcgidMaxProcesses 8
    FcgidOutputBufferSize 64
    ProcessLifeTime 240 
    FcgidMaxRequestsPerProcess 0
    FcgidInitialEnv PHP_FCGI_MAX_REQUESTS 0
    FcgidInitialEnv PHP_FCGI_CHILDREN 0
    FcgidProcessLifeTime 3600
    FcgidMinProcessesPerClass 0

    <Files ~ "\.php$">
        Options ExecCGI SymLinksIfOwnerMatch
        AddHandler fcgid-script .php
        FcgidWrapper "C:/tools/php82/php-cgi.exe" .php
    </Files>
</IfModule>

Restart the Apache service.

Closing Thoughts

Immediately after switching to mod_fcgid, there was a perceivable performance improvement. This is most likely due to mod_phprequiring a thread safe version of PHP which will include locks.

The actual upgrade to PHP 8.2 from 8.1 itself also went pretty smoothly, with no breakage to existing code – which is always nice!

Searching Paged Data In Jetpack Compose

Euan T — Sun, 06 Nov 2022 20:15:42 +0000

I've recently created a new Android app using the Jetpack Compose UI App Development Toolkit which consisted of a list of items which needed to be searchable. As the list was quite long, and the API I was talking to implemented support for pagination, I made use of the Paging library to lazy load items into the list as it scrolled.

Nice and easy so far. Now came the time to implement searching for items. I wanted to be able to toggle a search box at the top of the page and have the list of items filter itself based upon the search term as the user typed. After a little trial and error and a good while searching through Google's documentation given how rarely I work with Android, I came up with a nice and simple solution.

The end result that we're aiming for is something like this:

My ViewModel layers all use Kotlin Flows to store state, meaning my basic view model for the paged data looked something like the following:



@HiltViewModel
class ProductListViewModel @Inject constructor(
    private val repository: Repository
): ViewModel() {
    val products = Pager(
        PagingConfig(
            pageSize = 10,
            enablePlaceholders = false,
        )
    ) {
        ProductPagingSource(
            repository = repository,
            search = query,
        )
    }.flow.cachedIn(viewModelScope)
}

(Note, I use Hilt to inject dependencies into my view models, but that's not important for this post)

To add searching to this, I needed to add a few new properties and a couple of functions:

A boolean to toggle whether the search box is visible.
A string to contain the current search term.
Functions to toggle the search box visibility, and to set the search term.

My view model now looked like this:



@HiltViewModel
class ProductListViewModel @Inject constructor(
    private val repository: Repository
): ViewModel() {
    private val _search = MutableStateFlow("")

    val search = _search.asStateFlow()
        .stateIn(
            scope = viewModelScope,
            started = SharingStarted.WhileSubscribed(),
            initialValue = "",
        )

    private val _isSearchShowing = MutableStateFlow(false)

    val isSearchShowing = _isSearchShowing.asStateFlow()
        .stateIn(
            scope = viewModelScope,
            started = SharingStarted.WhileSubscribed(),
            initialValue = false,
        )

    val products = Pager(
        PagingConfig(
            pageSize = 10,
            enablePlaceholders = false,
        )
    ) {
        ProductPagingSource(
            repository = repository,
            search = query,
        )
    }.flow.cachedIn(viewModelScope)

    fun setSearch(query: String) {
        _search.value = query
    }

    fun toggleIsSearchShowing() {
        _isSearchShowing.value = !_isSearchShowing.value
    }
}

Now we just need to update the paged data when the search term changes. Luckily, this is extremely easy when working with flows as we can use a flow transformation (namely flatMapLatest) so that when the search term changes a new flow is emitted:



@HiltViewModel
class ProductListViewModel @Inject constructor(
    private val repository: Repository
): ViewModel() {
    private val _search = MutableStateFlow("")

    val search = _search.asStateFlow()
        .stateIn(
            scope = viewModelScope,
            started = SharingStarted.WhileSubscribed(),
            initialValue = "",
        )

    private val _isSearchShowing = MutableStateFlow(false)

    val isSearchShowing = _isSearchShowing.asStateFlow()
        .stateIn(
            scope = viewModelScope,
            started = SharingStarted.WhileSubscribed(),
            initialValue = false,
        )

    val productsSearchResults = search.debounce(300.milliseconds).flatMapLatest { query ->
        Pager(
            PagingConfig(
                pageSize = 10,
                enablePlaceholders = false,
            )
        ) {
            ProductPagingSource(
                repository = repository,
                search = query,
            )
        }.flow.cachedIn(viewModelScope)
    }


    fun setSearch(query: String) {
        _search.value = query
    }

    fun toggleIsSearchShowing() {
        _isSearchShowing.value = !_isSearchShowing.value
    }
}

We also apply the debounce transformation so that as the user is typing we don't constantly refresh the paging data. In this case, I set the debounce interval to 300 milliseconds.

All that's left to do is to hook the view model up to a view - an exercise left to the reader, though you can see my implementation here.

A full sample project is available on GitHub, which ties this approach together with dummy product data coming from a web request.

Implementing a cancellable asynchronous delay in JavaScript

Euan T — Tue, 19 Jul 2022 07:23:43 +0000

While working on a React project recently, I had a need to update some state periodically with data from an API retrieved using fetch(). Coming from a C# background, the way I would approach this problem there would be something like the following:

private async Task FetchDataContinuouslyAsync(CancellationToken cancellationToken)
{
    while (!cancellationToken.IsCancellationRequested)
    {
        await FetchDataAndSetStateAsync(cancellationToken);

        // now wait for 15 seconds before trying again
        await Task.Delay(15000, cancellationToken);
    }
}

Naturally, I went to approach the problem the same way in JavaScript. That's where I hit a snag though - there's no built in function analogous to Task.Delay().

This meant I had to come up with my own solution to the problem. Searching the internet yielded plenty of results where people were using setTimeout along with a Promise, but surprisingly few which supported early cancellation - and those that did tend to return a cancel function rather than observing a token for cancellation. As I was already using fetch() with an AbortController to cancel requests, I wanted to re-use that controller for cancellation.

Here's what I came up with:

/**
 * Return a promise that is resolved after a given delay, or after being cancelled.
 * 
 * @param  {number} duration The delay, in milliseconds.
 * @param  {AbortSignal|null} signal An optional AbortSignal to cancel the delay.
 * 
 * @return {Promise<void>} A promise that is either resolved after the delay, or rejected after the signal is cancelled.
 */
function asyncSleep(duration, signal) {
    function isAbortSignal(val) {
        return typeof val === 'object' && val.constructor.name === AbortSignal.name;
    }

    return new Promise(function (resolve, reject) {
        let timeoutHandle = null;

        function handleAbortEvent() {        
            if (timeoutHandle !== null) {
                clearTimeout(timeoutHandle);
            }

            reject(new DOMException('Sleep aborted', 'AbortError'));
        }

        if (signal !== null && isAbortSignal(signal)) {
            if (signal.aborted) {
                handleAbortEvent();
            }

            signal.addEventListener('abort', handleAbortEvent, {once: true});
        }

        timeoutHandle = setTimeout(function () {
            if (signal !== null && isAbortSignal(signal)) {
                signal.removeEventListener('abort', handleAbortEvent);
            }

            resolve();
        }, duration);
    });
}

This function takes a delay in milliseconds as its first parameter, and an optional AbortSignal as its second parameter. It returns a Promise<void> which will resolve after the specified delay, or be rejected with an AbortError if cancellation is requested.

In the context of a React project, this can be used like the following within a useEffect hook:

useEffect(() => {
    const ac = new AbortController();

    async function fetchDataContinuously(abortController) {
        while (!abortController.signal.aborted) {
            try {
                await getData(abortController.signal);

                await asyncSleep(refreshInterval, abortController.signal);
            } catch (e) {
                if (e.name === 'AbortError') {
                    break;
                }

                console.error('Error continuously refreshing', e);
            }
        }
    }

    fetchDataContinuously(ac).catch(console.error);

    return () => {
        ac.abort();
    };
}, []);

Of course, this could also be used with a traditional class based React component by simply aborting the AbortController in componentWillUnmount as well.

OpenVPN Server: Assigning Static IP Addresses Via A Script

Euan T — Wed, 29 Jun 2022 19:48:50 +0000

I recently had to set up an OpenVPN server where each client needed to be assigned a static IP address. The IP addresses were stored in a database table, alongside the client’s name and some other data. Rather than using static configuration via a client specific configuration file using the client-config-dir directive, I went looking for a way to do it dynamically. I eventually settled on using a client-connect script to assign the IP straight from the database, though found the documentation rather lacking.

I have a database table called clients which is used for a couple of other tasks and essentially has the following structure:

CREATE TABLE clients (
  id SERIAL PRIMARY KEY,
  name VARCHAR(255) NOT NULL,
  ip inet NOT NULL,
  CONSTRAINT clients_name_unique UNIQUE (name),
  CONSTRAINT clients_ip_unique UNIQUE (ip)
);

When a client connects, I want to look up their IP address based upon their X509 Common Name and push it to the client.

When using the client-config-dir option, this would be done by creating a file within the configured directory named the same as the Common Name for the client and using ifconfig-push IP_ADDRESS SUBNET_MASK. This is a very well documented approach, including being mentioned in the comments within the main server config file. The problem is that maintaining these files adds an additional chore when I already maintain the database table.

An alternative approach is to use one of the many possible scripts that OpenVPN Server can be configured to run when handling various events - in particular the client-connect script.

While exploring options, I found several questions on the StackExchange network from people looking for example scripts, and I eventually found the Reference manual for OpenVPN 2.4 and more specifically the Scripting and environment variables section. While this gets you part of the way there by explaining which environment variables are available for each type of script, it doesn’t do much to explain how you can actually influence the configuration of a connection itself.

It turns out that the OpenVPN server actually passes an argument to the configured script which is a path to a file that represents configuration for the client. So the flow basically ends up being:

Get the common_name environment variable if it is set.
Look up the IP address for the client using the value of the common_name.
If an IP address is found, open the file specified in the first argument to the program and write the text ifconfig-push IP_ADDRESS SUBNET_MASK to the file.

There are a couple of notes to this approach though:

If the script exits with any non-zero exit code, the client’s connection attempt will fail.
- The failure results in a hard crash when running OpenVPN Connect v3 on Windows in service daemon mode, and the service recovery options in Windows do not recover the service. Due to this, my script always returns exit code 0, even in case of error - this is unintuitive, but necessary.
The maximum length of a configuration value in the OpenVPN server configuration file is 256 characters, so passing a Data Source Name on the command line is a no-go. In this case, it’s best to write a wrapper script to call your actual script.
Any output from the script to stderr/stdout will be written straight out by the OpenVPN server - this means if you are on a Linux server with systemd running your OpenVPN server, logs from your script can be found in journald!

The path I ended up taking was a relatively simple piece of Go, with a bash script to call it. The bash script is saved to /usr/local/bin/resolve-vpn-ip.sh and made executable:

#!/usr/bin/env bash
set -Eeuo pipefail

DSN="postgresql://..."

/usr/local/bin/resolve-vpn-ip -db="$DSN" "$@"

The actual /usr/local/bin/resolve-vpn-ip program is a Go binary, as I re-use some common code from other systems also talking to this same database - it basically boils down to the following:

package main

import (
    "context"
    "flag"
    "fmt"
    "log"
    "net"
    "os"
    "os/signal"
    "syscall"

    "github.com/jackc/pgx/v4"
)

func main() {
    log.SetFlags(log.LstdFlags)

    flags := flag.NewFlagSet("resolve-vpn-ip", flag.ExitOnError)
    dsnFlag := flags.String("db", "", "the database connection string to use to connect to the database")

    if err := flags.Parse(os.Args[1:]); err != nil {
        flags.PrintDefaults()

        os.Exit(0)
    }

    remainingArgs := flags.Args()

    if len(remainingArgs) < 1 {
        log.Println("no arguments after flags, OpenVPN did not pass in a configuration file")

        os.Exit(0)
    }

    commonName, found := os.LookupEnv("common_name")

    if !found || commonName == "" {
        log.Println("common_name environment variable is not set or is empty")

        os.Exit(0)
    }

    log.Printf("getting IP address for client: %s\n", commonName)

    ctx, _ := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)

    ip, err := getIpForClient(ctx, *dsnFlag, commonName)

    if err != nil {
        log.Printf("error getting IP address for client: %s; error: %s\n", commonName, err)

        os.Exit(0)
    }

    f, err := os.OpenFile(remainingArgs[0], os.O_APPEND|os.O_WRONLY|os.O_CREATE, os.ModePerm)

    if err != nil {
        log.Printf(
            "failed to open config file for client: %s; file: %s; error: %s\n",
            commonName,
            remainingArgs[0],
            err,
        )

        os.Exit(0)
    }

    defer func(toClose *os.File) {
        _ = toClose.Close()
    }(f)

    if _, err = fmt.Fprintf(f, "ifconfig-push %s 255.255.255.0\n", ip.String()); err != nil {
        log.Printf(
            "failed to write to config file for client: %s; file: %s; error: %s\n",
            commonName,
            remainingArgs[0],
            err,
        )

        os.Exit(0)
    }
}

func getIpForClient(ctx context.Context, dsn, commonName string) (*net.IP, error) {
    conn, err := pgx.Connect(ctx, dsn)

    if err != nil {
        return nil, err
    }

    defer func(c *pgx.Conn) {
        _ = conn.Close(ctx)
    }(conn)

    var ip net.IP

    err = conn.QueryRow(ctx, `SELECT ip FROM clients WHERE name = $1;`, commonName).Scan(&ip)

    switch err {
    case nil:
        return &ip, nil
    case pgx.ErrNoRows:
        return nil, fmt.Errorf("no rows found for client: %s", commonName)
    default:
        return nil, fmt.Errorf("error getting ip for client: %s; error: %w", commonName, err)
    }
}

In the OpenVPN server config file I can then add something like the following to configure the server to execute my script:

client-connect "/usr/local/bin/resolve-vpn-ip.sh"

And I can tail my logs at any time via journald by running a command like journalctl -f /usr/local/bin/resolve-vpn-ip.

Using PHP's Typed Class Properties with PDO

Euan T — Fri, 03 Jun 2022 21:20:31 +0000

Ever since PHP introduced typed class properties in PHP 7.4, I've been making a lot of use of them. One of the places I find them most useful is when working with data models, for example when querying a database. PHP's PDO supports fetching result sets as objects, which when used in conjunction with the PostgreSQL driver or the MySQLnd driver can make use of typed properties.

After trying many different ORMs over the years such as Eloquent and Doctrine, I've eventually settled on using raw SQL for most smaller projects I work on. For more complex queries, I tend to reach for a query builder rather than a fully fledged ORM - generally this is the query system from Atlas PHP which I use in conjunction with the extended PDO library from Atlas too. I find this gives me the right degree of control and ease of development that lets me focus on the actual problem at hand rather than fighting with a rigid ORM's ways of doing things.

There are a few things to be aware of when working with typed properties alongside PDO though, for example:

Only some basic types are supported, such as int, bool, string. Nullability can also be handled, with type annotations such as ?int, ?bool, ?string.
PDO assigns class properties before running class constructors, so any logic in class constructors must be aware of properties that may already be set. However, PDO's methods related to fetching results as objects may pass arguments to this constructor.

There is an approach I've settled on that works quite well for me, which I wanted to share. A quick example:

<?php

declare(strict_types=1);

final class Post
{
    public int $id;

    public string $title;

    public string $content;

    public int $authorId;

    public bool $isDraft;

    public \DateTimeInterface $createdAt;

    public \DateTimeInterface $updatedAt;

    public function __set(string $name, $value): void
    {
        switch ($name)
        {
            case 'created_at':
                $this->createdAt = new \DateTimeImmutable($value);
                break;
            case 'updated_at':
                $this->updatedAt = new \DateTimeImmutable($value);
                break;
        }
    }
}

We handle types that aren't natively supported here using PHP's magic __set method, which is called with the name of the column from the database and its value. We then simply switch on the name of the column and do whatever we need to in order to create the actual type. In this case, we're handling date types but the same logic could be expanded to other types such as JSON.

There is usually also a repository class for each entity, which handles the actual query and marshalling results which looks a little like the following:

<?php

declare(strict_types=1);

final class PostRepository
{
    private \PDO $pdo;

    public function __construct(\PDO $pdo)
    {
        $this->pdo = $pdo;
    }

    public function allPosts(): \Generator
    {
        $query = <<<SQL
SELECT id, 
       title, 
       content, 
       author_id AS "authorId", 
       is_draft AS "isDraft", 
       created_at, 
       updated_at
FROM posts
ORDER BY created_at DESC;
SQL;

        $stmt = $this->pdo->query($query);

        while (($row = $stmt->fetchObject(Post::class)) !== false) {
            yield $row;
        }
    }

    public function find(int $id): ?Post
    {
        $query = <<<SQL
SELECT id, 
       title, 
       content, 
       author_id AS "authorId", 
       is_draft AS "isDraft", 
       created_at, 
       updated_at
FROM posts
WHERE id = :id;
SQL;

        $stmt = $this->pdo->prepare($query);

        $stmt->execute(
            [
                'id' => $id,
            ]
        );

        $result = $stmt->fetchObject(Post::class);

        if (false === $result) {
            return null;
        }

        return $result;
    }
}

Of note here is that we make use of a Generator when fetching multiple results, allowing us to choose at the call site whether we need to collect all the results instantly or can do so lazily when we (for example) build some HTML or a JSON result.

We also have to do some simple handling for the fact that in the case of no results being found, PDO returns false. As I like to work with strict types, my functions are declared as only returning a single type such as a nullable Post instance.

Usually there would also be a repository interface rather than just an implementation, so that we can make use of the decorator design pattern to add things such as caching where needed.

I've put together a repository on GitHub which showcases this approach along with a basic test to show it in action.

I've been using this approach for over a year, with quite a lot of success. Having raw SQL queries in our repository files makes it easy to troubleshoot queries directly against a database backup as well as being able to share queries between PHP projects and projects in other languages such as C#.

Troubleshooting Firebase Cloud Messaging on Android

Euan T — Sat, 09 Apr 2022 18:34:00 +0000

I’ve spent the first part of this year at work writing and testing a new Android application, and during the course of this I’ve stumbled across some devices which seem to have recurring issues with conenctivity to Google’s Firebase Cloud Messaging service. This issue manifests with the device no longer receiving notifications after being left to idle for a few hours. In this post, I’ll go over some troubleshooting steps that I’ve found.

Check Firebase Service Status

The first step to take is to double check that the Firebase Cloud Messaging service is not having any operational issues. This can be checked on the Firebase Status Dashboard - simply ensure that the Cloud Messaging service has a green tick next to it. There is also a handy RSS feed you can subscribe to for incidents, though there doesn’t seem to be any way to filter this to only report services you are actually using.

Check Your Device’s Connectivity to FCM

The next step is to check your device’s connectivity to Firebase Cloud Messaging itself.

This can be done from the Phone app, though it should be noted that your device may have a custom phone app from your manufacturer (this is definitely true for Samsung devices) - if this is the case, you need to install the Phone by Google app from the Google Play Store.

The steps to take are as follows:

Open the Phone app.
Open the dialler.
Dial the following: *#*#426#*#*.
The FCM Diagnostics screen should open as soon as you finish typing, without having to hit the call button - if it doesn’t appear, hit the call button. If nothing happens, you might be using a custom phone app from your device vendor (looking at you again Samsung…) in which case you need to install the Phone by Google app as explained above.

The screen should look something like the below screenshot:

There should be a line that says something like Server: Connected. If it says Not connected or something, then there should also be some information about the last close code and when the connection was lost.

You can also press the Events button to see a live trail of events relating to the connection, including incoming messages.

To see further information about the status, click on the menu button at the top right of the screen and check the Advanced view checkbox.

If the status shows FCM as disconnected and it doesn’t reconnect, you should check your firewall - see Google’s documentation on FCM ports and your firewall.

Unfortunately, I do still have a device which disconnects and steadfastly refuses to reconnect until it’s rebooted. Further troubleshooting is needed!

Setting Up A Jenkins Server On NetBSD

Euan T — Sat, 03 Oct 2020 09:38:05 +0000

I’ve been working on improving Nim’s BSD support for a little while now, first by adding continuous integration running on FreeBSD and then on OpenBSD. The next step was obvious: running CI on NetBSD. Unfortunately, I couldn’t find any CI services currently offering NetBSD as an option, so off I went down the rabbit hole of setting up a Jenkins install myself on NetBSD. In this post, I’ll run through the process of doing so, as it’s not as obvious as it may be on other platforms.

The first step is obvious: get NetBSD installed. I wanted to run it on a VPS rather than dedicating some local hardware to the cause, so my first step was actually finding a host that supported either installing NetBSD from a pre-built image or installing from an ISO. Fortunately, BuyVM KVM slices support custom ISOs and have a NetBSD 9.0 ISO readily available. I installed a clean copy of NetBSD 9.0 using the standard installation procedure.

During the install, I went for a full installation rather than a minimal install. I also took the chance to use some of the final optional steps such as setting up networking, setting up binary package installation, setting up pkgsrc, adding a user, etc. It’s worth doing these at install time just to save some effort at a later date.

Initial set up

There are a few initial set up steps I like to do on any system before we even get to installing Jenkins or anything else. These are mostly for convenience, and are entirely up to you.

The first steps should be ran in a root shell:

Install some standard packages that are useful:

pkgin install sudo nano htop git fish mozilla-rootcerts

Set up Mozilla’s root cetificate bundle, which is used for SSL certificate verification:

mozilla-rootcerts install

Configure sudo to allow members of the wheel group to execute commands:

visudo
# find the line `# %wheel ALL=(ALL) ALL` and un-comment it

Now that I had sudo set up I dropped back to my standard user’s shell, which was added to the wheel group during install, and ran some further steps:

Configure fish with some standard variables and functions:

mkdir -p $HOME/.config/fish
echo 'set fish_greeting ""

set -gx EDITOR (which nano)
set -gx VISUAL $EDITOR

function sudo!
 set -l cmd $history[1]
 echo "sudo $cmd" 1>&2
 eval sudo $cmd
end' > $HOME/.config/fish/config.fish

Set my shell to fish:

chsh -s $(which fish)

Set up my SSH public key:

mkdir -p $HOME/.ssh
chmod 0700 $HOME/.ssh
echo 'my-ssh-public-key' > $HOME/.ssh/authorized_keys
chmod 0600 $HOME/.ssh/authorized_keys

Configure SSH to deny root login, disallow password authentication, disable X11 forwarding and a few other tweaks:

sudo sed -i 's/^LoginGraceTime .*/LoginGraceTime 60/g' /etc/ssh/sshd_config
sudo sed -i 's/^#PermitRootLogin prohibit-password/PermitRootLogin no/g' /etc/ssh/sshd_config
sudo sed -i 's/^#MaxAuthTries 6/MaxAuthTries 3/g' /etc/ssh/sshd_config
sudo sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config
sudo sed -i 's/^#ChallengeResponseAuthentication yes/ChallengeResponseAuthentication no/g' /etc/ssh/sshd_config
sudo sed -i 's/^#X11Forwarding no/X11Forwarding no/g' /etc/ssh/sshd_config
sudo sed -i 's/^#UseDNS no/UseDNS no/g' /etc/ssh/sshd_config
sudo service sshd restart

Configure some basic firewall rules for NPF to block all incoming traffic except SSH, HTTP and HTTPS. Note that the network interface here may be different:

sudo touch /etc/npf_blacklist
echo '# NOTE: change the `wm0` to the name of your external network interface
$public_if = ifaddrs(wm0)

# create 2 tables - one for blacklisted IPs, the other for suspicious traffic
table <blacklist> type ipset file "/etc/npf_blacklist"
table <suspicious> type lpm

# create a variable for the TCP services we wish to allow
$tcp_services = { http, https }

alg "icmp"

procedure "log" {
 log: npflog0
}

procedure "normalize" {
 normalize: "random-id", "min-ttl" 64, "max-mss" 1432, "no-df"
}

group default {
 # pass everything on loopback
 pass final on lo0 all

 # block blacklisted IPs
 block in final from <blacklist>

 # block suspicious IPs
 block in final from <suspicious>

 # allow all outgoing
 pass stateful out final all

 # allow ICMP
 pass in final proto icmp all

 # allow and log ssh
 pass stateful in final proto tcp from any to $public_if port ssh apply "log"

 # allow incoming TCP
 pass stateful in final proto tcp to $public_if port $tcp_services apply "normalize"

 # reject everything else
 block return in final all apply "log"
}' | sudo tee /etc/npf.conf
echo 'npf=YES' | sudo tee -a /etc/rc.conf
sudo /etc/rc.d/npf reload
sudo /etc/rc.d/npf start

Install and configure Nginx

We’re going to use Nginx as a reverse proxy to Jenkins, so let’s install that. Luckily, there’s a relatively up to date binary package available:

sudo pkgin install nginx

Now let’s configure Nginx to start at boot:

# copy the RC script into place
sudo cp /usr/pkg/share/examples/rc.d/nginx /etc/rc.d/nginx
echo 'nginx=YES' | sudo tee -a /etc/rc.conf
echo '/var/log/nginx/access.log nginx:nginx 640 7 * 24 Z /var/run/nginx.pid SIGUSR1
/var/log/nginx/error.log nginx:nginx 640 7 * 24 Z /var/run/nginx.pid SIGUSR1' | sudo tee -a /etc/newsyslog.conf
sudo /etc/rc.d/syslogd restart

Then let’s do some Nginx configuration to point to where Jenkins will eventually be running:

# take a backup of the existing configuration
sudo tar -czvf /usr/pkg/etc/nginx.tar.gz /usr/pkg/etc/nginx
sudo mkdir -p /usr/pkg/etc/nginx/conf.d
echo 'user nginx nginx;

worker_processes auto;

worker_rlimit_nofile 8192;

events {
 worker_connections 8000;
}

error_log /var/log/nginx/error.log warn;

pid /var/run/nginx.pid;

http {
 server_tokens off;

 include mime.types;

 default_type application/octet-stream;

 charset utf-8;

 charset_types
 text/css
 text/plain
 text/vnd.wap.wml
 text/javascript
 text/markdown
 text/calendar
 text/x-component
 text/vcard
 text/cache-manifest
 text/vtt
 application/json
 application/manifest+json;

 log_format main '\''$remote_addr - $remote_user [$time_local] "$request" '\''
 '\''$status $body_bytes_sent "$http_referer" '\''
 '\''"$http_user_agent" "$http_x_forwarded_for"'\'';

 access_log /var/log/nginx/access.log main;

 keepalive_timeout 20s;

 sendfile on;

 tcp_nopush on;

 gzip on;

 gzip_comp_level 5;

 gzip_min_length 256;

 gzip_proxied any;

 gzip_vary on;

 gzip_types
 application/atom+xml
 application/geo+json
 application/javascript
 application/x-javascript
 application/json
 application/ld+json
 application/manifest+json
 application/rdf+xml
 application/rss+xml
 application/vnd.ms-fontobject
 application/wasm
 application/x-web-app-manifest+json
 application/xhtml+xml
 application/xml
 font/eot
 font/otf
 font/ttf
 image/bmp
 image/svg+xml
 text/cache-manifest
 text/calendar
 text/css
 text/javascript
 text/markdown
 text/plain
 text/xml
 text/vcard
 text/vnd.rim.location.xloc
 text/vtt
 text/x-component
 text/x-cross-domain-policy;

 include conf.d/*.conf;
}' | sudo tee /usr/pkg/etc/nginx/nginx.conf
echo 'server {
 listen [::]:80 default_server;
 listen 80 default_server;

 server_name _;

 return 444;
}' | sudo tee /usr/pkg/etc/nginx/conf.d/no-default.conf
echo 'server {
 listen [::]:80;
 listen 80;

 # change this to your actual domain name
 server_name jenkins.euantorano.co.uk;

 location / {
 proxy_set_header Host $host:$server_port;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;

 proxy_pass http://127.0.0.1:8080;
 proxy_read_timeout 90s;

 # change this to your actual domain name
 proxy_redirect http://127.0.0.1:8080 http://jenkins.euantorano.co.uk;

 proxy_http_version 1.1;
 proxy_request_buffering off;

 }
}' | sudo tee /usr/pkg/etc/nginx/conf.d/jenkins.conf

And now let’s start nginx:

sudo /etc/rc.d/nginx start

Install Java

Jenkins uses Java, which we must install. We’ll use the binary package for OpenJDK 8, as OpenJDK 11 on NetBSD currently (at the time of writing) has a bug related to network connections.

sudo pkgin install openjdk8

Install Jenkins

Jenkins is available as a package on NetBSD, but unfortunately the version available is well out of date as of the time of writing (the package version 2.73, while the current version is version 2.259). As such, we’ll download the current version from the Jenkins website and use that instead:

sudo mkdir -p /usr/lib/jenkins
sudo mkdir -p /var/lib/jenkins
sudo mkdir -p /var/log/jenkins
sudo chown jenkin:jenkins /var/log/jenkins
sudo ftp -o /usr/lib/jenkins/jenkins.war https://get.jenkins.io/war/latest/jenkins.war
# create a group for jenkins to run under
sudo groupadd jenkins
# create a user for jenkins to run under
sudo useradd -s /sbin/nologin -g jenkins -m -d /home/jenkins jenkins
# Create an RC script for Jenkins
echo '#!/bin/sh

# PROVIDE: jenkins
# REQUIRE: DAEMON

. /etc/rc.subr

JENKINS_USER=jenkins
JENKINS_GROUP=jenkins

JENKINS_HOME=/home/jenkins
export JENKINS_HOME

name="jenkins"
rcvar=$name
pidfile="/var/run/jenkins.pid"
start_cmd="jenkins_start"
stop_cmd="jenkins_stop"
status_cmd="jenkins_status"
version_cmd="jenkins_version"
extra_commands="status version"

jenkins_start()
{
 su -m jenkins:jenkins -c '\''nohup \
 /usr/pkg/java/openjdk8/bin/java -jar /usr/lib/jenkins/jenkins.war \
 --httpPort=8080 --httpListenAddress=127.0.0.1 \
 </dev/null >> /var/log/jenkins/jenkins.log 2>&1 &'\''
 pid=$!
 echo "$pid" > $pidfile
}

jenkins_stop()
{
 if [! -f "$pidfile"]; then
 echo "Jenkins is not running"
 exit 1
 fi

 pid=$(cat "$pidfile")
 kill -15 "$pid"
 rm "$pidfile"
}

jenkins_status()
{
 if [! -f "$pidfile"]; then
 echo "Jenkins is not running"
 exit 1
 fi

 pid=$(cat "$pidfile")
 kill -0 "$pid"
 if [$? -eq 0]; then
 echo "Jenkins is running (PID $pid)"
 exit 0
 else
 echo "Jenkins is not running"
 rm "$pidfile"
 exit 1
 fi
}

jenkins_version()
{
 /usr/pkg/java/openjdk8/bin/java -jar /usr/lib/jenkins/jenkins.war --version
}

load_rc_config "$name"
run_rc_command "$1"' | sudo tee /etc/rc.d/jenkins
sudo chmod +x /etc/rc.d/jenkins

And now let’s start Jenkins:

sudo /etc/rc.d/jenkins start

Finishing up

Jenkins is now running behind Nginx as a reverse proxy, but there are still a few tasks to do, including running through the Jenkins install web UI and setitng up build jobs. It’s probably a good idea to set up HTTPS too. I’m afraid these are left to the reader for the time being!

First is my first time really using NetBSD, as I usually use either FreeBSD or OpenBSD, so if there's anything that can be done in a more concise or better way please let me know!

Parsing Data Packets In .NET

Euan T — Sat, 26 Sep 2020 20:47:38 +0000

One of the most common tasks that I come up against when using .NET at work is to parse data received via some transport mechanism into a form that can be processed and further handled. Over the years, my approach has changed along with some of the newer APIs that have become available within .NET. In this post, I want to briefly look at the approach I currently take, making use of the <Span<T> type.

For those unaware, Span<T> is a value type that represents a contiguous array of memory. For those familiar with other modern languages such as Go or Rust, Span<T> is essentially the same as a slice. It can be thought of as a pointer and length pair, which can be easily iterated over and sliced into sub-sections.

There's a companion repository for this post on GitHub, which has all of the code along with some basic unit tests for the parsing logic.

Example: A simple binary based protocol

Let's start with a simple made up binary protocol. I wanted something simple to illustrate my approach, so let's define a packet type that consists of two parts:

Command — some kind of textual command that tells the application what to do with the packet.
Data — some kind of textual data for the packet.

On the wire, this will be encoded as follows:

A start byte, SOH (byte value 1), to indicate the start of the packet.
4 bytes for the length of the Command, in little endian byte order.
4 bytes for the length of he Data, in little endian byte order.
The Command bytes,
The Data bytes.

This can be represented as a structure like the following:

public readonly struct Packet
{
    public const byte Soh = 1;

    public readonly int CommandLength;

    public readonly int DataLength;

    public readonly string Command;

    public readonly string Data;

    public Packet(string command, string data)
    {
        CommandLength = Encoding.UTF8.GetByteCount(command);
        DataLength = Encoding.UTF8.GetByteCount(data);
        Command = command;
        Data = data;
    }

    public Packet(int commandLength, int dataLength, string command, string data)
    {
        CommandLength = commandLength;
        DataLength = dataLength;
        Command = command;
        Data = data;
    }
}

Writing a simple array based parser

We'll write a parser function that will take an array, offset and length.

I generally write state-less parsers, which tend to have a method signature like the following:

public static ParseResult ParsePacket(
    byte[] buff,
    int offset,
    int count,
    out Packet? packet,
    out string? error,
    out int bytesConsumed
);

Where ParseResult is an enum of three possible values:

public enum ParseResult
{
    Partial,
    Complete,
    Error
}

This keeps the buffering logic within the consumer of the parser and makes testing much easier.

The array based parse function

The parse function will start off by looking for the starting SOH byte within the buffer. If we don't find an SOH byte, then we know there isn't a packet within the buffer:

public static ParseResult ParsePacket(
    byte[] buff,
    int offset,
    int count,
    out Packet? packet,
    out string? error,
    out int bytesConsumed
)
{
    bytesConsumed = 0;

    // packets start with a SOH byte - if there is no such byte in the buffer, then we don't have a packet.
    int indexOfSoh = Array.IndexOf(buff, Packet.Soh, offset, length);

    if (indexOfSoh == -1)
    {
        bytesConsumed = length;

        packet = default;
        error = default;
        return ParseResult.Partial;
    }

We know that all packets have a fixed length header, which consists of the SOH byte and two 32-bit integer values (each of which take up 4 bytes), so the next step is to ensure that there is enough data for that header:

if (length < indexOfSoh + 9)
{
    // the packet header is two 32-bit integers encoded as little endian following a SOH byte - this means a packet must be at least 9 bytes long
    packet = default;
    return ParseResult.Partial;
}

Now we need to read those two 32-bit integer values. Remember, they're encoded in little endian byte order and we need to make sure that we handle that if the system we're parsing on isn't little endian:

if (!BitConverter.IsLittleEndian)
{
    // flip the bits as the system is not little endian
    Array.Reverse(buff, indexOfSoh + 1, sizeof(int));
}

int commandLength = BitConverter.ToInt32(buff, indexOfSoh + 1);

if (!BitConverter.IsLittleEndian)
{
    // flip the bits as the system is not little endian
    Array.Reverse(buff, indexOfSoh + 1 + sizeof(int), sizeof(int));
}

int dataLength = BitConverter.ToInt32(buff, indexOfSoh + 1 + sizeof(int));

Now that we have the length of the command and the length of the data in bytes, reading them is fairly trivial — we first ensure the buffer is long enough again, then we simply extract the strings:

if (length < indexOfSoh + 9 + commandLength + dataLength)
{
    // not enough data for the command and data
    packet = default;
    return ParseResult.Partial;
}

string command = Encoding.UTF8.GetString(buff, indexOfSoh + ((2 * sizeof(int)) + 1), commandLength);

string data = Encoding.UTF8.GetString(buff, indexOfSoh + ((2 * sizeof(int)) + 1) + commandLength, dataLength);

Then it's simply a matter of returning the right parse status and creating a Packet instance from the values:

packet = new Packet(commandLength, dataLength, command, data);
bytesConsumed = indexOfSoh + ((2 * sizeof(int)) + 1) + commandLength + dataLength;
return ParseResult.Complete;

Note that we set bytesConsumed to the total number of bytes that make up the packet, including any data that occurred before the SOH byte — this is so the consumer knows it can remove that data from its buffer.

As you can see, there's quite a lot of book-keeping of lengths and offsets. This can be made more simple by creating an internal offset variable to work from, or we can use Span<T>.

The `Span<T>` based parse function

The span based solution looks very similar to the array based version, except the book-keeping is simplified by a long way, and we make use of some new APIs such as the System.Buffers.Binary.BinaryPrimitives helper functions to read out integer values.

Again, we'll start of by finding the SOH byte:

public static ParseResult ParseBinaryPacket(
    in ReadOnlySpan<byte> buff,
    out Packet? packet,
    out string? error,
    out int bytesConsumed
)
{
    bytesConsumed = 0;

    // packets start with a SOH byte - if there is no such byte in the buffer, then we don't have a packet.
    int indexOfSoh = buff.IndexOf(Packet.Soh);

    if (indexOfSoh == -1)
    {
        bytesConsumed = buff.Length;

        packet = default;
        return ParseResult.Partial;
    }

We now diverge slightly, however. Remember I said earlier that spans can be sliced into sub-sections? This means we can create a new span without copying the underlying data at all, which simply points to a new region of the buffer. In this case, we'll create a new span that points to the content after the SOH byte:

// slice past the soh, as we do not wish to include it
ReadOnlySpan<byte> b = buff[(indexOfSoh + 1)..];

We again check the length is long enough to hold the two 32-bit integer values, then we use the BinaryPrimitive APIs to read them before once again re-slicing the span:

if (b.Length < sizeof(int) * 2)
{
    // the packet header is two integers encoded as little endian - this means a packet must be at least 8 bytes long
    packet = default;
    return ParseResult.Partial;
}

int commandLength = BinaryPrimitives.ReadInt32LittleEndian(b);
int dataLength = BinaryPrimitives.ReadInt32LittleEndian(b[sizeof(int)..]);

b = b[(sizeof(int) * 2)..];

Now that we have the lengths of the Command and Data, it's a case of reading them:

if (b.Length < commandLength)
{
    // not enough data for the command
    packet = default;
    return ParseResult.Partial;
}

string command = Encoding.UTF8.GetString(b[..commandLength]);

b = b[commandLength..];

if (b.Length < dataLength)
{
    // not enough data for the data
    packet = default;
    return ParseResult.Partial;
}

string data = Encoding.UTF8.GetString(b[..dataLength]);

b = b[dataLength..];

This looks very similar to before, as Encoding.UTF8.GetString has an overload that will take a Span<byte>.

We finish up by creating a Packet instance and returning the parse status:

packet = new Packet(commandLength, dataLength, command, data);
error = default;
bytesConsumed = buff.Length - b.Length;
return ParseResult.Complete;

Comparing the two

The two approaches bear a lot of similarities, but I personally find the span based approach much easier to follow. It also ties in with some other APIs such as the MemoryPool quite nicely, as you can easily get a Span<T> from a Memory<T>.

Example: A simple text based protocol

This same approach applies to text based protocols as well. Taking the previous Packet type, let's change it so that instead of encoding the Command length and the Data length as raw bytes we encode them both as hexadecimal ASCII characters. As they're both 32-bit integers, they'll take up 8 bytes (the maximum value for a 32-bit integer in hex is 8 ASCII characters long).

Our parsing logic doesn't change much, except when reading the lengths. When using a byte array, the common approach I usually see is to use int.TryParse:

string commandLengthString = Encoding.UTF8.GetString(buff, indexOfSoh + 1, 8);
if (
    !int.TryParse(
        commandLengthString,
        NumberStyles.HexNumber,
        NumberFormatInfo.InvariantInfo,
        out int commandLength
    )
)
{
    bytesConsumed = indexOfSoh + 17;

    packet = default;
    error = "Failed to parse command length";
    return ParseResult.Error;
}

string dataLengthString = Encoding.UTF8.GetString(buff, indexOfSoh + 9, 8);
if (
    !int.TryParse(
        dataLengthString,
        NumberStyles.HexNumber,
        NumberFormatInfo.InvariantInfo,
        out int dataLength
    )
)
{
    bytesConsumed = indexOfSoh + 17;

    packet = default;
    error = "Failed to parse data length";
    return ParseResult.Error;
}

You might be tempted to do something similar when working with a Span<T>, but there is a much nicer API: System.Buffers.Text.Utf8Parser. This has lots of handy overloads for reading all sorts of types from a ReadOnlySpan<byte>, including reading hexadecimal:

if (!Utf8Parser.TryParse(b[..8], out int commandLength, out int consumed, 'X') || consumed != 8)
{
    b = b[16..];
    bytesConsumed = buff.Length - b.Length;

    packet = default;
    error = "Failed to parse command length";
    return ParseResult.Error;
}

b = b[8..];

if (!Utf8Parser.TryParse(b[..8], out int dataLength, out consumed, 'X') || consumed != 8)
{
    b = b[8..];
    bytesConsumed = buff.Length - b.Length;

    packet = default;
    error = "Failed to parse data length";
    return ParseResult.Error;
}

b = b[8..];

Specifying the standardFormat parameter as X means the format is uppercase hexadecimal.

There is also a handy System.Buffers.Text.Utf8Formatter type for formatting values into Span<T> types, which I'll likely look at in a later post.

Wrap-up

I hope this post serves as a good starting point for playing with spans, they certainly help me reason about what's actually going on much more clearly arrays do in the vast majority of cases. And being able to use a ReadOnlySpan to ensure I don't accidentally write into the data is always a nice safety net!

Validating strings at compile time in Zig

Euan T — Wed, 05 Aug 2020 21:39:36 +0000

The scenario is simple, and unfortunately quite common when interfacing with libraries designed for C: you have a library function which takes a single string parameter which can only contain a list of pre-defined values.

Take, for example, the pledge(2) system call from OpenBSD. This system call is quite simple, and takes two parameters — both of which are strings that may only contain a pre-defined list of promises.

Passing a string with an invalid promise value (such as a spelling mistake) will result in an error at runtime. Wouldn't it be nice if we could instead catch such an issue at compile time?

That's exactly what we're going to do, using the Zig programming language.

Note: this is extracted from a blog post on my site, which looks at some alternative approaches in a couple of other languages

What is Zig?

Zig is described as:

Zig is a general-purpose programming language and toolchain for maintaining robust, optimal, and reusable software.

It is a language with a C-like syntax that aims to replace C. It brings some useful tools to the table, including comprehensive support for running code at compile time.

It's this functionality that we will be using in this post.

Defining the problem

Our problem is going to take some queues from the pledge(2) system call, though we are going to simplify it a bit:

In this case, the system call will only have 1 string parameter.
In this case, there will be drastically fewer promises available:
- stdio
- rpath
- wpath
- path
- dpath
Our function signature should look something like the following:
```
int pledge(const char *promiseList);
```
The promiseList parameter should be a string containing only the promises set of available promises, separated by a single space.

Writing some Zig

Let's write some code!

Creating an enum to represent the available promises

As we have a limited set of pre-defined promises, the easiest way to represent these values is as an enum.

Let's start off by defining one, named Promises:

const Promises = enum {
    stdio,
    rpath,
    wpath,
    path,
    dpath,
};

This should be quite familiar to most with some experience using c-like languages such as C# or Java.

Writing a function to validate a string of promises

We should probably create a function that will take a string of promises separated by spaces and ensure all the values are valid, well-known values.

Luckily, Zig's standard library has some functions to help with this:

std.mem.tokenize — to split a string by a particular delimiter.
std.meta.stringToEnum — to parse a string value into an enum member.

We can use these functions to iterate through all sub-values in a promises string and try to parse them as members of the Promises enum:

const std = @import("std");
const tokenize = std.mem.tokenize;
const stringToEnum = std.meta.stringToEnum;

fn validatePromises(promises: []const u8) bool {
    var splitter = tokenize(promises, " ");

    while (splitter.next()) |s| {
        _ = stringToEnum(Promises, s) orelse return false;
    }

    return true;
}

If any of the values within the promises string fail to parse, we return false. However, if they all parse successfully, we return true.

Calling `validatePromises` at compile time

This is where the magic happens. We now have a way to validate a promises string, but we need to call it at compile time.

Luckily, Zig has an awesome comptime concept. We can easily declare that certain arguments passed to functions should be known at compile time, and run whole blocks of code at compile time using those parameters. For example:

const warn = std.debug.warn;

fn promise(comptime promises: []const u8) void {
    comptime {
        if (!validatePromises(promises)) {
            @compileError("invalid promises: " ++ promises);
        }
    }

    warn("valid promises: {}", .{promises});
}

This code will call the validatePromises function at compile time with the passed in promises string. If the validation fails, a compiler error is emitted, including the invalid promises.

This results in a build failure like the following:

./src/main.zig:28:13: error: invalid promises: stdio foobar
            @compileError("invalid promises: " ++ promises);
            ^
./src/main.zig:37:16: note: called from here
        promise("stdio foobar");
               ^
./src/main.zig:35:29: note: called from here
pub fn main() anyerror!void {
                            ^
app...The following command exited with error code 1:

...

Putting it all together

I've only scratched the surface of what's possible with Zig's comptime facilities here, but I hope I've given you some inspiration to give it a try.

I'd love to see what other ideas you think up with this kind of approach. For now, I've got a simple repository available with the working source code for this Zig implementation, along with a Nim version and a D version.