DEV Community: Joshua Matthews

Named Params in C

Joshua Matthews — Tue, 14 Oct 2025 18:05:43 +0000

Table of Contents:

Introduction
Implementation in C
Using our Implementation
Conclusion

Introduction

In a language like python you have the concept of **kwargs as a function parameter. This special double * syntax allows you to pass in named arguments captured as a dictionary, like the example below:

def print_item(**kwargs):
    # kwargs is a dictionary
    print(f"Item name: {kwargs['name']}")
    if "kind" in kwargs:
        print(f"Kind: {kwargs['kind']}")
    print(f"Quantity: {kwargs['quantity']}")
    print(f"Liters: {kwargs['liters']}")

print_item(name="Milk", quantity=2, liters=1)
# output:
# Item name: Milk
# Quantity: 2
# Liters: 1

This allows for a convenient way to pass arguments by their name in any order you'd like and have optional parameters without complicating the function signature.

What if we could do this in C? (spoiler: there is a way)

Implementation in C

Through the magic of the preprocessor we can accomplish this effect, optional parameters and all.

First lets define our item structure.

struct item_t {
    const char *name;
    const char *kind;
    int quantity;
    float liters;
};

Next we'll define our print function like you would normally in C.

// name it with `__t` for the concrete function.
void print_item__t(struct item_t item) {
    printf("Item name: %s\n", item.name);
    if (item.kind != NULL) {
        printf("Kind: %s\n", item.kind);
    }
    printf("Quantity: %d\n", item.quantity);
    printf("Liters: %f\n", item.liters);
}

Now we can write the magically preprocessor part.

// name the macro what we want and use `...` to signify va_args
// Then call our function with a structure with default values
// and the va_args at the end to overwrite any values.
#define print_item(...)            \
    print_item__t((struct item_t){ \
        .name = "default",         \
        .kind = NULL,              \
        .quantity = 0,             \
        .liters = 0,               \
        __VA_ARGS__                \
    })

So lets break down this macro.

First we define the macro name and specify we are using va_args with ... as the parameter.

#define print_item(...)            \

Next we call the function with a structure of our item_t with default values.

    print_item__t((struct item_t){ \
        .name = "default",         \
        .kind = NULL,              \
        .quantity = 0,             \
        .liters = 0,               \

Last we pass in the __VA_ARGS__ values at the end of our default structure. This works because C allows the later defined properties to overwrite earlier ones.

        .liters = 0,               \
        __VA_ARGS__                \
    })

Here's how all of this would be structured in actual files.

item.h

#ifndef ITEM_H
#define ITEM_H

struct item_t {
    const char *name;
    const char *kind;
    int quantity;
    float liters;
};

void print_item__t(struct item_t item);

#define print_item(...)            \
    print_item__t((struct item_t){ \
        .name = "default",         \
        .kind = NULL,              \
        .quantity = 0,             \
        .liters = 0,               \
        __VA_ARGS__                \
    })

#endif

item.c

#include "item.h"
#include <stdio.h>

void print_item__t(struct item_t item) {
    printf("Item name: %s\n", item.name);
    if (item.kind != NULL) {
        printf("Kind: %s\n", item.kind);
    }
    printf("Quantity: %d\n", item.quantity);
    printf("Liters: %f\n", item.liters);
}

Using our Implementation

Now we can use our macro similar to the python example in the introduction.

#include "item.h"

int main(void) {
    print_item(.name = "Milk", .quantity = 2, .liters = 1);
    return 0;
}
// output:
// Item name: Milk
// Quantity: 2
// Liters: 1

Conclusion

Through preprocessor magic we can achieve a syntax convenience with calling functions that have struct parameters as options. This can be extremely useful for functions with lots of optional parameters and\or default values. However, preprocessor magic is something that is either loved or hated so you may want to consider if it's worth implementing in your project or not.

Automated Cleanup in C

Joshua Matthews — Wed, 18 Jun 2025 18:31:10 +0000

Table of Contents:

Intro
Documentation
Usage
- Errors and Branching
- Inside Loops
- Structures
- Deferring Operations
Macro Helpers
Conclusion

Intro

House-keeping in C can be tedious with manually freeing memory and providing cleanup code at every logical branch of your functions. However, compilers have added attributes that you can take advantage of to help with memory cleanup nowadays. I will step through and show case some of the powerful ways to use __attribute__((cleanup(<function>))) within your code.

Documentation

This feature is supported by Clang and GCC and their documentation provide small descriptions on how the feature is meant to be used. The quick overview is that this attribute allows you to register a callback function to a variable definition and gets called when the variable leaves the scope it was defined in. The basic structure is as follows:

static void free_char(char **p) {
    if (p == NULL) return;
    if (*p == NULL) return;
    free(*p);
    *p = NULL;
}

int main(void) {
    __attribute__((cleanup(free_char))) char *test = malloc((sizeof(char)*14));
    strncpy(test, "Hello, World!", 13);
    test[14] = '\0';
    printf("test string = \"%s\"\n", test);
    // free_char is called when this function exits
    return 0;
}

We start with defining our cleanup function free_char. It accepts a char ** because the cleanup function passes the local variable as a pointer to the function. Since our variable is a pointer, it gets passed in as a double pointer.

We then attach the attribute to the variable we want the cleanup function to happen to. This attribute can be in a few places and it will work.
All of these definitions are valid:

__attribute__((cleanup(free_char))) char *test = malloc(/* size */);
char __attribute__((cleanup(free_char))) *test = malloc(/* size */);
char *test __attribute__((cleanup(free_char))) = malloc(/* size */);

When defining your cleanup function you could technically get away with something like:

static void generic_cleanup(void **p);

But if you use a proper linter it'll throw warnings that the types don't match, so it's better to use explicit cleanup functions for your types.

Usage

With a basic overview of the feature understood, we'll cover a few more use cases.

Errors and Branching

The immediate use-case is clean up of allocated variables. This can be very advantageous for functions with lots of error handling or branching logic with multiple return statements.

static void free_char(char **p) {
    if (p == NULL) return;
    if (*p == NULL) return;
    free(*p);
    *p = NULL;
}

bool http_message_set_header(struct http_message_t *msg, const char *key,
                             char *value) {
  size_t key_len = strlen(key);
  __attribute__((cleanup(free_char))) char *normalized_key = normalized_cstr(key, key_len);
  // free previous value.
  char *out = NULL;
  http_message_get_header(msg, key, &out);
  if (out != NULL) {
    free(out);
  }
  // set new value, we copy the value to have ownership
  if (!hash_map_set(msg->headers, normalized_key,
                    str_dup(value, strlen(value)))) {
    fprintf(stderr, "failed to set header.\n");
    // normalize_key is freed here
    return false;
  }
  // normalize_key is freed here
  return true;
}

In the above example we attach a clean up function to the normalized_key variable because it's only a temporary variable that we always want to free by the end of the function.

Inside Loops

Another example with temporary variables is cleanup with variables inside loops.

void debug_print(struct item_t *arr, size_t n) {
    for (size_t i = 0; i < n; ++i) {
        __attribute__((cleanup(free_char))) char *tmp = item_detailed_str(arr[i]);
        if (tmp != NULL) {
            printf("%s\n", tmp);
        }
        // tmp is freed here
    }
}

This is a trivial example but can be quite useful to not deal with cleanup inside loops.

Structures

Cleaning up structures are made easy as well.

void item_free(struct item_t *item) {
    // free item internals
}

void item_operation(char *name, int quantity, char *label) {
    __attribute__((cleanup(item_free))) struct item_t tmp;
    if (!item_init(&tmp, name, quantity, label)) {
        // handle error
    }
    // performa operations
    // tmp is freed here.
}

Deferring Operations

The last most useful example is deferred actions. This can range from needing DB operations called by the end of a scope, freeing mutexes, etc.

void state_free_mutex(struct state_t **s){
    pthread_mutex_unlock(&((*s)->mutex));
}

void state_update_list(struct state_t *s, struct item_t *items, size_t item_len) {
    __attribute__((cleanup(state_free_mutex))) struct state_t* local = s;
    pthread_mutex_lock(&local->mutex);
    // operate on items array and state

    // state's mutex is unlocked
}

Macro Helpers

The use-cases showcase how powerful this attribute can be, but it does come with cumbersome syntax. However, we can alleviate this with simple macros.

The simplest approach is with the following macro:

#define DEFER(cb) __attribute__((cleanup(cb)))

And that's it! Now we can change the original example to look like this:

int main(void) {
    DEFER(free_char) char *test = malloc((sizeof(char)*14));
    strncpy(test, "Hello, World!", 13);
    test[14] = '\0';
    printf("test string = \"%s\"\n", test);
    // free_char is called when this function exits
    return 0;
}

That's much easier to read and type.

You can take this a step further, if you'd like, and add predefined simple macros for common types.

extern void free_char(char **p);

#define AUTO_C DEFER(free_char)

Which then allows the example to turn into this:

int main(void) {
    AUTO_C char *test = malloc((sizeof(char)*14));
    strncpy(test, "Hello, World!", 13);
    test[14] = '\0';
    printf("test string = \"%s\"\n", test);
    // free_char is called when this function exits
    return 0;
}

Conclusion

The cleanup attribute is a versatile and powerful tool to utilize within your projects. Though C can be a tedious language to work with, compiler features like these help make working with C less problematic and provide a more robust way of interacting with the language.

The 90% Makefile

Joshua Matthews — Mon, 05 May 2025 23:27:03 +0000

Contents:

Introduction
Project Setup
Makefile
Conclusion

Introduction

I've been doing a lot of side projects in the C language lately.
Which has lead me to write Makefiles for all the scenarios I encounter with each new project.

The more I modified the Makefiles I was using to satisfy each new project structure
has lead to a Makefile I barely need to change now.

!!
I know this Makefile won't solve all usecases but it tends to cover most of my needs.
!!

Project Setup

To start, I'll show what a typical project setup looks like for me.

├── LICENSE
├── Makefile
├── README.md
├── compile_flags.txt
├── install_deps.sh
├── run.sh
├── resources
│   └── <resource files>
├── bin
│   ├── lib.(so|a)
│   └── exe
├── obj
│   └── <object files>
├── deps
│   └── <third party deps>
└── src
    ├── lib
    └── main.c

I'll quickly give a highlight of everything, but the folders are the important part.

LICENSE the license.
Makefile The Makefile.
README.md The readme.
compile_flags.txt Clangd file for LSP configuration.
install_deps.sh Shell file to install dependencies.
run.sh Shell script to run the generated executable.
resources The resources folder.
bin The folder for the target executable or library.
obj The object files when compiling.
deps The third party dependencies.
src The source code.
src/lib The common code (like for a library).

Makefile

Now that you know what folder structure to expect we can look at the Makefile.
First I'll present the Makefile and then we will break it down.

# define our compiler
CC=gcc
# define our generic compiler flags
CFLAGS=-Wall -Wextra -std=c11
# define the paths to our third party libraries
LIBS=-L./deps/example_lib/build -lexample_lib.so -lm
# define the paths to our include directories
INCLUDES=-I./src -I./deps/example_lib/include

# define variables for our source files
# we use find to grab them
SOURCES=$(shell find ./src -name '*.c')

# define variable for our dependencies' Makefiles.
# we use find to grab only the top level Makefiles and also some convenient ignores.
DEPS=$(shell find ./deps -maxdepth 2 -name Makefile -printf '%h\n' | grep -v 'unittest' | grep -v '^.$$')

# define folder paths and names
OBJ=obj
BIN=bin
TARGET=main

# setup up conditional build flags
# if debug is set to 1, add debug specific flags
ifeq ($(DEBUG), 1)
    CFLAGS += -DDEBUG=1 -ggdb
endif
# Release specific flags
ifeq ($(RELEASE), 1)
    CFLAGS += -O2
endif
# if SHARED flag is set, we prepare variables for building a shared/static library.
# we change the SOURCES variable to point to only the common source files.
# we also rename the TARGET to our library name.
ifeq ($(SHARED), 1)
    SOURCES=$(shell find ./src/lib -name '*.c')
    TARGET=my_lib
endif

# This variable is for our object files.
# We take the files in SOURCES and rename them to end in .o
# Then we add our OBJ folder prefix to all files.
OBJECTS=$(addprefix $(OBJ)/,$(SOURCES:%.c=%.o))

# We setup our default job
# it will build dependencies first then our source files.
.PHONY: all
all: deps src

# Build the source files.
# Conditional change to building for an executable or libraries.
# We also create the output BIN directory if it doesn't exist.
# This job depends on the OBJECT files.
.PHONY: src
src: $(OBJECTS)
    @mkdir -p $(BIN)
ifeq ($(SHARED), 1)
    $(CC) -shared -fPIC -o $(BIN)/$(TARGET).so $^ $(LIBS)
    ar -rcs $(BIN)/$(TARGET).a $^
else
    $(CC) $(CFLAGS) $(LIBS) $^ -o $(BIN)/$(TARGET)
endif

# Compile all source files to object files
# This job executes because the `src` job depends on all the files in OBJECTS
# which has the `$(OBJ)/%.o` file signature.
$(OBJ)/%.o: %.c
    @mkdir -p $(dir $@)
    $(CC) -c -o $@ $< $(CFLAGS) $(INCLUDES)

# Job to clean out all object files and exe/libs.
.PHONY: clean
clean:
    @rm -rf $(OBJ)/* 2> /dev/null
    @rm -f $(BIN)/* 2> /dev/null

# Job to run `make clean` on all dependencies.
.PHONY: clean_deps
clean_deps:
    $(foreach dir, $(DEPS), $(shell cd $(dir) && $(MAKE) clean))

# Job to clean dependencies and our files.
.PHONY: clean_all
clean_all: clean clean_deps

# Job to run `make` on all of our dependencies.
# This only works if the dependencies' Makefile is at the top and implements a
# default job.
.PHONY: deps
deps:
    $(foreach dir, $(DEPS), $(shell cd $(dir) && $(MAKE)))

The Common Variables

At the top we have our common variables to define our compiler, compiler flags,
library paths, and includes paths.

Add or remove anything from these variables to fit your project's requirements.

# define our compiler
CC=gcc
# define our generic compiler flags
CFLAGS=-Wall -Wextra -std=c11
# define the paths to our third party libraries
LIBS=-L./deps/example_lib/build -lexample_lib.so -lm
# define the paths to our include directories
INCLUDES=-I./src -I./deps/example_lib/include

The Source Files

We expect all of our source files to be under the src folder. So we use
find to grab all of them. By default we assume we are compiling for an executable,
we can change this later with some conditional logic.

# define variables for our source files
# we use find to grab them
SOURCES=$(shell find ./src -name '*.c')

The Dependencies

We want to automate compiling our third party dependencies as well so we grab their
Makefiles at the top of their directories. We exclude any unittest directories.

This might need to change as well depending on what third party dependencies you use.

# define variable for our dependencies' Makefiles.
# we use find to grab only the top level Makefiles and also some convenient ignores.
DEPS=$(shell find ./deps -maxdepth 2 -name Makefile -printf '%h\n' | grep -v 'unittest')

Folders & Target

We always need a bin and obj folder. We default the target to an executable
name (this can change with conditional logic).

# define folder paths and names
OBJ=obj
BIN=bin
TARGET=main

Conditional Flags

We setup some conditional flags we can pass to make. We define DEBUG,
RELEASE, and SHARED already. The DEBUG and RELEASE modify the CFLAGS
to better align with their respective states.

The SHARED flag is independant from the other conditional flags because it changes
what source files we use and the TARGET name.

# setup up conditional build flags
# if debug is set to 1, add debug specific flags
ifeq ($(DEBUG), 1)
    CFLAGS += -DDEBUG=1 -ggdb
endif
# Release specific flags
ifeq ($(RELEASE), 1)
    CFLAGS += -O2
endif
# if SHARED flag is set, we prepare variables for building a shared/static library.
# we change the SOURCES variable to point to only the common source files.
# we also rename the TARGET to our library name.
ifeq ($(SHARED), 1)
    SOURCES=$(shell find ./src/lib -name '*.c')
    TARGET=my_lib
endif

Object Files

We setup our Object files next so we can construct them based on what the SOURCES
variable is conditionally set to.

# This variable is for our object files.
# We take the files in SOURCES and rename them to end in .o
# Then we add our OBJ folder prefix to all files.
OBJECTS=$(addprefix $(OBJ)/,$(SOURCES:%.c=%.o))

The Jobs

The rest of the Makefile are the jobs.

Default Job

The first one is the default job. It's an empty job but we use it to set
up the dependency tree for our Makefile.

We first depend on deps then src.

# We setup our default job
# it will build dependencies first then our source files.
.PHONY: all
all: deps src

Source Job

We setup our src job to have a dependency on the Object files we defined.

Inside the job we create the bin directory if it doesn't exist then we conditionally
execute code to build either libraries or an executable.

All the places you see $^ is Makefile magic to reference everything in the dependency
list (which are all the files in OBJECTS).

# Build the source files.
# Conditional change to building for an executable or libraries.
# We also create the output BIN directory if it doesn't exist.
# This job depends on the OBJECT files.
.PHONY: src
src: $(OBJECTS)
    @mkdir -p $(BIN)
ifeq ($(SHARED), 1)
    $(CC) -shared -fPIC -o $(BIN)/$(TARGET).so $^ $(LIBS)
    ar -rcs $(BIN)/$(TARGET).a $^
else
    $(CC) $(CFLAGS) $(LIBS) $^ -o $(BIN)/$(TARGET)
endif

Object Files Job

The object files job looks a little weird at first but the job name is a wildcard.
It matches anything with the signature $(OBJ)/%.o which is what all of our files
listed in the OBJECTS variable match.

The %.c dependency is some Makefile magic to find the source file with the same path
except the $(OBJ)/ part.

There are 2 other Makefile magic symbols that need some explaining in here.
First is the $@ symbol to reference the left side of the job signature
(the $(OBJ)/%.o side).
Second is the $< symbol to reference the first item in the dependency side of the
job signature (the %.c side).

# Compile all source files to object files
# This job executes because the `src` job depends on all the files in OBJECTS
# which has the `$(OBJ)/%.o` file signature.
$(OBJ)/%.o: %.c
    @mkdir -p $(dir $@)
    $(CC) -c -o $@ $< $(CFLAGS) $(INCLUDES)

Clean Jobs

The next section are all the jobs related to clean up.

The first job is to clean our project. You'll notice we pipe the errors to
/dev/null this is because we get errors if the folders are empty when cleaning.

The second job is to run make clean on all of the dependencies. We need to call
make as $(MAKE) for it to execute properly.

The last job is to do both of the cleanup jobs together.

# Job to clean out all object files and exe/libs.
.PHONY: clean
clean:
    @rm -rf $(OBJ)/* 2> /dev/null
    @rm -f $(BIN)/* 2> /dev/null

# Job to run `make clean` on all dependencies.
.PHONY: clean_deps
clean_deps:
    $(foreach dir, $(DEPS), $(shell cd $(dir) && $(MAKE) clean))

# Job to clean dependencies and our files.
.PHONY: clean_all
clean_all: clean clean_deps

Dependency Job

The last job in the Makefile is to compile our third party dependencies.
This command iterates over the Makefiles we captured in the DEPS variable
and runs make. This assumes the third party dependencies have a default job setup
and it's the one you want to run.

This section may need to change depending on your projects needs.

# Job to run `make` on all of our dependencies.
# This only works if the dependencies' Makefile is at the top and implements a
# default job.
.PHONY: deps
deps:
    $(foreach dir, $(DEPS), $(shell cd $(dir) && $(MAKE)))

Conclusion

Hopefully this Makefile helps you, if not fully, maybe it at least gets you started
in having something that can cover the uses you need.

Unique Shared Library Problem

Joshua Matthews — Tue, 28 Jan 2025 05:35:11 +0000

The Problem

I was trying to work with a library the other day that I built from source, however, I could not get it to work with my local C project. I was getting undefined reference errors when linking to the shared library.

/bin/ld: /tmp/ccHb7mJ8.o: in function `SDL_main':
main.c:(.text+0x3c): undefined reference to `SDL_EnterAppMainCallbacks'
/bin/ld: /tmp/ccHb7mJ8.o: in function `main':
main.c:(.text+0x6b): undefined reference to `SDL_RunApp'
/bin/ld: /tmp/ccHb7mJ8.o: in function `SDL_AppInit':
main.c:(.text+0xa7): undefined reference to `SDL_SetAppMetadata'
/bin/ld: main.c:(.text+0xb1): undefined reference to `SDL_Init'
/bin/ld: main.c:(.text+0xbd): undefined reference to `SDL_GetError'
/bin/ld: main.c:(.text+0xd4): undefined reference to `SDL_Log'
/bin/ld: main.c:(.text+0x107): undefined reference to `SDL_CreateWindowAndRenderer'
/bin/ld: main.c:(.text+0x113): undefined reference to `SDL_GetError'
/bin/ld: main.c:(.text+0x12a): undefined reference to `SDL_Log'
collect2: error: ld returned 1 exit status
make: *** [Makefile:7: all] Error 1

The Investigation

I recompiled the library several times trying different methods and nothing helped. After searching google for a couple of hours I came across a forum where someone was having a similar issue but they were using a 32bit version of their toolchain instead of a 64bit. I figured this wasn't the issue but I investigated anyways. Everything I checked showed it was 64bit, so I was back to square one. Though after thinking about toolchains I did think maybe I could try a different compiler.

The Solution?

I was using gcc at the time so I switched to clang and reran my Makefile.

To my surprise, everything worked! I was shocked.

It couldn't be that simple? Why has this never been a problem before?

Understanding the Solution

So I decided to create a control project to test this phenomenon. I setup a small add shared library built with clang and then tried to compile a driver program with gcc.

lib.h

#pragma once
int add(int a, int b);

lib.c

#include "lib.h"
int add(int a, int b) {
  return a + b;
}

main.c

#include "lib.h"
#include <stdio.h>
int main () {
  printf("4+3=%d\n", add(4, 3));
  return 0;
}

build_so.sh

clang -std=c11 -c -o lib.o lib.c
clang -shared -fPIC -o libm.so lib.o

build_main.sh

gcc -std=c11 -L. -l:libm.so main.c -o main

It failed, just like in my C project.

/bin/ld: /tmp/ccymm8ki.o: in function `main':
main.c:(.text+0x13): undefined reference to `add'
collect2: error: ld returned 1 exit status

Changing the build.sh to use clang obviously fixes the problem but I wanted to know if this problem happens the other way round, so I tried to switch the compiler positions, to see if it would fail the other way.

build_so.sh

gcc -std=c11 -c -o lib.o lib.c
gcc -shared -fPIC -o libm.so lib.o

build_main.sh

clang -std=c11 -L. -l:libm.so main.c -o main

This time and everything worked. So this might be why I had never encountered this before.

The Conclusion

clang shared libraries have problems being used with gcc for some reason, but gcc shared libraries can be used with clang no problem.

If there is a way around this problem, I would love to know!

I think in most cases you probably will use the same compiler for your projects but in my case where the project I was compiling from source decided to use clang instead of gcc to create this unique shared library problem I had not encountered before.

Implementing I2C for the ATtiny85

Joshua Matthews — Sun, 24 Nov 2024 06:05:38 +0000

tl;dr

Implemented the I2C protocol for controller devices on an ATtiny85 using just the AVR base library. Link to Github

The Precursor
Understanding the Protocol
- High Level
- The Format
The Research
- Pinouts
- Port Registers
- Universal Serial Interface
- Speed Support
The Implementation
- Header File
- Source File
  - Initialize
  - Start
  - Stop
  - Transfer
  - Write
  - Read
  - Write Target Address
Conclusion and Demo

The Precursor

I've done some Arduino projects before but it was with using the Arduino IDE and their helper libraries.
So I wanted to try my hand at programming something at the base level -- No Arduino IDE or helper libraries.

I decided the I2C protocol would be a big enough project to get my feet wet with AVR programming but small enough to not feel overwhelmed. It would also be easy to debug with a target device (I used an Arduino Uno) to ensure everything was working appropriately.

Understanding the Protocol

Reference: Historically the terms Master/Slave were used to refer to the relationship of these devices, but modern naming conventions use Controller/Target ¹

High Level

Inter-Integrated Circuit (or I2C) is a widely supported, simple synchronous method of communication between lower-speed peripherals. The setup usually involves a Controller with one or more Target devices talking across two bidirectional signal lines: Serial Data Line (SDA) and Serial Clock Line (SCL). ¹

The basic concept of these two lines is SDA is for data and SCL is for keeping the timing correct between the two devices. There are several speeds that are supported but the standard speed is 100kbit/s.¹

The Format

The format for communication is pretty simple and I feel the easiest way to understand it is with this diagram:

I2C Diagram - (Sparkfun, 2013)³

The first thing to notice is both SDA and SCL are set to high at the beginning. SDA is then brought to the low state and after a short wait the SCL line is brought low as well. This will wake every target device on the line to check if they will be the one the controller is talking to.

Next the controller sends the 7 bit address of the target device. We use a 7 bit address because the least significant bit is reserved as the Read/Write flag to tell the target device which mode it should be in (1 is Read, 0 is Write). Through this data exchange you'll notice the SCL line is toggled in the middle of the SDA line's state. This is important to not trigger the stop state (we'll talk about this soon).

At this point the target devices that do not match the address go back to ignoring the messages. Then the controller changes its SDA line to input and the target device that matches the address sends an N/ACK to ACKnowledge or Not ACKnowledge that data transfer.

Now the controller and target are connected and depending on the Read/Write flag the controller/target will execute accordingly until done. In the case of a Write, the controller sends data in 8 byte intervals until done or until the target returns a NACK. In the case of a Read, the controller reads data and sends back an ACK to the target until the controller is done, at that point the controller sends a NACK followed by a stop message.

Last is the stop message, which is basically the opposite of the start message. Both SDA and SCL lines are forced to the low state. Then the SCL is brought high and after a short wait, the SDA line is brought high.

That's the basics of the protocol, there are some extended rules for 8 or 10-bit addressing⁴ and some special cases like repeat starts³ but we aren't really concerned with those for our purposes.

The Research

The research portion of this project took the longest. I knew I'd have to comb through the data sheet of the ATtiny85 to figure out most things, but still being quite new to bare-metal programming I found myself lost a few times searching through it.
Using a combo of reference material from the TinyWire⁵ library and the datasheet helped guide me through most of the troubles I encountered.

Pinouts

The first thing I needed to do was look at the pinout diagram to figure out what pins supported I2C on the ATtiny85. Looking through the datasheet⁶ they have this handy diagram that explains all the pins.

Pinout ATtiny25/45/85 - (Atmel, 2013 - Figure 1-1 pg. 2)⁶

We can see that PB0 and PB2 are our SDA and SCL pins respectively. So we will start here by searching how to configure these pins for I2C serial communication.

Port Registers

Searching through the document we find chapter 10 on I/O Ports and within this chapter has this diagram for pin configuration.

Port Pin Configurations - (Atmel, 2013 - Table 10-1 pg. 55)⁶

Knowing that we need to use pullup resistors for our SDA and SCL lines we are interested in row two. This shows to setup pullup resistors we need to have the configuration (0, 1, 0) for our DDxn, PORTxn, PUD flag respectively.
It is important to note that we are just using the internal pullup resistors for our use case, but depending on your device or the number of devices, you will need to use external pullups to ensure consistent results.

Next we find these registers, which are defined in these tables.

Register Descriptions - (Atmel, 2013 - Section 10.4 pg. 64)⁶

We see that the PUD flag and all of DDRB has initial values set to zero, so we really only need to worry about setting PORTB0 and PORTB2 to 1. If you are writing a drop in library you might want to be safe and write the other bits to zero to ensure correct behavior.

Universal Serial Interface

Now that we've figured out the Port registers we need to figure out how serial communication works on the ATtiny85. Chapter 15 covers USI - Universal Serial Interface. This chapter has a lot to read through, but we can pull out the highlights.

The first thing from this chapter, in the overview, we learn that this chip has hardware level helpers to aid in serial communication which will allow less overhead than a pure software implementation. This also allows us to take advantage of the hardware controls to allow for less code on our end. Things included are: Pullups, Interrupts for flags and overflows, control flags to toggle the clock, and a data register for sending and receiving.

Further in the chapter we find the registers we will be using for communication.

USIDR USI Data Register - For reading and sending data
USISR USI Status Register - For checking and managing state with interrupt flags and timers
USICR USI Control Register - For controlling and enabling different modes and timers I listed these because it was a little difficult to get as a screenshot. But we'll look at them in detail later.

Speed Support

The last bit of research I needed to find was how to implement and support the different speeds available for I2C. Judging from this document⁷, the AVR controllers support standard (100kbps) and Fast mode (400kbps). So what do we need to do to maintain this bit rate?

Looking back at the TinyWire⁵ they seem to have these hardcoded values for timing.

TWI Delay Constants - (TinyWire, 2019 - twi.h)⁵

But where did they get these values? It took me a while to track down but these values are from the NPX I2C Timing Specification⁸

Characteristics of the SDA and SCL bus - (NXP I2C Timing Specification, 2021 - Table 11, pg. 44)⁸

The rows we care about are the t_LOW and t_HIGH rows. From this we can see to meet the Standard and Fast requirements we will need delays of (4.7 and 4.0) and (1.6 and 0.6) respectively.

Now that we have most of our reference material we can start our implementation.

The Implementation

Header File

First we will define out our header file to lay out our constants and functions.

First we define the wait periods for the different bit rate speeds we can support.

// These numbers are from a calculation chart
// from I2C spec -- NXP I2C Timing Specification
#ifdef I2C_FAST_MODE // 400kHz fast
  // low period of SCL
  #define T2_TWI 2 // >1.3microseconds
  // high period of SCL
  #define T4_TWI 1 // >0.6microseconds
#else // 100kHz standard
  // low period of SCL
  #define T2_TWI 5 // >4.7microseconds
  // high period of SCL
  #define T4_TWI 4 // >4.0microseconds
#endif

Again, these numbers are defined in this document on page 44⁸. You can see we didn't use the numbers exactly. From the references⁵ I used, it seemed common to round up to the next whole number. This is the reasoning for the choice 2 over 1.3, as an example.
I'm not completely sure for the reasoning behind this other than maybe it's safer to use whole numbers with the delay functions than the fractional values when making sure you are staying within the cut off point of the range.

Next we define out the constants for the registers.

#ifndef i2c_ddr
  #define i2c_ddr DDRB
#endif
#ifndef i2c_port
  #define i2c_port PORTB
#endif
#ifndef i2c_pin
  #define i2c_pin PINB
#endif
#ifndef i2c_scl
  #define i2c_scl PORTB2
#endif
#ifndef i2c_sda
  #define i2c_sda PORTB0
#endif
#ifndef i2c_data
  #define i2c_data USIDR
#endif
#ifndef i2c_status
  #define i2c_status USISR
#endif
#ifndef i2c_control
  #define i2c_control USICR
#endif

This step isn't really necessary but I felt these names were a bit better at conveying what they are for.

The last thing for the header are the functions.

/**
 * Initialize I2C registers and ports.
 */
void i2c_init();

/**
 * Send start command.
 * @return True if the start successfully initiated, False otherwise.
 */
bool i2c_start();

/**
 * Send the stop command.
 */
void i2c_stop();

/**
 * Write the given byte.
 * @param[in] data The byte to send.
 * @return The N/ACK byte.
 */
unsigned char i2c_write_byte(unsigned char data);

/**
 * Read the next byte.
 * @param[in] nack True for reading more, false otherwise.
 * @return The read byte.
 */
unsigned char i2c_read_byte(bool nack);

/**
 * Write the target's address out.
 *
 * @param[in] address The target address.
 * @param[in] write Flag for Write or Read bit.
 * @return The N/ACK byte.
 */
unsigned char i2c_write_address(unsigned char address, bool write);

This should be the basics of the protocol with these functions. We will go into more detail in this next section.

Source File

Initialize

First we will write out our initialization code. We need to set defaults and configure our registers properly.

I'll show the function first and then break it down.

void i2c_init() {
  // preload data register with default HIGH
  i2c_data = 0xff;

  // setup for master
  i2c_control = (
    // disable Start condition interrupt
    (0 << USISIE) |
    // disable overflow interrupt
    (0 << USIOIE) |
    // only set WM1 to high for normal 2wire mode
    _BV(USIWM1) | (0<<USIWM0) |
    // set CS1 and CLK to high to use external clock source
    // with positive edge. Software Clock Strobe (with USITC register)
    _BV(USICS1) | (0<<USICS0) | _BV(USICLK) |
    (0<<USITC)
  );

  i2c_status = (
    // clear all flags
    _BV(USISIF) | _BV(USIOIF) | _BV(USIPF) | _BV(USIDC) |
    // reset overflow counter
    (0x0 << USICNT0)
  );
  // flip the ports to input mode so we can enable pullup resistors on the next line
  i2c_ddr &= ~_BV(i2c_sda);
  i2c_ddr &= ~_BV(i2c_scl);

  // set both pins to HIGH to enable pullup.
  i2c_port |= (_BV(i2c_sda) | _BV(i2c_scl));

  // flip the ports to output mode
  i2c_ddr |= (_BV(i2c_sda) | _BV(i2c_scl));
}

The first thing we do for house keeping is setting the data register to a default value of HIGH (0xFF).

i2c_data = 0xff;

Next we configure the Control register, this will tell the controller how we want to interact with the Universal Serial Interface.

// setup for master
  i2c_control = (
    // disable Start condition interrupt
    (0 << USISIE) |
    // disable overflow interrupt
    (0 << USIOIE) |
    // only set WM1 to high for normal 2wire mode
    _BV(USIWM1) | (0<<USIWM0) |
    // set CS1 and CLK to high to use external clock source
    // with positive edge. Software Clock Strobe (with USITC register)
    _BV(USICS1) | (0<<USICS0) | _BV(USICLK) |
    (0<<USITC)
  );

The comments explain a lot of what we are doing here but I will link to the datasheet where these values are explained.

On pages 116-117 of the Atmel datasheet⁶ the control register is explained. The first two bits are just interrupt enable flags and since we are the controller we don't need these enabled.

Next is setting the wire mode. Since the Universal Serial Interface can support multiple modes, we need to configure which one we are using. The configuration table shows us which values to set for two-wire mode (I2C).

Wire Mode - (Atmel, 2013 - Table 15-1, pg. 117)⁶

The next three bits are related to how we want to interact with the clock. We've configured the clock source to be an External, positive edge, and we want to use a software clock strobe.

Clock Settings - (Atmel, 2013 - Table 15-2, pg. 118)⁶

The jest of the clock source is we need to tell how the clock is sourced and how the data output latch changes with respect to the sampling of the data input. The 4-bit counter clock source is just to define how the clock is being controlled. Since we are the controller we want to strobe it ourselves.

The last bit is turned off and this is the bit for toggling the clock. To toggle the clock you need to write a 1 to it.

Next we initialize our status register to default values.

i2c_status = (
    // clear all flags
    _BV(USISIF) | _BV(USIOIF) | _BV(USIPF) | _BV(USIDC) |
    // reset overflow counter
    (0x0 << USICNT0)
  );

The first four bits are as follows:

Start Condition Interrupt Flag
Counter Overflow Interrupt Flag
Stop Condition Flag
Data Output Collision

Writing a logical one to these bits clears them out if they were set.
The remaining 4 bits in this register are the 4-bit counter value used for the SCL line. 4 bits allows for 16 values, which translates to 8 byte transactions, which is the size of our payload. So when this overflows we've sent one payload's worth. So we need to reset it to zero.

The last portion of the initialization function is to enable the on-board pullups of the ATtiny85.

// flip the ports to input mode so we can enable pullup resistors on the next line
  i2c_ddr &= ~_BV(i2c_sda);
  i2c_ddr &= ~_BV(i2c_scl);

  // set both pins to HIGH to enable pullup.
  i2c_port |= (_BV(i2c_sda) | _BV(i2c_scl));

  // flip the ports to output mode
  i2c_ddr |= (_BV(i2c_sda) | _BV(i2c_scl));

The datasheet explains that setting the DDRB in the off position while setting the associated port bits in the on position enables the pullups. We then finish by setting the DDRB bits for SDA and SCL to output mode. The settings and caveats of the registers and pullup resistors are talked about in Port registers.

Start

Next, the start condition. Most of this code should be understood from the Understanding the Protocol section. But there are a couple of bits that might need some explaining.

Here is the code for the start:

bool i2c_start() {
  // ensure both lines are high
  i2c_port |= (_BV(i2c_sda) | _BV(i2c_scl));
  // wait till clock pin is high
  while (!(i2c_port & _BV(i2c_scl)));
  _delay_us(T2_TWI);

  // pull data line low
  i2c_port &= ~_BV(i2c_sda);
  // this is sampling time for the attiny85 to recognize the start command
  _delay_us(SAMPLING_WAIT);
  // pull clock line low
  i2c_port &= ~_BV(i2c_scl);

  // release data line to high
  i2c_port |= _BV(i2c_sda);

  // check for valid start
  return (i2c_status & _BV(USISIF));
}

We start off like normal, bringing both lines high. Then we need to wait for the SCL line to be high so we continually check to see if the pin value is set. Once it is, we wait a defined number of microseconds stated in the chart referenced in the Speed Support section.

Next we bring the data line low and then wait a sampling amount of time. This time frame is defined in both the NXP I2C Timing Specification⁸ on Table 11, page 44 and the Atmel datasheet⁶ in section 15.3.5 on page 113. Both define it as a max of 300 nanoseconds with the Atmel datasheet defining the minimum as 50 nanoseconds.
So I've defined the sample wait constant as so:

// this is 0.2 microseconds which is 200 nanoseconds
#define SAMPLING_WAIT 0.2

Next, bring the clock line low and then release the data line to finish up. The last thing I've added is a validity check. If the USISIF bit is set then the start condition was recognized, if not something went wrong.

Stop

The stop condition is about the same except you don't have to worry about a sampling time with this one.

void i2c_stop() {

  // ensure data line is low
  i2c_port &= ~_BV(i2c_sda);

  // relase clock line to high
  i2c_port |= _BV(i2c_scl);
  // wait for clock pin to read high
  while (!(i2c_pin & _BV(i2c_scl)));
  _delay_us(T4_TWI);

  // relase data line to high
  i2c_port |= _BV(i2c_sda);
  _delay_us(T2_TWI);
}

This pretty much follows the spec and uses the delays in accordance with the table specified in Speed Support. Bringing both lines low and then releasing the clock line. After it reaches the on state, release the data line.

Transfer

Next we will handle the general transferring of data.

unsigned char transfer(unsigned char mask) {
  // ensure clock line is low
  i2c_port &= ~_BV(i2c_scl);
  i2c_status = mask;

  do {
    // wait a little bit
    _delay_us(T2_TWI);
    // toggle clock
    i2c_control |= _BV(USITC);
    // wait for SCL to go high
    while (! (i2c_pin & _BV(i2c_scl)));
    // wait short
    _delay_us(T4_TWI);
    // toggle clock again
    i2c_control |= _BV(USITC);

  } while (!(i2c_status & _BV(USIOIF)));
  _delay_us(T2_TWI);

  // clear counter overflow status
  i2c_status |= _BV(USIOIF);

  unsigned char data = i2c_data;
  i2c_data = 0xff;
  return data;
}

The first line is probably not necessary but we just want to ensure the clock is low before beginning. The next line is setting the status to the mask parameter. (we will touch on this in the write/read functions).

Next we start our loop. The ATtiny85 will handle sending the data as we toggle the clock. So we wait the specified time between the HIGH and LOW of the clock line referenced in the table in the Speed Support section and toggle the clock on the control register by setting the USITC bit flag.

Once we receive the counter overflow condition we exit the loop, wait a little more and then clear out that overflow status flag. We then grab the data from the data register, reset the data register, and then return our received byte value.

Write

With most of the scaffolding setup now the next few functions will be pretty easy to flesh out.

Starting with the write function:

unsigned char i2c_write_byte(unsigned char data) {
  i2c_data = data;
  transfer(STATUS_CLOCK_8_BITS);

  // change data pin to input
  i2c_ddr &= ~_BV(i2c_sda);
  unsigned char nack = transfer(STATUS_CLOCK_1_BIT);
  // change back to output
  i2c_ddr |= _BV(i2c_sda);
  return nack;
}

Stepping through this function we see we are setting the data register to the data we would like to send over. Then we call the transfer function with this new STATUS_CLOCK_8_BITS constant. This constant is to reset the status register to be ready for a full 8 bits transfer.

It is defined as such:

#define STATUS_CLOCK_8_BITS (_BV(USISIF)|_BV(USIOIF)|_BV(USIPF)|_BV(USIDC) | \
    (0x0 << USICNT0)) // reset clock to allow for full 8 bit transfer

We clear the status flags and reset the counter to 0.

After sending our data we need to flip our pin to input mode to read the ACK bit from the target. We then call the transfer method with another constant which sets up the status register to clear status flags and set the counter to a value that will force an overflow after 1 bit transfer.

It is defined as such:

#define STATUS_CLOCK_1_BIT (_BV(USISIF) | \
   _BV(USIOIF) | \
   _BV(USIPF) | \
   _BV(USIDC) | \
   (0xE << USICNT0))// we set the clock to 1110 so it overflows after 1 exchange

After the N/ACK is returned we set the SDA line back to output and return the received byte.

Read

The read function has slightly more logic, because we need to tell the target if we would like to read more or to stop.

unsigned char i2c_read_byte(bool ack) {
  // HIGH value means stop sending
  unsigned char response = 0xff;
  if (ack) {
    // LOW means read another byte
    response = 0x00;
  }
  // change data pin to input
  i2c_ddr &= ~_BV(i2c_sda);
  unsigned char data = transfer(STATUS_CLOCK_8_BITS);
  // change back to output
  i2c_ddr |= _BV(i2c_sda);
  // send n/ack
  i2c_data = response;
  transfer(STATUS_CLOCK_1_BIT);
  return data;
}

We change our response value based on if we want to read another byte or not. This is determined by the ACK (0x00) or NACK (0xFF) value.

Next we operate similarly like the write function except we capture the data first and then send our N/ACK response.

Write Target Address

We basically have everything we need to perform I2C now but I wanted a helper function to handle sending the target address properly for me. So we have this last method.

unsigned char i2c_write_address(unsigned char address, bool write) {
  unsigned char rw_flag = 1;
  if (write) rw_flag = 0;
  // shift address over 1 because Least sig bit is RW flag
  return i2c_write_byte(((address << 1) | rw_flag));
}

In our case we are only dealing with the basic 7-bit addressing so we shift the address over by 1 bit and then logically or it with the rw_flag bit. This format is explained in The Format section.

Conclusion and Demo

After all that, we can now properly handle I2C on our ATtiny85. This project has helped me explore the embedded space more than I had experience with before. It's given me a new appreciation for the craft that goes into these projects and has me excited to tackle the next embedded journey I dig into.

Here is the link⁹ and the demo video of the finished project.

The demo has an ATtiny85 loaded with the I2C driver I wrote and hooked up to an Arduino uno using the Wire library to act as a target device to talk to.

Actions performed:

The controller (ATtiny85) sends "Hello, World!" to the target (Arduino Uno) device.
The controller requests a read from the target to toggle the LED.

References

Quick Zig and C String Conversion Conundrums

Joshua Matthews — Sun, 13 Oct 2024 22:10:36 +0000

Intro

My background is mostly in C and as I am still new to zig some of the type conversions needed for C and Zig to talk were not crystal clear at the beginning. Now I understand them and I'll give a quick rundown to hopefully help anyone else that needs it.

C String types

Let's start off with what a C string type is in Zig. There are 2 recommended¹ ways of denoting a C string.

// Sentinel slice of unknown amount
[*:0]const u8
// Slice of unknown amount
[*]const u8

If you can expect the string to be null-terminated you want the first option which can be converted into a Zig slice with the std.mem.span function. Otherwise, you'll want the second option with you usually requiring a length parameter passed into your exported function so you can get a slice-by-length.

Examples:

export pub fn test_c_string(str: [*:0]const u8) void {
    const local_slice: []const u8 = std.mem.span(str);
    // rest of function
}

export pub fn test_c_string(str: [*]const u8, len: usize) void {
    const local_slice: []const u8 = str[0..len];
    // rest of the function
}

That's really all you need to know for your C string needs. The rest of Zig's strings can convert between Zig slice/array types fairly easily without much intervention.

One type that confused me at first was array sentinel types (i.e. [5:0]const u8) because I assumed it was similar to [*:0]const u8 but the difference is the comptime length (i.e. 5) which turns this slice into a known length so Zig can do it's slice conversions between similar types easily.

You can also do [*c] to signify a C pointer but it is noted this should only be used in autogenerated code.

Using Postgres Notifications To Enhance Dev Testing

Joshua Matthews — Wed, 09 Oct 2024 03:02:55 +0000

tl;dr Link to Github repo

The Setup

Sometimes your database can seem like a black box. Especially as you are running your application and it makes tons of calls and actions out to it. So many inserts and updates it's hard to keep track of what's happening.

"What was the ID of that latest entry?"

"What does this foreign key point to again?"

All valid questions as you debug your latest changes to your application. The DB keeps it's curtain drawn while you have to search for these new updates. Usually your front-end can help grab them all for you, but that won't be the case where you can view your DB changes easily through the front-end all the time.

So what can you do?

The Investigation

In my case, our database is Postgres and Postgres supports a nifty feature called pg_notify. I'm sure other DBs support this feature in some form or another, but for the purposes of this article I will be focusing on my use case.

The basic usage of this function is as follows:

pg_notify('channel', 'notification text');

You have the channel you'd like to post the notification on and the notification text data you'd like to send on that channel.

So we know how to notify, how do we listen?

In Postgres you can listen on a channel LISTEN 'channel'. However, we don't want to listen in Postgres, we want to listen in our listener application.

Stomping on the Curtains

To setup our own notifications for our tables we just need to setup a reusable trigger.

create or replace function person_notif_fn()returns trigger as $f$
begin perform pg_notify('person', to_jsonb(new)::text);
    return new;
end $f$ language plpgsql;

Here we have a simple function that returns a trigger that performs a pg_notify call to our channel (person) and takes the incoming new object and transforms it into a JSON string. Then we return new back out.

Now that we have our trigger function, we need to attach a trigger to call our trigger on our table.

create trigger person_notif_tr after insert or update on person
for each row execute function table_notif_fn();

That's it! Now our database will notify us of any inserts or deletes on our person table.

The Hookup

Creating an application that can tap into this is pretty easy with certain libraries. I created my app in Zig so I'll be using pg.zig but you can use whatever you'd like.

First we establish our connection to our DB

const db = try pg.Pool.init(std.heap.page_allocator, .{
            .size = 5,
            .connect = .{
                .port = 5432,
                .host = "127.0.0.1",
            },
            .auth = .{
                .username = "postgres",
                .password = "postgres",
                .database = "postgres",
                .timeout = 10_000,
            },
        });

We've created our Pool now we just need to create a listener and connect to it.

const listener = db.newListener();
listener.listen("person");

Now our DB driver is listening on the person channel for notifications. So let's listen for them.

// create a fixed buffer allocator for our JSON object
var out_buffer: [4096 * 6]u8 = undefined;
var fixed_alloc = std.heap.FixedBufferAllocator.init(&out_buffer);
// infinite loop to continually listen for updates
while (true) {
    // iterate on our listener
    while(listener.next()) |notif| {
        // reset the allocator
        fixed_alloc.reset();
        // create a dynamic string
        var string_writer = std.ArrayList(u8).init(fixed_alloc.allocator());
        // this is a custom object we are creating to store our info
            const info = notification{
                .channel = notif.channel,
                .payload = notif.payload,
            };
        // stringify the data
            try std.json.stringify(
                info,
                .{},
                string_writer.writer(),
            );
            // write out to buffer/websocket/etc.
    }
}

Here we've created an infinite loop to continually check our listener poll. It's an infinite loop because our listener times out every now and then when polling for updates. (There could be a way to fix this but I couldn't find it quickly). Then we stringify the data and from there we can send it off to a buffer/web-socket/ whatever you want the data to be sent to.

And that's pretty much it! From here we can let our imagination run wild with how we want to improve or customize on top of this.

The Next Level

Looking further into some of the niceties that Postgres gives us with it's built in functions and metadata from it's schema tables, we can automate getting all table data for any DB and adding triggers to them dynamically when connecting. This could streamline our listener application and become a really powerful development tool for us to use.

I won't go into full detail because you can reference my Github project for the full implementation, but here is a simple snippet to grab all base tables in a Postgres DB.

SELECT table_schema || '.' || table_name
FROM information_schema.tables
WHERE table_type = 'BASE TABLE' AND table_schema NOT IN (
  'pg_catalog', 'information_schema', 'typeorm'
);

This query will return a list of tables in the format <table_schema>.<table_name> for the entire DB -- excluding pg_catalog, information_schema, and typeorm. Below is an example output.

public.acconuts
public.mail_info
public.log_info

Now I'm going to do some hand wavy magic to skip over setting up the pipeline of getting the Postgres metadata into my server and displaying on a webpage, since the purpose of this article was just to be an introduction to DB notifications.

Nevertheless, here is a demo of the final prototype of setting up notifications and grabbing the metadata of all tables in my DB.

The Conclusion

There's probably a lot of untapped potential in the tools you integrate with that you could be using to amplify your developer experience. Don't leave any stone unturned and always ask if your tool could do more, because it probably can.

Implementing UTF-8 Encoding in Zig

Joshua Matthews — Tue, 24 Sep 2024 06:12:58 +0000

tl;dr
Created a library to read/write UTF-8 encoded Unicode values in Zig for my simple text editor. Link to GitHub repo

Context

One of my side projects is a simple text editor in C called "Editlite". The purpose of this editor was to explore creating one from the ground up and implementing functionality like Plugins and loading files incrementally to support editing large files. For this article I'll focus on just the Unicode support I wanted to add, though I may do a deeper dive into the editor in a future article.

Intro
The Beginning
The Format
The Build
C Header
The Integration

Intro

Learning Zig has been on my todo list for some time now and I have been trying to find more excuses to work it into projects to become more familiar with the language. I finally found a little project I could get my feet wet while also producing something that wouldn't be just a throw away program. I have a simple text editor and it currently only supports the ASCII character set, so I thought adding a basic Unicode support library would be the perfect byte-sized intro to Zig I wanted.

The Beginning

Unicode is a standard that lays out how numerous symbols are mapped to certain code points so that applications can read and correctly display the appropriate text on any system. For example the number 69 (Hex value: 45) maps to the character E. However, number 42069 (Hex value: A455) maps to character ꑕ.

There are different encodings to support this standard but the one I chose to implement was the UTF-8 standard, which breaks these numbers up into 8bit unsigned integer type representations over the range of 1 to 4 bytes.

The Format

The best way to visualize the UTF-8 encoding format is with this table from the RFC. Also to note, there are two things we will not cover in this article and that is the Byte-order Mark which is not required (default is big-endian) and the UTF-16 Surrogates (reserved byte range U+D800-U+DFFF) which should be considered invalid in the UTF-8 format.

(Fig. 1)

We'll break down each line and start codifying it into Zig for the baseline rules of our library.

The first example 0xxxxxxx allows for this standard to be backwards compatible with the ASCII character set. The first bit needs to be 0 and you have 7 free bits to use. So we need to ensure that first bit is zeroed out to be a valid 1 byte UTF-8 octet sequence.

(Fig. 2)

The next row starts defining the main pattern we will follow for the other sequence types. The leading 110 bits signify that this sequence is a 2 byte UTF-8 octet sequence and the following octet leads with 10 to signify this is the next byte in the current octet sequence. So we see this encoding uses a 0 padding bit to differentiate between all of these octet sequences.

This is how we could codify those two rules in Zig.

(Fig. 3)

The following rows follow the same rules but for the 3 byte and 4 byte octet sequences with 1110 and 11110 respectively.

(Fig. 4)

Now that we've codified these rules we can move forward with writing our library.

The Library

The first thing we'll do is define some convenience types to work with.

(Fig. 5)

I wanted an easy way to keep track of what type of octet sequence type a certain code point was without having to re-encode it. Also this library is meant to be used in an existing C project so we need to create our enum using c_int and add the extern keyword to our structure.

(Fig. 6)

Next we define a convenient function to verify multi-byte octet sequences.

(Fig. 7)

The last convenience function we will explain is this one which determines which octet type is the given 8bit value. We'll explain the export keyword with the next screenshot.

Now the core of the library! The first function we'll define is the parsing function. I like to start at the unit level so we'll write a function to read the "next" Unicode code point in a given u8 array. Here's the definition of the function:

(Fig. 8)

Of course, we add the export keyword to tell Zig to compile this function with the C ABI and we also need to use C compatible types -- so the arr parameter is a [*]const u8 which is a slice of "unknown" size.

Alright, now to the meat of the function:

(Fig. 9)

We start off with a reusable invalid_point object to return on errors. Next we do some housekeeping checks. We then define our initial result which we grab the starting octet out of the array and determine it's octet type with get_oct_type. Next we just switch on the type and try to parse from there.

The first two cases are easy. If the initial type was OCT_INVALID or OCT_NEXT then this isn't a correctly formatted UTF-8 string so we return an "invalid" code point. For a OCT_ONE type we just pull the value straight out.

The rest of the cases are a little more involved but still straightforward. We check to make sure there are an expected number of bytes based on the type. We also verify the rest of the bytes are formatted properly with verify_octets. Then we pull each value out and & it with it's corresponding free bits. Lastly, we shift the values based on their free bits (6 * offset) and logical or the values into our code point in big-endian format.

That's it for parsing! Now you can use this method to loop over your UTF-8 encoded buffers!

The next crucial function to implement is the "write" function -- to take a u32 code point and write it back out to a UTF-8 encoded buffer.

(Fig. 10)

This function is a thin function for the C API to take in a C-style array and turn it into a Zig slice then pass it onto the real write function.

(Fig. 11)

The write function is also straightforward in it's implementation. Switching on the code point's type and pulling out the appropriate byte information from the u32 in big-endian format. You'll notice some convenience functions that are just thin inline functions to ensure the value is in the correct format of the UTF-8 octet sequence markers.

Now we can continually write out code points to a given u8 buffer!

The Build

Next, the build.zig file. Most of this is standard but we needed to bundle the Zig compiler runtime into the static build, so this is how the build file looked.

(Fig. 12)

C Header

Next we need to generate the C header file to accompany the static library. Luckily the types easily map over.

(Fig. 13)

You can ignore the __THROWNL and __nonnull(()) calls, the important part to take away from this is we need to set the extern keyword in front of our functions. Now we can use this in our C programs like you would normally include a static library!

The Integration

After integrating this library into my simple text editor I can now read/write and accept Unicode input.

(Fig. 14)