DEV Community: Tony Miller

The Definitive Guide to _safe_ Docker-in-Docker with Gitea Actions

Tony Miller — Fri, 29 Aug 2025 14:41:58 +0000

The Definitive Guide to Docker-in-Docker with Gitea Actions

It has come to my attention that someone is wrong on the Internet. Actually, not just someone—seemingly everyone writing guides about setting up Docker-in-Docker (DIND) with Gitea Actions. The multitudes of tutorials, blog posts, and StackOverflow answers all seem to miss critical architectural limitations and security considerations that make their solutions either incomplete, insecure, or simply non-functional in real-world scenarios.

This guide presents a complete, almost production-ready example for isolated Docker-in-Docker CI/CD using Gitea Actions with proper security boundaries and full functionality.

You can find all the examples mentioned here in this repository on GitHub

⚠️ Enterprise Security Warning

This setup is intended for development and testing environments. For true enterprise-grade production deployments, additional security measures are required:

Critical Security Enhancements Needed for Enterprise-grade Production

Network Firewall Protection
- Deploy firewall rules to isolate the CI/CD network from internal corporate networks
- Implement egress filtering to prevent build containers from accessing internal services
- Use network segmentation to contain potential container breakouts
- Consider running the entire stack in a separate VLAN or VPC
Container Image Security
- Gitea needs a feature that only white listed images are allowed to run in priveleged-mode
- Only allow pre-approved, security-scanned base images
- Implement image signing and verification workflows
- Regular vulnerability scanning of all container images
DIND Image Hardening
- Remove unnecessary packages and tools from the custom DIND image
- Implement read-only root filesystem where possible
- Use distroless or minimal base images
Plenty more with various compliance stuff but the above state is a good start.

The configuration presented here prioritizes functionality and ease of setup over maximum security hardening.

Requirements & Use Case

We need a CI/CD environment that provides:

Complete isolation from the host Docker daemon
Docker functionality available to both services and build steps
Self-contained deployment with no external dependencies
Proper security boundaries between jobs and the host system
Full Docker API access for build, test, and deployment workflows

The Problem: Gitea Actions Limitations

Gitea's act_runner is based on the excellent nektos/act project, but it has several critical limitations when compared to GitHub Actions:

1. Incomplete Services Support

Incomplete volumes mounting capability for services in workflow YAML
Limited options support compared to full docker run functionality
No command override support in the services: section

2. Docker Configuration Challenges

Cannot mount daemon.json configuration files into service containers
No way to inject custom Docker daemon startup parameters
Services are treated as immutable "black boxes"

3. GitHub Actions Parity Issues

GitHub Actions itself doesn't support:

command overrides in the services: section

Why Custom DIND Images Are Required

Standard docker:dind images:

Listen on Unix socket by default (/var/run/docker.sock)
Have TLS enabled by default (requires certificates)
Cannot be configured via environment variables for network settings
Cannot be customized through workflow YAML due to services limitations

Our solution: Build a custom DIND image with hardcoded configuration:

FROM docker:dind
CMD ["dockerd", "--host", "tcp://0.0.0.0:2376", "--tls=false"]

Architecture Overview: Triple-Nested Isolation

Our architecture provides three layers of Docker isolation:

Host Docker (docker-compose level) - Orchestrates the entire CI/CD stack
Runner DIND (act_runner execution environment) - Provides Docker services for workflows
Build DIND (workflow build steps) - Enables Docker operations within build containers

This triple nesting is essential because:

Services run in the runner's Docker daemon
Build steps run inside containers with no Docker daemon access
Each layer provides different security boundaries and functional contexts

Step-by-Step Implementation

Step 1: Create the Custom DIND Image

Dockerfile:

FROM docker:dind

HEALTHCHECK --interval=30s --timeout=10s --start-period=30s --retries=3 \
    CMD docker version || exit 1

CMD ["dockerd", "--host", "tcp://0.0.0.0:2376", "--tls=false"]

Step 2: Configure Gitea for Actions

⚠️ Security Note: The INTERNAL_TOKEN and JWT_SECRET values in the example app.ini below are provided for convenience to make this docker-compose example work out-of-the-box. In production deployments, these tokens MUST be regenerated using fresh, cryptographically secure random values. Never use these example tokens in any environment beyond local development and testing.

app.ini:

APP_NAME = Gitea: Git with a cup of tea
RUN_MODE = prod
WORK_PATH = /data/gitea

[actions]
ENABLED = true

[repository]
ROOT = /data/git/repositories

[repository.local]
LOCAL_COPY_PATH = /data/gitea/tmp/local-repo

[repository.upload]
TEMP_PATH = /data/gitea/uploads

[server]
APP_DATA_PATH = /data/gitea
DOMAIN = localhost
SSH_DOMAIN = localhost
HTTP_PORT = 3000
ROOT_URL = http://localhost:3000
LOCAL_ROOT_URL= http://gitea:3000
DISABLE_SSH = false
SSH_PORT = 2222
SSH_LISTEN_PORT = 22
LFS_START_SERVER = false

[database]
PATH = /data/gitea/gitea.db
DB_TYPE = sqlite3
HOST = localhost:3306
NAME = gitea
USER = root
PASSWD = 
LOG_SQL = false

[indexer]
ISSUE_INDEXER_PATH = /data/gitea/indexers/issues.bleve

[session]
PROVIDER_CONFIG = /data/gitea/sessions

[picture]
AVATAR_UPLOAD_PATH = /data/gitea/avatars
REPOSITORY_AVATAR_UPLOAD_PATH = /data/gitea/repo-avatars

[attachment]
PATH = /data/gitea/attachments

[log]
MODE = console
LEVEL = info
ROOT_PATH = /data/gitea/log

[security]
INSTALL_LOCK = true
REVERSE_PROXY_LIMIT = 1
REVERSE_PROXY_TRUSTED_PROXIES = *
INTERNAL_TOKEN = eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYmYiOjE3NTY0NzI0OTF9.lXfJEgeQCkXcQx3VKm-TwLQktTYrccm_JK1P0xiDmEw

[service]
DISABLE_REGISTRATION = false
REQUIRE_SIGNIN_VIEW = false

[lfs]
PATH = /data/git/lfs

[oauth2]
JWT_SECRET = nq7Fpd5bAPFHOWHZJJah2rKfXdC3pKaF0pMgtaQwAdw

Step 3: Docker Compose Configuration

The complete docker-compose.yml orchestrates six services with proper dependency management:

docker-compose.yml:

services:
  gitea:
    image: gitea/gitea:latest
    container_name: gitea
    ports:
      - "3000:3000"
      - "2222:22"
    volumes:
      - gitea_data:/data
      - ./app.ini:/data/gitea/conf/app.ini:ro
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    environment:
      - USER_UID=1000
      - USER_GID=1000
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/api/healthz"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    restart: unless-stopped

  dind:
    build: .
    container_name: dind
    privileged: true
    volumes:
      - dind_data:/var/lib/docker
    environment:
      - DOCKER_HOST=tcp://localhost:2376
    healthcheck:
      test: ["CMD", "docker", "info"]
      interval: 10s
      timeout: 10s
      retries: 5
      start_period: 30s
    restart: unless-stopped

  registry:
    image: registry:2
    container_name: registry
    volumes:
      - registry_data:/var/lib/registry
    environment:
      - REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/var/lib/registry
    healthcheck:
      test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:5000/v2/"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s
    restart: unless-stopped

  image-builder:
    image: docker:dind
    container_name: image-builder
    depends_on:
      registry:
        condition: service_healthy
      dind:
        condition: service_healthy
    volumes:
      - ./Dockerfile:/workspace/Dockerfile
      - dind_data:/var/lib/docker
    working_dir: /workspace
    command: >
      sh -c "
        export DOCKER_HOST=tcp://dind:2376
        export DOCKER_TLS_VERIFY=\"\"
        echo 'Building and pushing DIND image to local registry...'
        docker build -t registry:5000/dind-plain:latest .
        docker push registry:5000/dind-plain:latest
        echo 'Image build and push complete!'
      "
    restart: "no"

  runner-configurator:
    image: gitea/gitea:latest
    container_name: runner-configurator
    depends_on:
      gitea:
        condition: service_healthy
    volumes:
      - runner_config:/config
      - ./app.ini:/data/gitea/conf/app.ini:ro
    command: >
      sh -c "
        echo 'Gitea is healthy, proceeding...'
        echo 'Generating runner token...'
        su - git -c 'cd /data && /usr/local/bin/gitea actions generate-runner-token' > /config/token
        echo 'Creating runner config...'
        GITEA_IP=$$(getent hosts gitea | awk '{print $$1}')
        REGISTRY_IP=$$(getent hosts registry | awk '{print $$1}')
        echo \"Resolved Gitea IP: $$GITEA_IP\"
        cat > /config/config.yaml << EOF
      log:
        level: info
      runner:
        file: .runner
        capacity: 1
        timeout: 3h
        insecure: false
        fetch_timeout: 5s
        fetch_interval: 2s
        labels:
          - 'ubuntu-latest:docker://gitea/runner-images:ubuntu-latest'
          - 'ubuntu-22.04:docker://gitea/runner-images:ubuntu-22.04'
          - 'ubuntu-20.04:docker://gitea/runner-images:ubuntu-20.04'
          - 'linux:docker://gitea/runner-images:ubuntu-latest'
      cache:
        enabled: true
        dir: '/tmp/cache'
        host: ''
        port: 0
      container:
        network_mode: bridge
        enable_ipv6: false
        privileged: true
        valid_volumes:
          - '**'
        docker_host: tcp://dind:2376
        options: '--add-host=gitea:$$GITEA_IP --add-host=registry:$$REGISTRY_IP'
      host:
        workdir_parent: /tmp
      EOF
        echo 'Configuration complete!'
      "
    restart: "no"

  admin-setup:
    image: gitea/gitea:latest
    container_name: admin-setup
    depends_on:
      gitea:
        condition: service_healthy
    environment:
      - GITEA_URL=http://gitea:3000
    volumes:
      - gitea_data:/data
      - ./app.ini:/data/gitea/conf/app.ini:ro
    command: >
      sh -c "
        echo 'Creating admin user...'
        su - git -c '/usr/local/bin/gitea admin user create --admin --username admin --password admin --email admin@localhost.local --must-change-password=false' || echo 'Admin user already exists'
        echo 'Admin setup complete!'
      "
    restart: "no"

  runner:
    image: gitea/act_runner:latest
    container_name: runner
    depends_on:
      gitea:
        condition: service_healthy
      dind:
        condition: service_healthy
      admin-setup:
        condition: service_completed_successfully
      runner-configurator:
        condition: service_completed_successfully
      image-builder:
        condition: service_completed_successfully
    volumes:
      - runner_config:/config:ro
      - runner_data:/data
    environment:
      - DOCKER_HOST=tcp://dind:2376
      - DOCKER_TLS_VERIFY=""
      - GITEA_INSTANCE_URL=http://gitea:3000
      - GITEA_RUNNER_REGISTRATION_TOKEN_FILE=/config/token
      - CONFIG_FILE=/config/config.yaml
    command: >
      sh -c "
        echo 'All dependencies ready, token file available...'
        echo 'Registering and starting runner...'
        act_runner register --config /config/config.yaml --no-interactive
        act_runner daemon --config /config/config.yaml
      "
    restart: unless-stopped

volumes:
  gitea_data:
  dind_data:
  runner_config:
  runner_data:
  registry_data:

Step 4: Example Workflow

.gitea/workflows/test-dind.yml:

name: Test DIND Integration

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]
  workflow_dispatch:

jobs:
  test-docker:
    runs-on: linux

    services:
      docker:
        image: registry:5000/dind-plain:latest

    env:
      DOCKER_HOST: tcp://docker:2376
      DOCKER_TLS_VERIFY: ""

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Wait
        run: sleep 10

      - name: Verify Docker daemon is running
        run: |
          echo "Testing Docker daemon connection..."
          docker info

      - name: List running containers
        run: |
          echo "Listing all containers..."
          docker ps -a

      - name: Test Docker functionality
        run: |
          echo "Testing basic Docker operations..."
          docker run --rm hello-world

      - name: Verify DIND isolation
        run: |
          echo "Testing container isolation..."
          docker run --rm alpine:latest echo "DIND is working perfectly!"

Security Considerations

This architecture provides multiple security boundaries:

Host Isolation: Docker-compose isolates the entire CI/CD stack from the host
Runner Isolation: Each workflow job gets its own Docker environment
Build Isolation: Docker operations in build steps use separate DIND instances
Network Isolation: Partial. Services and builds cannot directly access host resources but can access host network.

Conclusion

This guide provides a complete examplt of a production-ready solution for Docker-in-Docker with Gitea Actions. The architecture addresses the fundamental limitations of both GitHub Actions and Gitea's act_runner while providing proper security isolation and full Docker functionality.

The key insights that most guides miss:

Custom DIND images are required due to services configuration limitations
Triple-nested isolation provides both security and functionality

Achieving 30ms Zsh Startup

Tony Miller — Tue, 12 Aug 2025 16:14:54 +0000

Me and Claude went on a dotfiles journey the other day...

The Flex 💪

🍎  time zsh -i -c exit
zsh -i -c exit  0.02s user 0.01s system 87% cpu 0.033 total

33 milliseconds. That's faster than most people can blink. While others wait half a second
for their shell to load, we're already productive.

The Challenge

Most zsh configurations suffer from slow startup times ranging from 200ms to 500ms or more. Heavy frameworks like Oh-My-Zsh can push startup times beyond 1 second. Our goal was to achieve blazingly
fast shell initialization while maintaining full functionality for a modern development environment.

The Result

Final startup time: 33ms - putting us in the top 0.1% of shell performance.

Core Optimization Strategy

1. KISS Philosophy: Single-File Architecture

Instead of fragmenting configuration across multiple files, we organized everything within .zshrc using clear section headers:

### BREW
### FZF  
### COMPLETIONS
### NVM
### DOCKER
### KUBE
### PYTHON
### RUBY

This eliminates the overhead of sourcing multiple files while keeping code organized and maintainable.

2. Aggressive Lazy Loading

The biggest performance wins came from deferring expensive operations until actually needed:

NVM (Node Version Manager)

load_nvm() {
  unset -f npm npx yarn nvm load_nvm 2>/dev/null || true
  [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
  [ -s "$NVM_DIR/bash_completion" ] && \."$NVM_DIR/bash_completion"
}
npm() { load_nvm; npm "$@"; }
npx() { load_nvm; npx "$@"; }
yarn() { load_nvm; yarn "$@"; }
nvm() { load_nvm; nvm "$@"; }

Python Environment (pyenv)

function load_pyenv() {
  if command -v pyenv &> /dev/null; then
    eval "$(pyenv init --path)"
    eval "$(pyenv init - --no-rehash)"
    if which pyenv-virtualenv-init > /dev/null; then
      eval "$(pyenv virtualenv-init -)"
    fi
    pyenv virtualenvwrapper
    unset -f load_pyenv
  fi
}
alias pyenv='load_pyenv && pyenv'

The Fuck (command correction)

function init_thefuck() {
  unalias fuck
  unset -f init_thefuck
  eval $(thefuck --alias)
  fuck
}
alias fuck="init_thefuck"

3. Strategic Performance Settings

Completion Optimization

# Fast completion initialization
compinit -C  # Skip security checks for speed

# Completion caching
zstyle ':completion:*' use-cache on
zstyle ':completion:*' cache-path ~/.zsh/cache
zstyle ':completion:*' accept-exact '*(N)'

Disable Expensive Features

# Turn off features that slow startup
unsetopt AUTO_CD
unsetopt AUTO_PUSHD
unsetopt PUSHD_IGNORE_DUPS
unsetopt FLOW_CONTROL

4. Smart Git Branch Caching

Instead of running git commands on every prompt render, we cache branch information:

function git_ps1 {
    if ! git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
        return
    fi

    # Use cached branch info
    if [[ -n $GIT_BRANCH_CACHE ]]; then
        echo -n "($GIT_BRANCH_CACHE)"
    else
        local branch=$(git rev-parse --abbrev-ref HEAD 2>/dev/null)
        if [[ -n $branch && $branch != "HEAD" ]]; then
            export GIT_BRANCH_CACHE="$branch"
            echo -n "($branch)"
        fi
    fi
}

# Clear cache when changing directories
function git_branch_cache_clear() {
    unset GIT_BRANCH_CACHE
}
add-zsh-hook chpwd git_branch_cache_clear

5. Conditional Loading

Only load completions and tools that are actually installed:

### KUBE
if command -v kubectl &> /dev/null; then
    source <(kubectl completion zsh)
fi

### DOCKER
if command -v docker &> /dev/null; then
    fpath=("$HOME/.docker/completions" $fpath)
fi

6. Smart Hooks

Use hooks sparingly and efficiently:

# Auto-load NVM when entering projects with .nvmrc
maybe_nvm_after_cd() {
  local dir
  dir=$(pwd)
  while [ "$dir" != "/" ]; do
    if [ -f "$dir/.nvmrc" ]; then
      load_nvm
      nvm use 2>/dev/null || true
      return
    fi
    dir=$(dirname "$dir")
  done
}
add-zsh-hook chpwd maybe_nvm_after_cd

Key Principles Applied

Lazy Everything: Defer expensive operations until needed
Cache Aggressively: Store computation results and reuse them
Conditional Loading: Only load what's installed and needed
Single File: Avoid sourcing overhead by keeping everything in one organized file
Profile and Measure: Use time zsh -i -c exit to validate improvements

What We Avoided

Oh-My-Zsh: Adds 200-400ms overhead
Multiple sourced files: Each source adds ~5-10ms
Synchronous tool initialization: NVM, pyenv, etc. add 50-200ms each
Complex prompt calculations: Git status checks on every render
Unnecessary zsh options: Many defaults are performance killers

The Outcome

With these optimizations, we achieved:

33ms startup time (top 0.1% performance)
Full development environment (Node, Python, Docker, Kubernetes)
Rich completions for all tools
Maintainable single-file configuration
Zero functionality compromised

The key insight: aggressive lazy loading combined with smart caching and conditional execution can
deliver both speed and functionality without compromise.

Full config

TIL: How Node's event loop actually works.

Tony Miller — Wed, 03 Jul 2024 10:54:06 +0000

It has come to my attention that someone is wrong on the Internet. So here’s yet another page about Event Loop in Node, this time it is actually correct.

Why would you read that?
Here I’m talking about low level details of Node.JS, what it uses for async IO and how different parts of Node.JS (v8 and others) are glued together.
Read on if you want to understand Node.JS and the problem it’s solving a bit better.
There’s a bit of C and C++ examples that help to build understanding of what Node.JS is doing behind the scenes.

Also a warning:

Do not use this to reason about your code in Chrome. Chrome’s even loop is a separate implementation and might work in totally different way. Chrome uses libevent for event loop.

Those are not the droids you’re looking for.

Here are some breaking news for you:

there’s no event loop in Node.JS.
there are no event queue in Node.JS.
and there’s no micro-tasks queue.
there’s no thread in which your JavaScript runs continuously. I don’t know how widespread this one is but some people think that JS is just running all the time in a single thread. It is not. It is on and off, on and off, on and off. We enter V8, we exit V8, we enter V8, we exit V8.

Let’s get the simple stuff out of the sight first.

There’s no micro-tasks queue in Node.JS

Here’s C++ implementation of runMicrotasks():

static void RunMicrotasks(const FunctionCallbackInfo<Value>& args) {
  Environment* env = Environment::GetCurrent(args);
  env->context()->GetMicrotaskQueue()->PerformCheckpoint(env->isolate());
}

This:

env->context()->GetMicrotaskQueue()

is a call into V8 API which actually contains micro-tasks queue. And no, process.nextTick() does not go in there. process.nextTick() goes into this one:

const FixedQueue = require('internal/fixed_queue'); // tasks_queues.js:36
const queue = new FixedQueue(); // tasks_queues.js:55

and it is run by runNextTicks() and it so happens this one plugged into every place whenever your JS runs pretty much manually. We go into a bit of details about that later.
TL/DR: Micro tasks are V8 thing, managed and implemented by V8, Node.JS C++ code just tells it to run them (PerformCheckpoint()) and it has nothing to do with process.nextTick().

There’s no event loop in Node.JS.

But there used to be one 😉.

Another bits of news: Node.JS is low level programming platform that provides you with access to low level Linux, *BSD (including MacOS) and Windows primitives for asynchronous IO. It wraps those primitives with something called libuv which is a library that provides simplified API for those low level OS primitives and it allows you to write your callbacks in JavaScript. Besides the automatic memory management of JS the whole thing is waaaaaay more low level (and honest) than Go’s “gorutines”, any kind of green thread implementation or async/await of C# and Kotlin.

libuv

libuv is the event loop. This is the actual library that implements event loop. It splits it into stages and allows you to create handles for each stage and use those handles to schedule callbacks for those stages. It ref counts those handles and keeps on running as long as there are handles.

For demonstration purposes I have written a very primitive web server and that’s what we’re going to use to learn.

libuv allows you to create as many event loops as you want with the only caveat - one event loop per thread, please and thank you. You don’t have you create your own though, there’s a default:

loop = uv_default_loop();

Now, if you leave it like that it will exit just like Node exits when there are no stuff to poll left. In fact, Node exits exactly because libuv exits. Or even better: it is not Node who exists, it is libuv. So we need to give it something for poll stage:

uv_tcp_init(loop, &server);

Here, I’ve got a handle - server which libuv will register for poll stage. Still, nothing to poll yet, gotta set it up:

uv_ip4_addr("0.0.0.0", PORT, &addr);
uv_tcp_bind(&server, (const struct sockaddr *)&addr, 0);
int err = uv_listen(AS_STREAM(&server), 128, on_new_connection);
if(err) {
    fprintf(stderr, "Listen error: %s\n", uv_strerror(err));
    return 1;
}

and after that we can run the loop:

uv_run(loop, UV_RUN_DEFAULT);

In fact, when Node.JS is initialising it’s doing exactly the same kind of thing. It’s just that it sets up V8, then bootstraps and loads your code. Your code runs and (hopefully) calls some of the APIs that does similar kind of handles and callbacks registration for libuv’s poll stage. It’s just you use Node’s JS API to do that. Only after your file is finished running Node will actually start the event loop.
Now, back to our server, I have registered callback here for new connections - on_new_connection. So whenever new connection comes, libuv will execute this function:

void on_new_connection(uv_stream_t *server, int status) {
    if (status < 0) {
        fprintf(stderr, "New connection error: %s\n", uv_strerror(status));
        return;
    }
    uv_tcp_t *client = malloc(sizeof(uv_tcp_t));
    uv_tcp_init(loop, client);
    if (uv_accept(server, AS_STREAM(client)) == 0) {
        uv_read_start(AS_STREAM(client), alloc_buffer, on_receive);
    } else {
        uv_close(AS_HANDLE(client), on_close);
    }
}

Don’t read too much into it, what we do here is:

accept connection: uv_accept(server, AS_STREAM(client))
start reading: uv_read_start(AS_STREAM(client), alloc_buffer, on_receive)

Now, “start reading” part is interesting because that’s where we register another callback (actually two, but we omit one of them): on_receive:

void on_receive(uv_stream_t *client, ssize_t nread, const uv_buf_t *buf) {
    if (nread > 0) {
        serve_t * serve = malloc(sizeof(serve_t));
        serve->req = http_inflight_parse(http_inflight_new(buf));
        serve->client = client;
        serve->buf = uv_buf_init(serve->reserved, sizeof(serve->reserved));
        fs_req_t *fs = malloc(sizeof(fs_req_t));
        fs->serve = serve;
        uv_fs_open(loop, AS_FS(fs), serve->req->path, O_RDONLY, 0, on_open);
    } else if (nread < 0) {
        if (nread != UV_EOF) {
            fprintf(stderr, "Read error %s\n", uv_err_name(nread));
        }
        uv_close(AS_HANDLE(client), on_close);
        free(client);
    }
}

If we could read something then we do the following:

parse the request (:fingers_crossed: it all came in one TCP buffer, as I said very primitive HTTP server).
figure out which file client wants and open this file:

uv_fs_open(loop, AS_FS(fs), serve->req->path, O_RDONLY, 0, on_open);

Again, we register another callback here: on_open. Now whenever libuv opens the file it’ll run this on_open callback. I won’t print it here, it’s bit too long but what it’s doing is the following:

check if the open result is “ok”, not an error.
look into request to determine the type of the file requested and
pick appropriate HTTP headers.
send the headers.

That’s right, we don’t start reading it here, only sending headers:

uv_write(AS_WRITE(write), serve->client, &serve->buf, 1, on_send);

We politely ask libuv “please dear write this stuff into socket for us and once you’re done tell about it to this guy: on_send". on_send will be called once the buffer is completely written into the socket:

void on_send(uv_write_t *res, int status) {
    write_req_t *write = WRITE_REQ(res);
    serve_t *serve = write->serve;
    if (status) {
        fprintf(stderr, "Write error %s\n", uv_strerror(status));
    }
    if (write->done) {
        uv_close(AS_HANDLE(serve->client), on_close);
        free_serve(serve);
    } else {
        serve->buf.len = sizeof(serve->reserved);
        fs_req_t *read = malloc(sizeof(fs_req_t));
        read->serve = write->serve;
        uv_fs_read(loop, AS_FS(read), serve->file, &serve->buf, 1, -1, on_read);
    }
    free(write);
}

Again, a bit of validation and the important part is that we will ask libuv to start reading the actual file that the client requested:

uv_fs_read(loop, AS_FS(read), serve->file, &serve->buf, 1, -1, on_read);

Aaaaand, you guessed it, yet another call back: on_read. This callback is executed by libuv whenever it is able to read enough data from the file to fill the buffer or whenever it receives EOF or any other error.
on_read then validates the state and again asks libuv to send it down the socket with the same on_send callback.

void on_read(uv_fs_t *res) {
    fs_req_t *fs = FS_REQ(res);
    serve_t *serve = fs->serve;
    uv_fs_req_cleanup(res);
    bool done = false;
    if (res->result < 0) {
        fprintf(stderr, "Read error: %s\n", uv_strerror(res->result));
        serve->buf.len = 0;
        done = true;
    } else if (res->result == 0) {
        serve->buf.len = 0;
        done = true;
        uv_fs_close(loop, res, serve->file, NULL); // synchronous
    } else if (res->result > 0) {
        serve->buf.len = res->result;
    }
    write_req_t *write = malloc(sizeof(write_req_t));
    write->serve = serve;
    write->done = done;
    uv_write(AS_WRITE(write), serve->client, &serve->buf, 1, on_send);
    uv_fs_req_cleanup(res);
    free(res);
}

And so it goes, jumping between those callbacks back and forth.

Stages

libuv runs in stages:

You might’ve seen this picture in Node’s docs but it’s really from libuv docs.

Node

What does it have to do with Node.JS? Well, Node.JS does the same. Look at the starting sequence:

It configures the default loop:

uv_loop_configure(uv_default_loop(), UV_METRICS_IDLE_TIME); // node.cc:1173

Some time later it calls this:

MaybeLocal<Value> LoadEnvironment(
    Environment* env,
    StartExecutionCallback cb) {
  env->InitializeLibuv();
  env->InitializeDiagnostics();
  return StartExecution(env, cb);
}

The interesting part for us is: env->InitializeLibuv();
It registers a handle for timers stage:

CHECK_EQ(0, uv_timer_init(event_loop(), timer_handle()));

It creates a handle for check stage (which runs your setImmediate()s):

CHECK_EQ(0, uv_check_init(event_loop(), immediate_check_handle()));

It also creates a handle for idle stage that may run immediates as well:

CHECK_EQ(0, uv_idle_init(event_loop(), immediate_idle_handle()));

Why two places to run immediates ask you? Well, here’s a plot twist: the event loop actually blocks. In the poll stage it blocks until it is woken up by Linux/BSD/Windows to get data from one of the sockets/descriptors/handles it’s listening. If there are no timers and no idles then it blocks indefinitely. If there are timers it blocks until the next timer. If there’s at least one idle it doesn’t block. So Node.JS uses idle stage and its handle to prevent libuv from blocking and process your setImmediate()s ASAP. Note that the check stage is started (actually “started”, just marked as started) and callback is supplied right away:

CHECK_EQ(0, uv_check_start(immediate_check_handle(), CheckImmediate));

It doesn’t happen for the idle stage handle.
It does some more initialisation, reading bootstrapping node.js and your entry file and running all of that and then finally:

*exit_code = SpinEventLoop(env).FromMaybe(1); //node_main_instance.cc:140

where it does:

uv_run(env->event_loop(), UV_RUN_DEFAULT); //embed_helpers.cc:36

As you can see:

there’s no even loop in Node, it’s from libuv
there is no constant JS running, it is libuv that is running
JS code is only run at the startup and then in the event handlers from libuv.
Node.JS is just a fancy JS wrapper around libuv so you don’t have to chase your lost mallocs around all over of the place and so you don’t get too much SEGFAULTs in prod.

Potential questions (at least I asked myself).

How does process.nextTick() work?

The answer is simple, Node’s C++ side provides an API that is accessible from JS:

void SetupTimers(const FunctionCallbackInfo<Value>& args) {
  CHECK(args[0]->IsFunction());
  CHECK(args[1]->IsFunction());
  auto env = Environment::GetCurrent(args);
  env->set_immediate_callback_function(args[0].As<Function>());
  env->set_timers_callback_function(args[1].As<Function>());
}

Notice, that callbacks for that C++ function are JavaScript functions from V8. And notice that it adds callbacks to immediate handle (which we know runs on check and sometimes on idle)

And it also adds timers callback. So basically it adds callbacks into every stage of even loop when JS code is running. Why is that important? Well because in node.js bootstrapping script will register couple callbacks for those:

// Sets two per-Environment callbacks that will be run from libuv:
// - processImmediate will be run in the callback of the per-Environment
//   check handle.
// - processTimers will be run in the callback of the per-Environment timer.
setupTimers(processImmediate, processTimers);

Those two come from timers.js and they execute runNextTicks() which in turn does two things:

runs micro tasks from V8
runs process.nextTick() callbacks

From what I’m seeing is that JS runs twice per stage, first time in native callback to process IO (runs your code), then another libuv callback that was registered from node.js bootstrapper enters V8 again to run microtasks and process.nextTick(). I might be wrong here, I didn’t dig too deep.

What is libuv locking on during poll stage?
Various operating systems provide various facilities for asynchronous IO.

in Linux it is epoll. In a nutshell it allows you to create over 9000 file descriptors, give it to Linux and tell it “please wake me up whenever anyone has got anything for me”. The API allows you to lock until anything comes, lock for a limited amount of time or simply check without locking.
in *BSD (and MacOS, because it is BSD, duh) it is kqueue. Much better than epoll
and then there’s Windows, the pinnacle of async APIs: IO Completion Ports. This is not a sarcasm. Fun fact, Solaris also used approach of IO Completion Ports.

All of those APIs (except may be for Windows? not sure here) provide you with facilities to lock and wait until something comes, lock for limited amount of time or just check and not to lock at al.
Because libuv is an IO library it only makes sense that it locks and waits for new IO to happen.

What happens with file IO? I heard there’s a thread pool there as well?

Well, here’s a thing: async File IO API in POSIX SUCKS.

So instead of dealing with its quirks and features libuv simply does it synchronously but offloads it to a thread pool that is provided by default and by default the size of this thread pool is 4. Btw, it is not only for File IO, you can throw any long running task to run there. There are C++ encryption libraries for Node.JS that offload CPU intensive encryption onto this thread pool to avoid blocking the event loop.

How then libuv runs callbacks, they are supposed to run on the main thread, aren’t they?

What libuv does is:

if it is Unix, setup a pipe, epoll and kqueue support pipes, this pipe’s file descriptor is one of the descriptors to block on during poll stage. When file operation is completed on a worker thread it writes into this pipe to wake up the event loop and call handlers.
if it is Windows do the same with named pipe. Technically IO Completion Ports allow proper async File IO but as far as I understand libuv still does the same pipe trick, probably for the sake of consistency and simplicity.

Note: As far as I'm aware, support for io_uring has landed to libuv and in theory it is capable of doing proper async file io, at least on Linux. I'm don't know if Node is using it yet.

Honesty

Recently it became fashionable to speak about green threads, co-routines and etc. as primitives of async programming.

I loath those. Because they are fake.

I’m a firm believer that everything should be implemented in the right way and the only way to implement those in the right way is to have support from operating system.

Unfortunately not Linux nor MacOS and nor any kind of BSD I know provide you with facility to interrupt your own running code. So every time someone’s speaking about “green threads” or “coroutines” or anything pretending to be doing preemptive multitasking in user space - they are lying. What they are offering is one of two hacks:

you write your own complier and runtime and you sprinkle the generated code with “yield”s. This is what Go used to do. The least “bad” way but still produced some interesting quirks.
you setup a timer and a signal so the Kernel will interrupt your process normal execution and will call the function that you specified as signal handler. This comes with several “gotchas” and people have written about those extensively. Recently Go has switched to this mechanism and avoids the issue with “for” loop from the first bullet point. Unfortunately for this to work you have to hack your stack and rewrite instruction pointer registers so when your signal handler is done running and kernel resumes you process it end up in scheduler instead of previously running function. This is really a nasty nasty nasty hacking. Stack and instruction pointer belongs to Kernel, not us.

There’s one operating system that allows you to do that: Windows

there’s support for “fibers” which are user-space threads
there’s actually a framework to replace Kernel scheduler with your own for your process which unfortunately is dead in Windows 11. Or may it’ll live in Server versions?

So this is why I love Node so much - it is the only platform that doesn’t seem to hide and pile up hacks and gotchas when it comes to async io and just lets you use OS primitives through a somewhat simple programming language and comfortable interfaces.

Code references

I have checked out commit 331088f4a450e29f3ea8a28a9f98ccc9f8951386 so if there are any changes in the files after that then line numbers in code references might’ve drifted.

TIL: get strongly typed HTTP headers with TypeScript

Tony Miller — Mon, 20 Dec 2021 07:12:52 +0000

Goal

I’m a developer of a backend framework. It is written in TypeScript. I want to:

Hide the actual request object (http.IncomingMessage) from my users
Yet provide my users with access to HTTP headers on the request (http.IncomingHttpHeaders).
Provide IntelliSense (auto-completion) so it is easier to find headers people want to use.
Provide compile-time checking that there’s no type in a header.
Do not limit my users as to which headers they can use, so the list of headers must be extensible from their services.

Turns out all of that is possible.

Implementation

Consider http.IncomingHttpHeaders interface:



interface IncomingHttpHeaders {
    'accept-patch'?: string;
    'accept-ranges'?: string;
    'accept'?: string;
    …
    'warning'?: string;
    'www-authenticate'?: string;
    [header: string]: string | string[] | undefined;
}

The problem with it that while it does have header names hardcoded it:

does not provide a way to extend that list.
provides index signature, which means all type safety goes out the window.

So in order to conceal the actual request from my users, I’ve got a class called Context and I hand out instances of that to handlers for each request:



export class Context {
    constructor(private req: http.IncomingMessage) { }
    …
    getHeader(name: ?) {
        return req.headers[name];
    }
}
…

What we want to do is to introduce some kind of type instead of ? so that it allows only those headers from http.IncomingHttpHeaders that are hard-coded, we will call them “known keys”.

We also want our users to be able to extend this list easily.

Problem 1

Can’t use simple type StandardHeaders = keyof http.IncomingHtppHeaders because the interface has index signature, that resolves into StandardHeaders accepting anything so auto-completion and compile-time checking doesn’t work.

Solution - remove index signature from the interface. TypeScript 4.1 and newer allows key re-mapping and TypeScript 2.8 and newer has Conditional Types. We only provide 4.1 version here:




type StandardHeaders = {
    // copy every declared property from http.IncomingHttpHeaders
    // but remove index signatures
    [K in keyof http.IncomingHttpHeaders as string extends K
        ? never
        : number extends K
        ? never
        : K]: http.IncomingHttpHeaders[K];
};

That gives us copy of http.IncomingHttpHeaders with index signatures removed.

It is based on the fact that ‘a’ extends string is true but string extends ’a’ is false. Same for number.

Now we can just:



type StandardHeader = keyof StandardHeaders;

That’s what VSCode thinks about StandardHeader:

Nice type literal with only known headers. Let’s plug it into getHeader(name: StandardHeader) and try to use it:

Auto-completion works and compilation breaks if we type something wrong there:

Problem 2.

We’re a framework, this set of headers is pretty narrow, so we need to give people ability to extend it.

This one is easier to solve that the previous one. Let’s make our Context generic and add several things:

limit generic to string type literals
provide a sensible default



export class Context<TCustomHeader extends string = StandardHeader> {
    constructor(private req: http.IncomingMessage) { }
    …
    getHeader(name: StandardHeader | TCustomHeader) {
        return req.headers[name];
    }
    …
}

Ok, now our users can write something like this:



const ctx = new Context<'X-Foo' | 'X-Bar'>(...);
const foo = ctx.getHeader('X-Foo');
const bar = ctx.getHeader('X-Bar');

And it will auto-complete those headers:

And also it includes them into compile-time check:

Further improvements

Because we’re a framework, users won’t be creating instances of Context class themselves, we’re handing those out. So instead we should introduce a class ContextHeaders and replace getHeader(header: StandardHeader) with generic method headers< TCustomHeader extends string = StandardHeader>: ContextHeaders<StandardHeader | TCustomHeader>

That is left as exercise for reader =).