Understanding SQEs and CQEs with kqring on FreeBSD

If you have used an event loop, you have probably written some variation of this:

1. Wait for a socket to become readable.
2. Read from it.
3. Process the bytes.
4. Go back to waiting.

A completion-based I/O interface moves some of that work across the application/kernel boundary. Instead of asking which sockets might be ready, the application can describe the reads it wants and collect their results later.

kqring is an experimental FreeBSD kernel module that exposes an io_uring-oriented interface using FreeBSD's own kernel facilities. It is a useful example for understanding what submission and completion queues actually do, and why their memory-lifetime rules matter.

Readiness and completion answer different questions

For a network receive, a readiness notification answers:

Can this socket probably be read without waiting?

The application still needs to perform the receive. Another thread might consume the available data first, so the application also needs to handle readiness changing before it acts.

A receive completion answers:

What happened to the receive that was submitted?

That result might be 100 bytes read, an error, or end of file. The operation itself has been attempted or completed according to the interface's contract.

This distinction describes the network paths, not everything kqueue can do. FreeBSD's kqueue also supports other event sources, including asynchronous I/O completion notifications.

An SQE describes work

SQE stands for Submission Queue Entry. It contains the operation and its arguments.

Conceptually, a read request might look like this:

operation: READ
file:      descriptor 7
buffer:    address of a 4096-byte buffer
length:    4096
user_data: 42

A real program also has to set up the ring, publish the entry with the right memory ordering, and tell the kernel about it. A library such as liburing handles that bookkeeping; the real code is below.

The user_data value belongs to the application. It could be an integer identifying a request or a value associated with application state. The kernel returns it with the result so the application can match the completion to the right request.

A CQE reports a result

CQE stands for Completion Queue Entry. Its important fields are:

Field	Meaning
user_data	The value associated with the submitted request
res	The operation's result, such as bytes transferred or a negative error
flags	Additional information about the completion

If the example read transfers 100 bytes, a completion could contain:

user_data: 42
res:       100
flags:     0

An error is reported through a negative result. A read that reaches end of file normally reports zero bytes.

The basic exchange is:

Application prepares SQEs
          |
          v
Kernel consumes requests and performs or schedules work
          |
          v
Kernel posts CQEs
          |
          v
Application collects results and updates its own state

Completions need not arrive in submission order. Matching by user_data lets the application manage requests that finish at different times.

The same read in C

Here is that read with liburing, built against kqring's liburing 2.9 port and saved as readone.c. Error handling is trimmed to the essentials:

#include <liburing.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

int
main(void)
{
    struct io_uring ring;
    struct io_uring_sqe *sqe;
    struct io_uring_cqe *cqe;
    char buf[4096];
    int fd, ret;

    ret = io_uring_queue_init(8, &ring, 0);
    if (ret < 0) {
        fprintf(stderr, "queue_init: %s\n", strerror(-ret));
        return (1);
    }
    fd = open("/etc/hosts", O_RDONLY);
    if (fd < 0) {
        perror("open");
        return (1);
    }

    /* Describe the read: an SQE. */
    sqe = io_uring_get_sqe(&ring);
    io_uring_prep_read(sqe, fd, buf, sizeof(buf), 0);
    io_uring_sqe_set_data64(sqe, 42);
    io_uring_submit(&ring);

    /* Collect its result: a CQE. */
    ret = io_uring_wait_cqe(&ring, &cqe);
    if (ret < 0) {
        fprintf(stderr, "wait_cqe: %s\n", strerror(-ret));
        return (1);
    }
    printf("user_data=%llu res=%d\n",
        (unsigned long long)io_uring_cqe_get_data64(cqe), cqe->res);
    io_uring_cqe_seen(&ring, cqe);

    close(fd);
    io_uring_queue_exit(&ring);
    return (0);
}

From the repository's liburing directory, after make and with kqring.ko loaded:

cc -O2 -I work/liburing-liburing-2.9/src/include -I freebsd/include \
    -include freebsd/include/config-host.h -o readone readone.c work/out/liburing.a
./readone

On a FreeBSD 15.1 machine whose /etc/hosts is 1,035 bytes, it prints:

user_data=42 res=1035

The source is the same as it would be on Linux. What differs is underneath: liburing's three system calls go to kqring's, and values such as O_RDONLY and the error numbers in res are FreeBSD's.

One request can produce several completions

A multishot request can remain active after producing a result. For example, a multishot receive can produce completions for successive receives without requiring the application to submit an entirely new receive request each time.

IORING_CQE_F_MORE tells the application that the request remains active and can produce more completions. A completion without that flag ends the multishot request.

This changes the application's bookkeeping. It must not discard the request state just because the first CQE arrived. It also needs a buffer strategy appropriate for multiple results, such as provided buffers, and must handle the conditions that terminate the request.

The kernel still needs somewhere to execute the work

A ring is a communication mechanism. It does not make every operation cheap or automatically nonblocking.

kqring uses several execution paths:

• Ready socket operations can execute during the submitter's entry into the kernel.
• Sockets that would block wait for a socket callback that queues a retry.
• Eligible cached file reads and ready pipes can execute inline.
• Operations that need to wait can run on kernel workers associated with the owning process.
• Supported disk-device operations can use the native bio path.

The code also uses kqueue internally. Its non-socket polling adapter, called kpoll, registers events for objects such as pipes and eventfds and converts them into completions.

So kpoll and kqueue are not competing choices here. Kpoll is the part of kqring that connects internal kqueue events to the submission/completion model.

Batching can save work

Suppose many sockets already have data available. A readiness loop can collect their events together, but the application still performs the receives. A completion interface can describe those receives in advance and collect their results as a batch.

The benefit depends on the workload. Preparing queue entries, holding references, scheduling workers, and posting results all have costs. One operation per kernel entry may add overhead without enough batching or overlap to recover it.

That is why an evaluation needs more than the fastest benchmark row. Useful questions include:

• How many operations are submitted together?
• Is the data already available, or does the request need to wait?
• Does execution require a worker handoff?
• How much CPU time and memory does the workload consume?
• Which event marks the operation as finished for the measurement?

An interface can make one workload cheaper and another more expensive. The project documents both favorable and unfavorable microbenchmark results; they should not be read as a claim that every application becomes faster.

A completion also has an ownership meaning

The buffer address in a request is more than an argument. It is a promise that the memory remains usable for as long as the operation needs it.

Copying the request's metadata does not copy the payload. Once submission has consumed an iovec description, for example, its metadata may be reusable under the interface contract while the described data buffer still has to remain available.

Cancellation adds another distinction. In kqring, a successful asynchronous cancellation result can arrive before the target worker has returned. The target's final completion is the event to wait for before reusing its ordinary I/O buffer. Requesting interruption is not the same as proving the worker has stopped.

Zero-copy sends have a further boundary. The primary send result can arrive while outgoing network buffers still retain references to the application's pages. A later release notification tells the application when it may reuse that transmission memory. Neither notification means that the remote application has processed the data.

For each request, ask: which completion actually ends the resource's lifetime obligation?

Compatibility has limits

kqring follows io_uring structures and operation identifiers, but uses native FreeBSD values for flags such as O_* and MSG_*, and native error numbers in completion results.

Its public header supplies three entry points:

Linux interface	Native kqring wrapper
io_uring_setup()	kqring_setup()
io_uring_enter()	kqring_enter()
io_uring_register()	kqring_register()

The repository includes an overlay for liburing 2.9. This can preserve the library-level programming model, but Linux-specific assumptions in an application still need review. This is a native source-porting interface, not a way to execute an unchanged Linux binary.

There are behavioral differences too. Rings belong to one process, and a forked child cannot generally submit work through its parent's ring. Signal-mask handling and exec behavior also differ. A familiar operation name does not remove the need to read those contracts.

Where the implementation stands

The current project documentation identifies FreeBSD 15.1-RELEASE, amd64, with a GENERIC kernel and matching kernel sources as the supported system. Other releases and architectures have not been tested according to that documentation.

The module is an experimental preview and has not had an independent security audit. Functional tests and compatibility tests help check specific behaviors; they do not establish the absence of kernel concurrency or memory-safety bugs.

The source repository contains the implementation, architecture notes, compatibility differences, tests, and benchmark procedure. For the version discussed here, see the project overview at release 0.1.0.

The useful mental model is a request with an explicit lifetime: the SQE describes it, the kernel chooses how it executes, and CQEs report the state transitions the application needs to observe. Understanding those transitions is what makes the interface usable safely.

Disclosure: this article was drafted with AI assistance, and kqring itself was developed with AI assistance.