DEV Community: Marius-Florin Cristian

gRPC over QUIC: faster seeks, slower bulk

Marius-Florin Cristian — Sun, 19 Jul 2026 16:33:05 +0000

Abstract. Notes from testing whether to move KeibiDrop, a post-quantum encrypted peer-to-peer filesystem, from gRPC over TCP to gRPC over QUIC. The product reads remote files on demand, so the goal is to survive an IP change, keep seek latency low, stay correct, then go fast, in that order. Measured on a 12-core macOS laptop, a 4-core Linux VPS, and a real link between a laptop in Barcelona and the VPS at 72 ms round trip. Findings: on macOS the QUIC transport is syscall-bound at about 137 MB/s because the platform has no UDP segmentation offload, and the undocumented sendmsg_x batching syscall lifts it to about 700 MB/s; a teardown deadlock traced to closing a QUIC stream concurrently with a write, fixed with a stream reset; connection migration moves a live gRPC channel to a new socket, verified with packet counters; on the real WAN a single QUIC stream loses to TCP on bulk transfer by up to 16x under Wi-Fi loss, but a random seek under saturating prefetch lands in 109 ms on QUIC against about 3 seconds on TCP, a 28x difference, because the seek gets its own stream instead of queuing behind the prefetch. The design that follows: the interactive on-demand path over QUIC for seek isolation and migration, bulk on lossy links over TCP.

The question, and what we optimize for

KeibiDrop is a peer-to-peer filesystem. You mount a remote peer's files and read them on demand. Open a 40 GB video, seek to the middle, and only the bytes you touch cross the wire. The transport underneath is gRPC over our own encrypted TCP connection.

TCP has one problem we cannot fix from userspace. When the 5-tuple dies, the connection dies. A laptop that moves from café Wi-Fi onto a phone hotspot changes its IP, and the transfer stops. QUIC identifies a connection by a connection ID rather than the 5-tuple, so it can migrate across an address change with the stream intact. We wanted to know whether gRPC could run over QUIC and keep that property.

We optimize for the experience of an on-demand filesystem, not for bulk throughput in MB/s. The order of priorities is: survive an IP change, keep seek latency low, stay correct, and then speed. All the numbers below are measured. Where a number surprised us, we looked for the cause before believing it. The hardware is a 12-core macOS laptop, a 4-core Linux VPS, and a real link between a laptop in Barcelona and that VPS at 72 ms round trip.

Getting gRPC onto a QUIC stream

gRPC already speaks net.Conn. In KeibiDrop it runs over our encrypted SecureConn, which is a net.Conn wrapping a net.Conn, so QUIC slots in underneath the same seam. The adapter that makes one QUIC stream look like a net.Conn is small, because a quic.Stream is already a reliable, ordered byte stream with no framing to add:

type quicConn struct {
    *quic.Stream            // Read/Write/deadlines come from the stream
    conn *quic.Conn         // for the two address methods the stream lacks
}

func (c *quicConn) LocalAddr() net.Addr  { return c.conn.LocalAddr() }
func (c *quicConn) RemoteAddr() net.Addr { return c.conn.RemoteAddr() }

One detail is worth stating now, because it returns in the deadlock section. quic.Stream.Close() is a graceful, send-only half-close, and quic-go documents that it must not be called concurrently with Write. A net.Conn.Close has to tear the whole connection down. The correct call is CancelWrite, a stream reset, not Close. We used Close first and paid for it later.

With the adapter, a listener that hands one stream up as a conn, and gRPC's NewClient pointed at a passthrough:/// target so it skips DNS, a gRPC call completes over QUIC. Unary, server-streaming, and concurrent calls all work, verified byte for byte and clean under the race detector. gRPC's own TLS stays off. QUIC's mandatory TLS 1.3 sits underneath with a throwaway certificate, because the real confidentiality is a post-quantum handshake we add on top (ML-KEM-1024 and X25519, the same primitives KeibiDrop already uses; QUIC's TLS 1.3 is not post-quantum and ours is).

Throughput on macOS

The first benchmark on the macOS laptop streamed a large file both ways on loopback:

transport	throughput
gRPC over TCP	1650 MB/s
gRPC over QUIC	140 MB/s

Twelve times slower. That is large enough to want an explanation, so we profiled it. A raw QUIC stream with no gRPC gave 137 MB/s, so the gRPC layer was not the cause; the QUIC transport was. The CPU profile showed where the time went:

56.67%  syscall.rawsyscalln          # the syscall itself
33.33%  ...sendmsg   (one WriteMsgUDP per ~1400-byte packet)
23.44%  ...recvmsg
 0.6 %  crypto (AES-GCM)             # negligible

Fifty-seven percent of the time was in sendmsg and recvmsg. Crypto, which people assume is the cost of an encrypted transport, was 0.6 percent. The reason is that macOS has no UDP Generic Segmentation Offload. On Linux, quic-go hands the kernel a 64 KB buffer and the kernel splits it into packets, so one syscall sends dozens. On macOS there is no such offload, so quic-go sends one sendmsg per 1400-byte packet. At 137 MB/s that is about 100,000 send syscalls per second and as many receives.

We checked this by running the same code on the Linux VPS, which has GSO. The VPS is the slower machine (its loopback TCP is 3.4x slower than the laptop's), and its QUIC still ran at 272 MB/s against the laptop's 137. Normalized against each machine's own TCP, QUIC went from 2 percent on macOS to 14 percent on Linux. So the macOS number is a platform limitation, not a design or crypto cost.

Batching sends with sendmsg_x

macOS has no GSO and no sendmmsg. The syscall-batching primitives that fix this on Linux do not exist here. It does have sendmsg_x, a Darwin syscall that sends many datagrams in one call, similar to Linux sendmmsg. It is undocumented. The syscall number is in the SDK (SYS_sendmsg_x = 481), but the struct msghdr_x it takes is not in any header. You declare it yourself from the XNU source and get the byte layout right, or the kernel reads garbage.

We wrote a small program that prints the struct offsets, checks them with a real batched send the receiver verifies, and benchmarks batch size against raw UDP send throughput:

batch	syscalls	throughput
1 (one per packet)	500,000	390 MB/s
8	62,500	675 MB/s
32	15,625	701 MB/s

Batching lifts the ceiling from 390 to about 700 MB/s and cuts the syscall count by 32 times. It levels off at batch 32, where the cost moves from syscall count to per-packet kernel work. So there is a real macOS lever here, about 1.8 times, with no root access. It helps in a second way. The 137 MB/s QUIC transfer used about five CPU cores, all of it the per-packet work TCP hands to the kernel, and in KeibiDrop that CPU competes with the FUSE filesystem, which is busy with encryption and chunking. Fewer syscalls per byte means fewer cores taken from the filesystem.

The primitive is now in our quic-go fork as the Darwin counterpart of the Linux GSO path, with the verified struct layout, ready to wire into the send loop.

The teardown deadlock

The test suite passed under the race detector and hung about half the time without it. That combination points at a timing-dependent bug rather than a random flake, because the race detector slows the code down. The goroutine dump named the test, a client cancelling a server-stream, and showed Server.Stop blocked while a QUIC write stayed stuck.

Two guesses were wrong, and both were quick to disprove by measuring. Sizing the download to fit the flow-control window did not help; it still wedged. A short idle timeout had no effect, because a connection that is actively trying to send is not idle. Reading the source settled it.

gRPC's http2Server.Close closes the underlying net.Conn, which is our quicConn.Close, which called quic.Stream.Close(), the graceful half-close from the adapter section, the one quic-go says must not be called concurrently with Write. gRPC's writer goroutine was concurrently blocked in a Write on that stream. The contract violation meant the write never returned, the conn never finished closing, and Server.Stop waited forever.

func (c *quicConn) Close() error {
    c.Stream.CancelRead(0)   // STOP_SENDING
    c.Stream.CancelWrite(0)  // RESET_STREAM: safe against a concurrent Write, and unblocks it
    return c.conn.CloseWithError(0, "")
}

Replacing Close with CancelWrite turned a 50 percent hang into ten green runs out of ten. The general point: when the race detector flips a test from pass to fail, look for the timing bug rather than adding sleeps. And a lightweight atomic counter is better than a print for diagnosing one, because the print changes the timing you are measuring. Ours made the test pass for a while and pointed us the wrong way.

Migration

Migration is the reason we started. quic-go exposes it directly: AddPath a new socket, Probe to validate the path, Switch. Getting it to fire took reading quic-go's own test. The client has to keep sending after the switch so the peer migrates too, and you must not close the old transport to prove it, because that tears the connection down.

We checked it with per-socket packet counters, moving a live gRPC channel to a new UDP socket mid-transfer:

migration: local addr 127.0.0.1:52598 -> 127.0.0.1:59126
path 1 writes +0        # old path goes silent
path 2 writes +90, reads +92   # new path carries both directions

After the switch the old path sent nothing and the new path carried both directions. The gRPC channel kept running. On the real product this is a laptop changing networks during playback without a stall.

The real WAN test

Loopback measures a transport's overhead, not its behaviour on a real network. So we ran it for real: a laptop on hotel Wi-Fi in Barcelona, about 45 Mbps down and 72 ms round trip, downloading from the VPS. No emulation and no offload tricks, so both TCP and QUIC send ordinary packets end to end.

concurrent streams	QUIC	TCP
1	32.8 s	2.0 s
4	5.5 s	1.7 s
16	4.7 s	1.7 s

A single QUIC stream dropped to about 2 Mbps of a 45 Mbps link while TCP filled it. The cause is real Wi-Fi packet loss and quic-go's congestion control, which backs off hard, while Linux kernel TCP handles the same loss well. We had gone in half-expecting the common claim that QUIC does better on lossy links. On a real lossy link the opposite held. For plain bulk transfer on a bad link, TCP won.

On its own this result argues against QUIC. We nearly stopped here.

Seek latency under prefetch

An on-demand filesystem is not used by streaming a file end to end. It is used by seeking. You open a video, jump to a position, jump again. Each jump is a small read at a random offset while a prefetch pulls bulk data in the background. The number that matters is how long that small read takes while the prefetch saturates the link.

On TCP, the prefetch and the seek share one byte stream, so the seek's bytes sit behind the prefetch already queued in the send buffer. That is head-of-line blocking, and it is the playback stutter we had seen before as self-congestion. On QUIC the seek gets its own stream, and the library interleaves its packets with the bulk, so it does not queue behind it.

We measured it on the same Barcelona link. Thirty small 16 KB reads under a saturating bulk stream:

	p50	p99	max
QUIC (seek on its own stream)	109 ms	322 ms	387 ms
TCP (seek behind prefetch)	3066 ms	4827 ms	4924 ms

A seek lands in about 109 ms on QUIC, roughly one round trip. On TCP it takes about three seconds, behind the prefetch. For someone scrubbing through a video the first is usable and the second is not, and the gap is larger than the bulk-throughput gap from the previous section.

This fits the earlier results together. QUIC's advantage for us is latency isolation, a seek that does not wait behind a prefetch, together with migration, a transfer that survives a network change. For an on-demand seekable filesystem that trade is worth it. TCP can match the fast lane only with a second connection per read, which is the data-channels idea we had been circling; QUIC gives it per stream, with shared congestion control.

What we build

The measurements point at a specific design. The interactive on-demand path runs over QUIC. Seeks and prefetch on separate streams is where the seek result comes from, and migration comes with it. Plain bulk on a lossy link stays on TCP, where quic-go's congestion control loses. On fast, clean links the macOS syscall ceiling is real, and sendmsg_x batching is the lever that lifts QUIC to about 700 MB/s while freeing CPU for the filesystem. Control traffic such as heartbeats and metadata rides QUIC, since it is small and it is the channel that has to survive a network change.

None of this was obvious at the start, and several steps were the opposite of what we expected. Measuring, rather than reasoning from what QUIC is supposed to be good at, changed the answer twice: once when the slow macOS transport turned out to be a syscall cost with a fix, and once when the throughput test's verdict flipped as soon as we measured seek latency.

The experiments are small and self-contained: the gRPC-over-QUIC adapter, the sendmsg_x program, and the two WAN benchmarks.

Measured on a 12-core macOS laptop, a 4-core Linux VPS, and a real link between a laptop in Barcelona and the VPS at 72 ms round trip. Originally published at keibisoft.com/blog/grpc-over-quic-seek-latency.html.

KEIBIDROP: Instant file transfer Mac to Windows

Marius-Florin Cristian — Sun, 28 Jun 2026 09:20:32 +0000

(Left side macos, rightside windows)

Work on shared files instantly.
No more waiting for uploads or downloads.

Open a 40 GB file and start working right away.
Your edits, available instantly.

This being said, this post is short, and its about the video :D

KEIBIDROP: Romania to Singapore Benchmarks

Marius-Florin Cristian — Sun, 14 Jun 2026 08:57:37 +0000

Measuring an encrypted peer-to-peer filesystem over a 200 to 330 ms intercontinental link to a Windows host: throughput, connect time, and FUSE on-demand latency, and what bounds them.

Abstract. Measurement of KeibiDrop, a post-quantum encrypted peer-to-peer filesystem, over a high-latency intercontinental link. Two Linux peers in Romania run against one Windows Server peer in Singapore, at 200 ms (datacenter to datacenter) and 330 ms (residential) round-trip. The Windows host is IPv4 only, so all traffic traverses the TCP bridge. Three dimensions are reported: bulk throughput, connect time, and FUSE on-demand read latency. Each cell is the median over repeated trials, with the per-table trial count n shown in the tables, all peers on commit b5a7a3e. Findings: 30 to 38 MB/s into the Windows host and 13 to 25 MB/s out; connect time about 6.3 s, dominated by an IPv6 dial timeout before bridge fallback; FUSE first byte under 1.5 s with sequential on-demand read near 1 MB/s. The throughput ceiling is the bandwidth-delay product, not link capacity or CPU. FUSE mode did not start on the Windows host on this build; a fix is in review.

Setup

Three machines, two pairings. One end is always the Singapore box.

machine	details
SG	Singapore, Windows Server, 4 vCPU / 8 GB, WinFsp installed. IPv4 only.
VPS	Timisoara, Linux. Also runs the relay and the TCP bridge.
laptop	Iasi, Linux, residential link about 500 to 600 Mbps wired (62 to 75 MB/s).
Pairing A	VPS to SG, about 200 ms RTT, datacenter to datacenter.
Pairing B	laptop to SG, about 330 ms RTT, home to datacenter.

Method: each cell is the median across repeated trials, with the trial count n shown in every table: 5 at 10 and 100 MB, 3 at 1 GB, 1 at 5 and 10 GB for bulk; 10 for handshake; 3 for FUSE. All three peers run the same commit. X to Y means X sends and Y receives. Bulk transfer uses kd add then a timed pull, no FUSE in the path.

Bulk throughput

Pairing A, VPS to SG, about 200 ms

Transfer	Size	n	Median MB/s	p95
SG to VPS	10 MB	5	25.6	28.0
VPS to SG	100 MB	5	35.5	41.5
VPS to SG	1 GB	3	37.6	40.4
SG to VPS	1 GB	3	13.2	13.6
VPS to SG	5 GB	1	34.6	34.6
SG to VPS	5 GB	1	8.6	8.6
VPS to SG	10 GB	1	18.8	18.8
SG to VPS	10 GB	1	12.9	12.9

Pairing B, laptop to SG, about 330 ms

Transfer	Size	n	Median MB/s	p95
SG to laptop	10 MB	5	20.4	21.8
laptop to SG	100 MB	5	29.6	30.5
laptop to SG	1 GB	3	29.0	29.3
SG to laptop	1 GB	3	13.4	15.1
laptop to SG	5 GB	1	32.9	32.9
laptop to SG	10 GB	1	31.3	31.3

Handshake

Time to connect to the Singapore box, parsed from the joiner's own log (relay lookup to connected).

Connects to SG from	n	Median ms	p95	min	max
VPS (pairing A)	10	6316	6473	6094	6476
laptop (pairing B)	10	6220	6481	5977	6507

FUSE on-demand

A Linux peer mounts the virtual drive and reads a file hosted on SG. Content streams on demand over the link. ttfb is the time to the first byte (the first 512 KB chunk). The mount is Linux because the Windows mount has a bug, covered below.

Mount reads from SG	Size	n	ttfb median ms	p95	read MB/s
VPS mount (pairing A)	10 MB	3	632	1242	0.9
VPS mount (pairing A)	100 MB	3	509	635	1.0
laptop mount (pairing B)	10 MB	3	709	1423	0.8
laptop mount (pairing B)	100 MB	3	482	549	0.8

What the numbers say

The limit is round-trip time, not bandwidth and not CPU. The home link is 500 to 600 Mbps, which is 62 to 75 MB/s, and the best run used about half of it. The box is a 4 vCPU machine that stayed near idle, and ChaCha20 runs at gigabytes per second per core, so encryption is not the wall either. The wall is the bandwidth-delay product: one gRPC stream with a 16 MiB window cannot keep a 330 ms pipe full, because filling it needs roughly 18 MB in flight at the home link rate. A bigger window or parallel pull streams would close that gap.

The Singapore box sends at about half the rate it receives. At 1 GB on pairing A it pulls in 37.6 MB/s but pushes out 13.2 MB/s, and the same shape holds across sizes and on pairing B. The receive path is fine. The send path is where the work is.

The 6.3 second connect is one timeout. The box has no IPv6, so the direct IPv6 dial always fails and waits out its deadline before the bridge takes over. Skipping the direct dial when a peer advertises no IPv6 removes most of that. The same change helps phones, which are usually IPv4 only and hit the same wait.

On-demand FUSE is for opening and seeking, not for streaming a whole large file. The first byte comes back in half a second to a second and a half, which is fine for opening a document, scrubbing a video, or reading part of a dataset. Sequential read settles near 1 MB/s because each 512 KB chunk costs one to three round trips at this latency. For a whole large file, a bulk pull at 30 MB/s or a prefetch is the right path.

Defects identified

FUSE mode did not start on the Windows host on this build. host.Mount returned false and the daemon fell back to no-FUSE: kd list showed the files, but the mount point stayed an empty folder and neither local nor remote files surfaced. The cause is the mount point being created before WinFsp runs. WinFsp takes the mount point as a drive letter (for example K:), an auto-assigned drive (*), or a directory it creates itself, and it rejects a path that already exists. PR #181 removes the pre-creation on Windows. No-FUSE transfer is unaffected, which is why the FUSE tables above use Linux mounts.

One 10 GB transfer looked like it stalled at 12 MB. A re-run moved 3.7 GB in 4 minutes at a steady 15.6 MB/s, so it was a blip on the residential link, not a defect. Not filed.

Planned changes

Skip the direct dial when a peer advertises no IPv6. This removes most of the 6.3 s connect time on the common case and applies to mobile, which is also IPv4 only.

Land the Windows mount fix (PR #181) so FUSE starts on the Windows host and surfaces files.

Add a push-based send to remove the asymmetry where the host sends at half its receive rate.

For bulk throughput, raise the window or pull with several streams. Tradeoff: saturating the link degrades on-demand read latency, so this stays opt-in for bulk transfers rather than the default for the interactive mount.

Medians over the per-table trial counts (n), all peers on commit b5a7a3e. Pairing A is VPS to SG at about 200 ms. Pairing B is laptop to SG at about 330 ms. All traffic over the TCP bridge.

KEIBIDROP: Also on r/selfhosted

Marius-Florin Cristian — Mon, 01 Jun 2026 19:20:41 +0000

I made an open source tool that allows two people to directly connect via WAN, or LAN, and mount a virtual mountpoint.

(The first video in link URL shows side by side my, and my mac and brothers ubuntu use the shared mounted filesystem)

https://github.com/KeibiSoft/KeibiDrop

The idea is that two people establish a connection, and mount virtual folders. Metadata is synced automatically when things are dragged in the virtual folder, and it supports reading with offset, which means that if Alice shares a 10GB video, or folder with files, Bob can see it in a few seconds, and start playing it.
(Here is a fresher video where I add a Popeye 600 MB Video on my iphone, and start playing it with QuickTime and skim through it before the download completes)
https://youtu.be/UHw1WtkZ8dE

The tool has been in development for a little over a year, and just made it public last month.

There are a series of technical blog posts about the pain points, optimization paths, and how we set benchmarks in order to make work fast at
https://keibisoft.com/blog.html#keibidrop

The most notable thing is that once Alice and Bob are connected they can use programs as if the filesystems were native. (thus Alice and git clone a repo, and bob git commit, or checkout instantly on the same repo).

It is still in development, and as me and my brother use it and polish it we also fix any issues that arrise.

Happy to discuss about it, and things that I learned while grinding on the problem :D

KEIBIDROP: Every tool just works

Marius-Florin Cristian — Mon, 01 Jun 2026 03:48:28 +0000

We connected two machines with KEIBIDROP and ran every program we could think of on the shared virtual folder. Compilers, version control, video processing, databases, scripting languages, security scanners. Everything worked. File integrity held across every operation.

KEIBIDROP presents the peer's files through the operating system's filesystem interface. Programs do not know the files are remote. They read and write to a folder. The data arrives from the other machine over an encrypted channel.

How the shared folder works between two machines

Machine A adds files to the shared folder. They appear on Machine B within a second. Machine B opens them, compiles them, processes them, and saves the output back to the shared folder. Machine A sees the output immediately. No upload, no download, no sync button. Both machines read and write to the same folder in real time, over an encrypted peer-to-peer connection.

We tested this with 45+ programs. Alice writes source code, Bob compiles it. Bob writes analysis results, Alice reads them. Alice has a video, Bob creates thumbnails from it. Every tool that reads files from a folder works. The programs do not know the files are on another machine.

Build tools

Tool	Test	Result
git	Clone repo, run git status and git log from the other side	Works
make (C)	Compile 87 source files inside the shared folder	Works
cargo (Rust)	cargo build --release on the shared folder	Works
go build	Init module, build binary on the shared folder	Works
swiftc	Compile Swift program on the shared folder	Works
xelatex	Compile LaTeX slides across mounts	Works

Media tools

Tool	Test	Result
ffprobe	Inspect 192 MB video from peer's folder	Works
ffmpeg	Extract clip, create thumbnail grid from peer's video	Works
ImageMagick	Read PNG, resize, identify 2048x5568 tileset	Works

Data and databases

Tool	Test	Result
SQLite	Create DB on one side, JOIN query from the other	Works
SQLite (multi-table)	RPG save with players + inventory, JOIN from peer	Works
jq	Write JSON on one side, parse from the other	Works
Python csv	Write CSV, read with DictReader from peer	Works
Node.js	Write file on one side, read from the other	Works
Jupyter	Write .ipynb on one side, parse from the other	Works

Languages

Tool	Test	Result
Python	Write module on Alice, import from Bob	Works
Perl	SHA256 hash with Digest::SHA across mounts	Works
Ruby	JSON + Digest::MD5 across mounts	Works
Node.js	npm install on one side, require from the other	Works
Bash	Write script on one side, execute from the other	Works
Swift	Compile and run across mounts	Works

File operations

Tool	Test	Result
zip / unzip	Zip repo on one side, unzip on the other	Works
tar / gzip	Create archive, extract on the other side	Works
rsync --checksum	Checksummed copy from shared folder	Works
dd	1 MB + 100 MB random data, SHA256 verified both sides	Works
split + cat	Split file on one side, reassemble on the other	Works
base64	Encode on one side, decode on the other, hash match	Works

Integrity verification

Tool	Test	Result
sha256	Same file hashed from both sides	Identical
sha512	Same file hashed from both sides	Identical
openssl dgst	SHA256 digest from both sides	Identical
md5 (bulk)	All 71 .c files hashed, aggregate compared	Identical
diff	Source files compared across mounts	Identical
crc32	CRC32 checksum from both sides	Identical

Security tools

Tool	Test	Result
golangci-lint	Lint Go source from peer's mount	Works
gosec	Security scan from peer's mount	Works
openssl enc/dec	AES-256-CBC encrypt on one side, decrypt from the other	Works
exiftool	Read image metadata from peer's file	Works

Collaborative workflows

The shared folder is bidirectional. Both sides can write. Two people can each handle a different step of the same project.

Workflow	Alice does	Bob does	Result
C compilation	Writes main.c + headers	Compiles from his view	Alice runs the binary Bob built
LaTeX slides	Writes .tex source	Compiles PDF with xelatex	Alice opens the PDF Bob produced
Data pipeline	Writes 100-row CSV	Runs Python analysis, writes JSON	Alice reads Bob's results
Video processing	Has a 190 MB video	Creates thumbnail grid with ffmpeg	Alice sees the thumbnails

Large files

Test	Size	Result
100 MB random file, SHA256 from both sides	100 MB	Identical
192 MB video, ffprobe from peer's mount	192 MB	Works
ffmpeg transcode: Bob processes Alice's video, output appears on Alice	192 MB in, 153 KB out	Works
50 files written by Alice, aggregate hash verified by Bob	50 files	Identical
71 source files tar'd, extracted on other side, all hashes verified	71 files	All match

What does not work

Operation	Reason
Symlinks / hardlinks	Not implemented. Tools like uv, npm with bin links fail. Use --no-bin-links or install outside the shared folder.
Execute binaries from mount (macOS)	macOS blocks execution from FUSE mounts. Copy the binary locally first. Linux does not have this restriction.

How it works

KEIBIDROP presents the peer's files through the OS filesystem interface (FUSE on macOS/Linux, WinFsp on Windows). Every program that reads files through standard system calls works without modification. The file metadata syncs when the connection establishes. File contents stream on demand as programs read them. The encryption overhead is under 2%.

The connection is end-to-end encrypted with post-quantum cryptography (ML-KEM-1024 + X25519). No server stores your files.

Install: keibidrop.com/install
Source: github.com/KeibiSoft/KeibiDrop

KeibiDrop: Instant share 600 MB video from phone to Destkop

Marius-Florin Cristian — Sat, 30 May 2026 18:07:51 +0000

Here I add a 600MB Popeye video from my iPhone and can open it instantly on my desktop and jump through it, while the downloads happen in the background.

It works with all types of files and programs, including git clone.

https://keibidrop.com

and on github:

Mobile apps coming soon!

Marius-Florin Cristian — Sat, 09 May 2026 15:41:01 +0000

Marius-Florin Cristian

Apr 27

KEIBIDROP: Secure File Transfer!

#go #security #filesystem #rust

1 min read

KEIBIDROP: Secure File Transfer!

Marius-Florin Cristian — Mon, 27 Apr 2026 15:30:09 +0000

KEIBIDROP encrypted file transfer and shared filesystem between two devices

https://keibidrop.com

KeibiDrop connects two devices directly and lets them share files over an encrypted channel. It works over the internet or a local network.

Two modes:

Direct transfer - drag files in, your peer saves what they need
Virtual folder (FUSE) - your peer's files appear as a real folder on your machine. git, rsync, your IDE, all work on it. Files stream on access, so you can open a 4 GB video before it finishes downloading.

Post-quantum encryption (ML-KEM-1024 + X25519). The relay, if used, cannot decrypt anything.

macOS, Windows, Linux. Desktop GUI, interactive CLI, agent CLI for automation.

Free, open source (MPL-2.0): github.com/KeibiSoft/KeibiDrop

KEIBIDROP v0.1.0 is out! https://github.com/KeibiSoft/KeibiDrop/releases/tag/v0.1.0 Now you can send files, or mount files in real time as a distributed system :D Direct peer to peer or fallback via datarelay. Open source, and no transfer limits!

Marius-Florin Cristian — Mon, 27 Apr 2026 09:59:44 +0000

Release KeibiDrop v0.1.0 · KeibiSoft/KeibiDrop · GitHub

First public release. Peer-to-peer encrypted file sharing across Linux, macOS, and Windows. What's in this release End-to-end encrypted transfer (AES-256-GCM / ChaCha20-Poly1305, auto-negotiated b...

github.com

Can We Compute the Minimum Spanning Tree in Linear Time?

Marius-Florin Cristian — Wed, 01 Apr 2026 11:43:03 +0000

Easter egg at the end of how the cover photo is related to the post

Given a connected graph with weighted edges, the minimum spanning tree (MST) is the cheapest set of edges that keeps all vertices connected. Boruvka gave the first algorithm in 1926. Kruskal, Prim, and others refined it. The best randomized algorithm runs in O(m) expected time (Karger-Klein-Tarjan, 1995). The best deterministic algorithm runs in O(m α(m,n)) (Chazelle, 2000), where α is the inverse Ackermann function.

α(m,n) ≤ 4 for any graph that could physically exist. In practice, the problem is solved. In theory, the question "can we do O(m) deterministically?" has been open for over forty years.

This post covers the three attack vectors from my 2018 thesis at DIKU, updated with connections to research published since then.

Where the bottleneck is

Every MST algorithm uses some form of the disjoint-set (union-find) data structure. You process edges, and for each edge you ask: are these two vertices already connected? If not, merge their components.

With path compression and union by rank, m operations on n elements cost O(m α(m,n)). That α factor is the bottleneck. It comes from the analysis of path compression on a pointer machine.

On a Word RAM, Gabow and Tarjan (1983) showed you can do O(m + n) — if you know the union tree structure in advance. You precompute lookup tables for small components and answer FIND queries in O(1).

The catch: knowing the union tree means knowing the MST, which is what you're computing.

Two computation models

This matters because the bottleneck lives in the gap between them.

A pointer machine can follow pointers, compare values, and create new nodes. No arithmetic on addresses, no random access. This is how most graph algorithms actually work. On this model, the α factor from path compression cannot be removed.

A Word RAM has O(1) random access and can do arithmetic and bitwise operations on words. This enables table lookups — precomputing answers for small inputs and reading them in constant time.

The MST gap is a model gap. On a pointer machine, O(m α(m,n)) seems tight. On a Word RAM, O(m + n) is possible if the structure is known. The question is whether you can learn enough structure during computation to exploit the RAM model.

Attack vector 1: Approximate union-find

What if you approximately know the union tree?

Build a predicted tree T' using some fast heuristic (a few Boruvka phases, random sampling, anything that runs in O(m)). Use Gabow-Tarjan lookup tables based on T'. When a FIND query returns an answer that disagrees with the true structure, pay for a correction.

If the prediction is close enough to the true MST, most queries hit the lookup tables and cost O(1). The faulty queries create "phantom cycles" — the algorithm temporarily believes two vertices are connected when they aren't, or vice versa. These phantoms are bounded by the prediction error and can be cleaned up in a linear-time verification pass using King's MST verification algorithm.

This approach fits a framework that didn't exist when the thesis was written: learning-augmented algorithms (Mitzenmacher and Vassilvitskii, 2020). The formal question is: what prediction quality guarantees O(m + n) total cost?

The thesis proves the disjoint-set operations can be bounded to O(m + n) on the RAM model when allowing a bounded degree of error. The precise bounds on prediction quality needed for this to work are stated but not tight.

Attack vector 2: Density partition

Different algorithms are linear at different densities. Prim is linear on dense graphs (m ≥ n^p for p > 1). Kruskal-style algorithms are nearly linear on sparse graphs. Neither covers all graphs.

The density partition classifies vertices by degree:

HD (high degree): degree > c · log³(n)
LD (low degree): degree < c · log³(n)

Then by neighbor types: HDN (high-degree neighbors), LDN (low-degree neighbors), MDN (mixed). This classification takes O(m + n) — two passes over the adjacency lists.

For HD-HDN components (dense), run Prim. For LD-LDN components (sparse), run modified Kruskal with the approximate union-find from attack vector 1. The MDN boundary edges form a small graph — O(n / log³(n)) edges — that can be solved recursively.

At each recursion level, total work is O(m + n). The recursion depth is bounded. This is the glue that combines the other two subproblems.

Attack vector 3: Cycle hierarchy

Consider a sparse graph that is almost a tree. Most vertices have degree 1 or 2. The MST must include all edges connecting degree-1 vertices — there is no alternative path. So you can trim those edges and contract the leaves.

The problem: trimming cascades. Removing a leaf can create a new leaf, which creates another, propagating across the graph. Naive trimming is O(n²).

If you know the cycle structure, you know where each cascade stops — at a vertex that participates in a cycle. With this information, trimming runs in O(n).

The cycle hierarchy organizes cycles by nesting depth:

Level 0: triangles
Level 1: 4-cycles not decomposable into triangles
Level k: (k+3)-cycles that may contain lower-level cycles

Building the hierarchy seems circular — you need a spanning tree to find cycles, but you're computing the MST. The escape: cycle detection doesn't require edge weights. Any spanning tree (from DFS or BFS, computed in O(m + n) without weight comparisons) reveals the cycle structure. Each non-tree edge closes exactly one fundamental cycle.

You compute the topological hierarchy weight-free, then apply it to the weighted MST computation. The hierarchy tells trimming where to stop.

This is the hardest subproblem. The construction is clear in outline, but the formal proof that it runs in O(m + n) for arbitrary graphs is incomplete.

How they compose

Subproblem 1                    Subproblem 3
(Approximate Union-Find)        (Cycle Hierarchy)
         \                          /
          v                        v
           Subproblem 2
           (Density Partition)
                  |
                  v
          Full MST Algorithm
              O(m + n)

Subproblems 1 and 3 are independent. Subproblem 2 combines their results. If all three are solved, they compose into a deterministic O(m + n) MST algorithm on the Word RAM.

On a pointer machine, this approach does not help. The disjoint-set operations without table lookup are bounded by O(m α(m,n)), and that factor cannot be removed without random access.

What's incomplete

Subproblem 1 has a theorem and proof sketch. The formal bounds on prediction quality are not tight. Subproblem 3 has a clear construction but the O(m + n) proof for arbitrary graphs is missing. Subproblem 2 depends on both.

Since 2018, each subproblem has connected to active research:

Approximate union-find → learning-augmented algorithms
Density partition → local graph algorithms and graph sparsification
Cycle hierarchy → dynamic connectivity and top trees

The three subproblems remain open. Whether O(m) MST is achievable deterministically, or whether α(m,n) is fundamental, is one of the oldest open problems in computer science.

The full technical writeup with proofs, definitions, and historical context is on my blog: The MST Problem: Three Subproblems to Linear Time.

The thesis was written at DIKU, University of Copenhagen, supervised by Jyrki Katajainen.

Easter egg time: Bernard Chazelle the CS professor from Toronto who has the best algorithm for the problem up to date is the father of Damien Chazelle, the director of Whiplash and LaLa Land.

Optimizing encrypted P2P file transfer - from 225 to 441 MB/s

Marius-Florin Cristian — Sun, 29 Mar 2026 08:36:21 +0000

Part of the KEIBIDROP development blog. KEIBIDROP is in active development. Release is coming soon.

KEIBIDROP transfers files between two peers over encrypted gRPC. The full stack:

Disk I/O -> FUSE kernel -> FUSE daemon -> gRPC framing -> ChaCha20-Poly1305 -> TCP -> Peer

We built micro-benchmarks for each layer and measured throughput with 1GB files on an Intel MacBook Pro.

Baseline numbers

Layer	Throughput	Overhead
Raw disk (SSD)	~5 GB/s	--
Raw gRPC (no encryption)	981 MB/s	5x vs disk
Encrypted gRPC (ChaCha20)	437 MB/s	2.2x vs raw gRPC
FUSE end-to-end	225 MB/s	1.9x vs encrypted gRPC

The encryption layer costs 2.2x. FUSE adds another 1.9x.

Six optimizations

1. Cache the AEAD cipher. The original code created a new ChaCha20-Poly1305 cipher for every message. Caching it in the constructor is safe because the nonce is a monotonic counter.

// Before: creating cipher per-message
aead, _ := chacha20poly1305.NewX(s.key)  // expensive!

// After: created once in constructor
type SecureWriter struct {
    aead cipher.AEAD  // created once
}

2. Single combined TCP write. Two syscalls per message (header + payload) became one. At 512KB chunks, this halves the write() call count.

3. In-place decryption. aead.Open(ciphertext[:0], nonce, ciphertext, nil) reuses the ciphertext buffer instead of allocating a new one. ~2000 fewer allocations per 1GB transfer.

4. Async cache writes in FUSE Read. FUSE needs the data returned to the kernel. Writing that data to the local cache file can happen in a background goroutine. A WaitGroup ensures everything completes before file close. Cache write overhead dropped from 16% to 1.8%.

n := copy(buff, data)
if cacheFD != nil {
    f.CacheWg.Add(1)
    go func() {
        defer f.CacheWg.Done()
        cacheFD.WriteAt(cacheData, cacheOffset)
    }()
}
return n

5. Push-based StreamFile RPC. The original streaming used request-response per chunk. A new server-streaming RPC pushes all chunks without waiting. On-demand reads (random access) still use bidirectional streaming.

6. Silencing hot-path logs. ~120 slog.Info/Debug/Warn calls in FUSE handlers. Even structured logging allocates. Commenting out non-error logs on Read/Write/Getattr gave 2-6% throughput improvement.

After optimizations

Layer	Before	After	Gain
Encrypted gRPC	437 MB/s	441 MB/s	+0.9%
FUSE end-to-end	225 MB/s	231 MB/s	+2.7%

The encryption tax: 1.49x

We built a controlled benchmark comparing identical gRPC transfers over plain TCP vs ChaCha20-Poly1305:

Transport	MB/s	Duration (100MB)
Plain TCP + gRPC	657	152ms
Encrypted (ChaCha20-Poly1305)	441	227ms

33% of transfer time goes to encryption. Our earlier 2.2x estimate was comparing across different test runs with different system load. The controlled A/B benchmark showed the real cost is much lower.

The irreducible 48%

The gap between encrypted gRPC (441 MB/s) and FUSE end-to-end (231 MB/s) is 48%. Every FUSE Read requires two kernel-to-userspace context switches. We confirmed this by measuring raw FUSE overhead (local file read through FUSE vs direct): ~48%, matching exactly.

Plain gRPC:     657 MB/s  (ceiling)
  -33% encryption
Encrypted gRPC: 441 MB/s
  -48% FUSE kernel overhead
FUSE E2E:       231 MB/s

The FUSE overhead is nearly twice the encryption cost. If we wanted to improve throughput, reducing FUSE transitions would have more impact than switching ciphers. But FUSE is the cost of running a userspace filesystem, and for secure P2P file sharing the tradeoff is worth it.

What we learned

Without isolating each layer, we would have optimized the wrong thing. The biggest win (async cache writes, 16% to 1.8%) was only visible when measured independently.
Caching the AEAD cipher and in-place decryption are tiny code changes with measurable impact at 2000+ messages per second.
Structured logging is not free on hot paths. Even checking the log level has overhead.
FUSE has an inherent 48% overhead from context switches. No amount of userspace optimization can fix this.
Server-streaming eliminates round-trip latency for sequential prefetch. Request-response is still needed for random access.
Our initial 2.2x encryption estimate was wrong. A controlled A/B benchmark (same test, same system) showed 1.49x. Always compare apples to apples.

More from the KEIBIDROP blog: full series | product page | FAQ

Make git clone work between encrypted P2P FUSE peers (7 bugs)

Marius-Florin Cristian — Sun, 29 Mar 2026 08:34:07 +0000

Part of the KEIBIDROP development blog. KEIBIDROP is in active development. Release is coming soon.

KEIBIDROP mounts a virtual FUSE filesystem for each peer. When Peer A creates or modifies files, the changes sync to Peer B's mount in real-time through encrypted gRPC. We already made git work on a single FUSE mount (previous post). The question was: what happens when Peer A clones a repo into their mount, and Peer B tries to use it through theirs?

Bob (FUSE mount)                     Alice (FUSE mount)
+-- .git/                            +-- .git/          <- synced from Bob
|   +-- HEAD                         |   +-- HEAD
|   +-- config                       |   +-- config
|   +-- objects/pack/                |   +-- objects/pack/
|   |   +-- pack-abc123.pack (2MB)   |   |   +-- pack-abc123.pack
|   +-- refs/heads/main              |   +-- refs/heads/main
+-- go.mod                           +-- go.mod
+-- README.md                        +-- README.md

Seven bugs appeared across three debugging sessions.

Bug 1: The rename race

Git uses atomic writes: create config.lock, write content, close, rename to config. KEIBIDROP sends ADD_FILE on close and RENAME_FILE on rename. The prefetch goroutine would download config.lock, finish before the RENAME_FILE arrived, and skip the disk rename because the in-memory path hadn't changed yet.

Fix: the RENAME_FILE handler now does os.Rename on disk before updating maps. The prefetch cleanup becomes a redundant safety net.

Bug 2: Pack file corruption (20 missing bytes)

Alice's pack file was 2,182,186 bytes. Bob's was 2,182,206 bytes. Git's index-pack appends a 20-byte SHA-1 checksum after the file is closed but before the rename. KEIBIDROP sent the ADD_FILE notification at close time with the pre-checksum size.

Fix: after renaming on disk, compare local file size with the size in the RENAME_FILE notification. If they differ, trigger a re-download.

Bug 3: macFUSE cache poisoning

git status showed deleted: go.mod even though the file existed on disk. macFUSE's negative_vncache option caches ENOENT results. macOS probed Alice's mount (Spotlight, fsevents) before the file arrived, the kernel cached "doesn't exist", and subsequent reads never reached our FUSE handler.

Fix: remove negative_vncache from mount options.

Bug 4: The 612-file hang

Cloning a large repo (612 files) hung after checkout completed. Every FUSE Release sent a gRPC notification. 612 files closing in rapid succession overwhelmed the connection.

Three-part fix:

BatchNotify RPC -- batch 64 notifications into a single gRPC call
Per-path debounce (200ms deadline, reset on each update) on the sender side
Prefetch semaphore (max 8 concurrent) on the receiver side

rpc BatchNotify (BatchNotifyRequest) returns (BatchNotifyResponse);

message BatchNotifyRequest {
  repeated NotifyRequest notifications = 1;
  uint64 seq = 2;
  uint64 timestamp = 3;
}

Bug 5: 190,000 log lines

After fixing the notification flood, the clone still hung for 30 seconds. Turns out we had 207,000 getattr calls and 190,000 ENOENT error logs, all written synchronously. macOS probes hundreds of paths per directory (Spotlight, fsevents, .DS_Store, resource forks) and without negative_vncache every probe hits our handler.

Fix: remove trace logs from hot-path handlers. Stop logging ENOENT as an error.

grep -c "FUSE getattr" Log_Bob.txt    207,137
grep -c "Failed to lstat" Log_Bob.txt  190,614

Bug 6: Hook permission denied

fatal: cannot exec '.git/hooks/post-checkout': Operation not permitted. macOS Gatekeeper blocks executing scripts from FUSE mounts.

Fix: add defer_permissions to mount options.

Bug 7: LFS corruption (intermediate sizes + stale notifications)

Git LFS downloads a 420MB XML file incrementally into .git/lfs/incomplete/<hash>. Each intermediate close triggers ADD_FILE with the current size. The peer starts prefetching at 100MB, restarts at 200MB, restarts at 300MB. Then LFS renames incomplete/<hash> to objects/<sha>/<hash>, but the debounced ADD_FILE for the old path fires after the rename, pointing to a file that no longer exists.

Fix: per-path debounce with RENAME retargeting. When a RENAME arrives, the pending ADD_FILE for the old path gets its path rewritten to the new path instead of being deleted or sent stale.

case bindings.NotifyType_RENAME_FILE:
    if old, exists := pending[req.OldPath]; exists {
        delete(pending, req.OldPath)
        old.req.Path = req.Path  // retarget to new path
        pending[req.Path] = old
    }
    immediate = append(immediate, req)

We tried three approaches: deleting the pending notification (broke content delivery), sending it stale (wrong path), retargeting (correct).

The result

After all fixes, the full git workflow works between encrypted P2P FUSE peers:

# Bob clones a repo
$ cd MountBob && git clone git@github.com:user/repo.git
Cloning into 'repo'... done.

# Alice sees it immediately
$ cd MountAlice/repo && git status
On branch main -- nothing to commit, working tree clean

# Alice creates a branch and commits
$ git checkout -b test_me
$ echo "test" > tst.txt && git add tst.txt && git commit -m "Commit for test"

# Bob sees the branch and commit
$ cd MountBob/repo && git log
commit 38e23bc (HEAD -> test_me)
    Commit for test

Tested with a small repo (24 objects, 2MB, 8 files) and a large repo (2339 objects, 257MB + 3.71 GiB LFS, 612 files).

What we learned

FUSE handlers must rename on disk, not just in memory. Goroutines race with notification handlers, especially for tiny files that download in milliseconds.
Git writes data, closes the file, then renames. Between close and rename, other processes can modify the file (index-pack appending checksums). The rename notification has the correct size. Use it.
macFUSE's negative_vncache silently caches ENOENT results. Our handler was never called. These bugs don't show up in FUSE logs because the kernel short-circuits the call.
612 files = 612 goroutines = 612 gRPC streams = connection collapse. A bounded channel + batching worker gives backpressure without blocking the caller.
190,000 synchronous log writes caused a 30-second hang. The logging seemed harmless with 20 files. 612 files with macOS probing turned it into 400,000 disk writes.
For debounce + rename: you can't delete the pending notification (peer never gets content) and you can't send it stale (wrong path). Retarget it to the new path. Three attempts to get this right.
None of these bugs are git-specific. Any application that creates files through one peer's mount and reads them through another would hit the same problems. Git just exercises every edge case.

More from the KEIBIDROP blog: full series | product page | FAQ