DEV Community: Marius-Florin Cristian

KEIBIDROP: Romania to Singapore Benchmarks

Marius-Florin Cristian — Sun, 14 Jun 2026 08:57:37 +0000

Measuring an encrypted peer-to-peer filesystem over a 200 to 330 ms intercontinental link to a Windows host: throughput, connect time, and FUSE on-demand latency, and what bounds them.

Abstract. Measurement of KeibiDrop, a post-quantum encrypted peer-to-peer filesystem, over a high-latency intercontinental link. Two Linux peers in Romania run against one Windows Server peer in Singapore, at 200 ms (datacenter to datacenter) and 330 ms (residential) round-trip. The Windows host is IPv4 only, so all traffic traverses the TCP bridge. Three dimensions are reported: bulk throughput, connect time, and FUSE on-demand read latency. Each cell is the median over repeated trials, with the per-table trial count n shown in the tables, all peers on commit b5a7a3e. Findings: 30 to 38 MB/s into the Windows host and 13 to 25 MB/s out; connect time about 6.3 s, dominated by an IPv6 dial timeout before bridge fallback; FUSE first byte under 1.5 s with sequential on-demand read near 1 MB/s. The throughput ceiling is the bandwidth-delay product, not link capacity or CPU. FUSE mode did not start on the Windows host on this build; a fix is in review.

Setup

Three machines, two pairings. One end is always the Singapore box.

machine	details
SG	Singapore, Windows Server, 4 vCPU / 8 GB, WinFsp installed. IPv4 only.
VPS	Timisoara, Linux. Also runs the relay and the TCP bridge.
laptop	Iasi, Linux, residential link about 500 to 600 Mbps wired (62 to 75 MB/s).
Pairing A	VPS to SG, about 200 ms RTT, datacenter to datacenter.
Pairing B	laptop to SG, about 330 ms RTT, home to datacenter.

Method: each cell is the median across repeated trials, with the trial count n shown in every table: 5 at 10 and 100 MB, 3 at 1 GB, 1 at 5 and 10 GB for bulk; 10 for handshake; 3 for FUSE. All three peers run the same commit. X to Y means X sends and Y receives. Bulk transfer uses kd add then a timed pull, no FUSE in the path.

Bulk throughput

Pairing A, VPS to SG, about 200 ms

Transfer	Size	n	Median MB/s	p95
SG to VPS	10 MB	5	25.6	28.0
VPS to SG	100 MB	5	35.5	41.5
VPS to SG	1 GB	3	37.6	40.4
SG to VPS	1 GB	3	13.2	13.6
VPS to SG	5 GB	1	34.6	34.6
SG to VPS	5 GB	1	8.6	8.6
VPS to SG	10 GB	1	18.8	18.8
SG to VPS	10 GB	1	12.9	12.9

Pairing B, laptop to SG, about 330 ms

Transfer	Size	n	Median MB/s	p95
SG to laptop	10 MB	5	20.4	21.8
laptop to SG	100 MB	5	29.6	30.5
laptop to SG	1 GB	3	29.0	29.3
SG to laptop	1 GB	3	13.4	15.1
laptop to SG	5 GB	1	32.9	32.9
laptop to SG	10 GB	1	31.3	31.3

Handshake

Time to connect to the Singapore box, parsed from the joiner's own log (relay lookup to connected).

Connects to SG from	n	Median ms	p95	min	max
VPS (pairing A)	10	6316	6473	6094	6476
laptop (pairing B)	10	6220	6481	5977	6507

FUSE on-demand

A Linux peer mounts the virtual drive and reads a file hosted on SG. Content streams on demand over the link. ttfb is the time to the first byte (the first 512 KB chunk). The mount is Linux because the Windows mount has a bug, covered below.

Mount reads from SG	Size	n	ttfb median ms	p95	read MB/s
VPS mount (pairing A)	10 MB	3	632	1242	0.9
VPS mount (pairing A)	100 MB	3	509	635	1.0
laptop mount (pairing B)	10 MB	3	709	1423	0.8
laptop mount (pairing B)	100 MB	3	482	549	0.8

What the numbers say

The limit is round-trip time, not bandwidth and not CPU. The home link is 500 to 600 Mbps, which is 62 to 75 MB/s, and the best run used about half of it. The box is a 4 vCPU machine that stayed near idle, and ChaCha20 runs at gigabytes per second per core, so encryption is not the wall either. The wall is the bandwidth-delay product: one gRPC stream with a 16 MiB window cannot keep a 330 ms pipe full, because filling it needs roughly 18 MB in flight at the home link rate. A bigger window or parallel pull streams would close that gap.

The Singapore box sends at about half the rate it receives. At 1 GB on pairing A it pulls in 37.6 MB/s but pushes out 13.2 MB/s, and the same shape holds across sizes and on pairing B. The receive path is fine. The send path is where the work is.

The 6.3 second connect is one timeout. The box has no IPv6, so the direct IPv6 dial always fails and waits out its deadline before the bridge takes over. Skipping the direct dial when a peer advertises no IPv6 removes most of that. The same change helps phones, which are usually IPv4 only and hit the same wait.

On-demand FUSE is for opening and seeking, not for streaming a whole large file. The first byte comes back in half a second to a second and a half, which is fine for opening a document, scrubbing a video, or reading part of a dataset. Sequential read settles near 1 MB/s because each 512 KB chunk costs one to three round trips at this latency. For a whole large file, a bulk pull at 30 MB/s or a prefetch is the right path.

Defects identified

FUSE mode did not start on the Windows host on this build. host.Mount returned false and the daemon fell back to no-FUSE: kd list showed the files, but the mount point stayed an empty folder and neither local nor remote files surfaced. The cause is the mount point being created before WinFsp runs. WinFsp takes the mount point as a drive letter (for example K:), an auto-assigned drive (*), or a directory it creates itself, and it rejects a path that already exists. PR #181 removes the pre-creation on Windows. No-FUSE transfer is unaffected, which is why the FUSE tables above use Linux mounts.

One 10 GB transfer looked like it stalled at 12 MB. A re-run moved 3.7 GB in 4 minutes at a steady 15.6 MB/s, so it was a blip on the residential link, not a defect. Not filed.

Planned changes

Skip the direct dial when a peer advertises no IPv6. This removes most of the 6.3 s connect time on the common case and applies to mobile, which is also IPv4 only.

Land the Windows mount fix (PR #181) so FUSE starts on the Windows host and surfaces files.

Add a push-based send to remove the asymmetry where the host sends at half its receive rate.

For bulk throughput, raise the window or pull with several streams. Tradeoff: saturating the link degrades on-demand read latency, so this stays opt-in for bulk transfers rather than the default for the interactive mount.

Medians over the per-table trial counts (n), all peers on commit b5a7a3e. Pairing A is VPS to SG at about 200 ms. Pairing B is laptop to SG at about 330 ms. All traffic over the TCP bridge.

KEIBIDROP: Also on r/selfhosted

Marius-Florin Cristian — Mon, 01 Jun 2026 19:20:41 +0000

I made an open source tool that allows two people to directly connect via WAN, or LAN, and mount a virtual mountpoint.

(The first video in link URL shows side by side my, and my mac and brothers ubuntu use the shared mounted filesystem)

https://github.com/KeibiSoft/KeibiDrop

The idea is that two people establish a connection, and mount virtual folders. Metadata is synced automatically when things are dragged in the virtual folder, and it supports reading with offset, which means that if Alice shares a 10GB video, or folder with files, Bob can see it in a few seconds, and start playing it.
(Here is a fresher video where I add a Popeye 600 MB Video on my iphone, and start playing it with QuickTime and skim through it before the download completes)
https://youtu.be/UHw1WtkZ8dE

The tool has been in development for a little over a year, and just made it public last month.

There are a series of technical blog posts about the pain points, optimization paths, and how we set benchmarks in order to make work fast at
https://keibisoft.com/blog.html#keibidrop

The most notable thing is that once Alice and Bob are connected they can use programs as if the filesystems were native. (thus Alice and git clone a repo, and bob git commit, or checkout instantly on the same repo).

It is still in development, and as me and my brother use it and polish it we also fix any issues that arrise.

Happy to discuss about it, and things that I learned while grinding on the problem :D

KEIBIDROP: Every tool just works

Marius-Florin Cristian — Mon, 01 Jun 2026 03:48:28 +0000

We connected two machines with KEIBIDROP and ran every program we could think of on the shared virtual folder. Compilers, version control, video processing, databases, scripting languages, security scanners. Everything worked. File integrity held across every operation.

KEIBIDROP presents the peer's files through the operating system's filesystem interface. Programs do not know the files are remote. They read and write to a folder. The data arrives from the other machine over an encrypted channel.

How the shared folder works between two machines

Machine A adds files to the shared folder. They appear on Machine B within a second. Machine B opens them, compiles them, processes them, and saves the output back to the shared folder. Machine A sees the output immediately. No upload, no download, no sync button. Both machines read and write to the same folder in real time, over an encrypted peer-to-peer connection.

We tested this with 45+ programs. Alice writes source code, Bob compiles it. Bob writes analysis results, Alice reads them. Alice has a video, Bob creates thumbnails from it. Every tool that reads files from a folder works. The programs do not know the files are on another machine.

Build tools

Tool	Test	Result
git	Clone repo, run git status and git log from the other side	Works
make (C)	Compile 87 source files inside the shared folder	Works
cargo (Rust)	cargo build --release on the shared folder	Works
go build	Init module, build binary on the shared folder	Works
swiftc	Compile Swift program on the shared folder	Works
xelatex	Compile LaTeX slides across mounts	Works

Media tools

Tool	Test	Result
ffprobe	Inspect 192 MB video from peer's folder	Works
ffmpeg	Extract clip, create thumbnail grid from peer's video	Works
ImageMagick	Read PNG, resize, identify 2048x5568 tileset	Works

Data and databases

Tool	Test	Result
SQLite	Create DB on one side, JOIN query from the other	Works
SQLite (multi-table)	RPG save with players + inventory, JOIN from peer	Works
jq	Write JSON on one side, parse from the other	Works
Python csv	Write CSV, read with DictReader from peer	Works
Node.js	Write file on one side, read from the other	Works
Jupyter	Write .ipynb on one side, parse from the other	Works

Languages

Tool	Test	Result
Python	Write module on Alice, import from Bob	Works
Perl	SHA256 hash with Digest::SHA across mounts	Works
Ruby	JSON + Digest::MD5 across mounts	Works
Node.js	npm install on one side, require from the other	Works
Bash	Write script on one side, execute from the other	Works
Swift	Compile and run across mounts	Works

File operations

Tool	Test	Result
zip / unzip	Zip repo on one side, unzip on the other	Works
tar / gzip	Create archive, extract on the other side	Works
rsync --checksum	Checksummed copy from shared folder	Works
dd	1 MB + 100 MB random data, SHA256 verified both sides	Works
split + cat	Split file on one side, reassemble on the other	Works
base64	Encode on one side, decode on the other, hash match	Works

Integrity verification

Tool	Test	Result
sha256	Same file hashed from both sides	Identical
sha512	Same file hashed from both sides	Identical
openssl dgst	SHA256 digest from both sides	Identical
md5 (bulk)	All 71 .c files hashed, aggregate compared	Identical
diff	Source files compared across mounts	Identical
crc32	CRC32 checksum from both sides	Identical

Security tools

Tool	Test	Result
golangci-lint	Lint Go source from peer's mount	Works
gosec	Security scan from peer's mount	Works
openssl enc/dec	AES-256-CBC encrypt on one side, decrypt from the other	Works
exiftool	Read image metadata from peer's file	Works

Collaborative workflows

The shared folder is bidirectional. Both sides can write. Two people can each handle a different step of the same project.

Workflow	Alice does	Bob does	Result
C compilation	Writes main.c + headers	Compiles from his view	Alice runs the binary Bob built
LaTeX slides	Writes .tex source	Compiles PDF with xelatex	Alice opens the PDF Bob produced
Data pipeline	Writes 100-row CSV	Runs Python analysis, writes JSON	Alice reads Bob's results
Video processing	Has a 190 MB video	Creates thumbnail grid with ffmpeg	Alice sees the thumbnails

Large files

Test	Size	Result
100 MB random file, SHA256 from both sides	100 MB	Identical
192 MB video, ffprobe from peer's mount	192 MB	Works
ffmpeg transcode: Bob processes Alice's video, output appears on Alice	192 MB in, 153 KB out	Works
50 files written by Alice, aggregate hash verified by Bob	50 files	Identical
71 source files tar'd, extracted on other side, all hashes verified	71 files	All match

What does not work

Operation	Reason
Symlinks / hardlinks	Not implemented. Tools like uv, npm with bin links fail. Use --no-bin-links or install outside the shared folder.
Execute binaries from mount (macOS)	macOS blocks execution from FUSE mounts. Copy the binary locally first. Linux does not have this restriction.

How it works

KEIBIDROP presents the peer's files through the OS filesystem interface (FUSE on macOS/Linux, WinFsp on Windows). Every program that reads files through standard system calls works without modification. The file metadata syncs when the connection establishes. File contents stream on demand as programs read them. The encryption overhead is under 2%.

The connection is end-to-end encrypted with post-quantum cryptography (ML-KEM-1024 + X25519). No server stores your files.

Install: keibidrop.com/install
Source: github.com/KeibiSoft/KeibiDrop

KeibiDrop: Instant share 600 MB video from phone to Destkop

Marius-Florin Cristian — Sat, 30 May 2026 18:07:51 +0000

Here I add a 600MB Popeye video from my iPhone and can open it instantly on my desktop and jump through it, while the downloads happen in the background.

It works with all types of files and programs, including git clone.

https://keibidrop.com

and on github:

Mobile apps coming soon!

Marius-Florin Cristian — Sat, 09 May 2026 15:41:01 +0000

Marius-Florin Cristian

Apr 27

KEIBIDROP: Secure File Transfer!

#go #security #filesystem #rust

1 min read

KEIBIDROP: Secure File Transfer!

Marius-Florin Cristian — Mon, 27 Apr 2026 15:30:09 +0000

KEIBIDROP encrypted file transfer and shared filesystem between two devices

https://keibidrop.com

KeibiDrop connects two devices directly and lets them share files over an encrypted channel. It works over the internet or a local network.

Two modes:

Direct transfer - drag files in, your peer saves what they need
Virtual folder (FUSE) - your peer's files appear as a real folder on your machine. git, rsync, your IDE, all work on it. Files stream on access, so you can open a 4 GB video before it finishes downloading.

Post-quantum encryption (ML-KEM-1024 + X25519). The relay, if used, cannot decrypt anything.

macOS, Windows, Linux. Desktop GUI, interactive CLI, agent CLI for automation.

Free, open source (MPL-2.0): github.com/KeibiSoft/KeibiDrop

KEIBIDROP v0.1.0 is out! https://github.com/KeibiSoft/KeibiDrop/releases/tag/v0.1.0 Now you can send files, or mount files in real time as a distributed system :D Direct peer to peer or fallback via datarelay. Open source, and no transfer limits!

Marius-Florin Cristian — Mon, 27 Apr 2026 09:59:44 +0000

Release KeibiDrop v0.1.0 · KeibiSoft/KeibiDrop · GitHub

First public release. Peer-to-peer encrypted file sharing across Linux, macOS, and Windows. What's in this release End-to-end encrypted transfer (AES-256-GCM / ChaCha20-Poly1305, auto-negotiated b...

github.com

Can We Compute the Minimum Spanning Tree in Linear Time?

Marius-Florin Cristian — Wed, 01 Apr 2026 11:43:03 +0000

Easter egg at the end of how the cover photo is related to the post

Given a connected graph with weighted edges, the minimum spanning tree (MST) is the cheapest set of edges that keeps all vertices connected. Boruvka gave the first algorithm in 1926. Kruskal, Prim, and others refined it. The best randomized algorithm runs in O(m) expected time (Karger-Klein-Tarjan, 1995). The best deterministic algorithm runs in O(m α(m,n)) (Chazelle, 2000), where α is the inverse Ackermann function.

α(m,n) ≤ 4 for any graph that could physically exist. In practice, the problem is solved. In theory, the question "can we do O(m) deterministically?" has been open for over forty years.

This post covers the three attack vectors from my 2018 thesis at DIKU, updated with connections to research published since then.

Where the bottleneck is

Every MST algorithm uses some form of the disjoint-set (union-find) data structure. You process edges, and for each edge you ask: are these two vertices already connected? If not, merge their components.

With path compression and union by rank, m operations on n elements cost O(m α(m,n)). That α factor is the bottleneck. It comes from the analysis of path compression on a pointer machine.

On a Word RAM, Gabow and Tarjan (1983) showed you can do O(m + n) — if you know the union tree structure in advance. You precompute lookup tables for small components and answer FIND queries in O(1).

The catch: knowing the union tree means knowing the MST, which is what you're computing.

Two computation models

This matters because the bottleneck lives in the gap between them.

A pointer machine can follow pointers, compare values, and create new nodes. No arithmetic on addresses, no random access. This is how most graph algorithms actually work. On this model, the α factor from path compression cannot be removed.

A Word RAM has O(1) random access and can do arithmetic and bitwise operations on words. This enables table lookups — precomputing answers for small inputs and reading them in constant time.

The MST gap is a model gap. On a pointer machine, O(m α(m,n)) seems tight. On a Word RAM, O(m + n) is possible if the structure is known. The question is whether you can learn enough structure during computation to exploit the RAM model.

Attack vector 1: Approximate union-find

What if you approximately know the union tree?

Build a predicted tree T' using some fast heuristic (a few Boruvka phases, random sampling, anything that runs in O(m)). Use Gabow-Tarjan lookup tables based on T'. When a FIND query returns an answer that disagrees with the true structure, pay for a correction.

If the prediction is close enough to the true MST, most queries hit the lookup tables and cost O(1). The faulty queries create "phantom cycles" — the algorithm temporarily believes two vertices are connected when they aren't, or vice versa. These phantoms are bounded by the prediction error and can be cleaned up in a linear-time verification pass using King's MST verification algorithm.

This approach fits a framework that didn't exist when the thesis was written: learning-augmented algorithms (Mitzenmacher and Vassilvitskii, 2020). The formal question is: what prediction quality guarantees O(m + n) total cost?

The thesis proves the disjoint-set operations can be bounded to O(m + n) on the RAM model when allowing a bounded degree of error. The precise bounds on prediction quality needed for this to work are stated but not tight.

Attack vector 2: Density partition

Different algorithms are linear at different densities. Prim is linear on dense graphs (m ≥ n^p for p > 1). Kruskal-style algorithms are nearly linear on sparse graphs. Neither covers all graphs.

The density partition classifies vertices by degree:

HD (high degree): degree > c · log³(n)
LD (low degree): degree < c · log³(n)

Then by neighbor types: HDN (high-degree neighbors), LDN (low-degree neighbors), MDN (mixed). This classification takes O(m + n) — two passes over the adjacency lists.

For HD-HDN components (dense), run Prim. For LD-LDN components (sparse), run modified Kruskal with the approximate union-find from attack vector 1. The MDN boundary edges form a small graph — O(n / log³(n)) edges — that can be solved recursively.

At each recursion level, total work is O(m + n). The recursion depth is bounded. This is the glue that combines the other two subproblems.

Attack vector 3: Cycle hierarchy

Consider a sparse graph that is almost a tree. Most vertices have degree 1 or 2. The MST must include all edges connecting degree-1 vertices — there is no alternative path. So you can trim those edges and contract the leaves.

The problem: trimming cascades. Removing a leaf can create a new leaf, which creates another, propagating across the graph. Naive trimming is O(n²).

If you know the cycle structure, you know where each cascade stops — at a vertex that participates in a cycle. With this information, trimming runs in O(n).

The cycle hierarchy organizes cycles by nesting depth:

Level 0: triangles
Level 1: 4-cycles not decomposable into triangles
Level k: (k+3)-cycles that may contain lower-level cycles

Building the hierarchy seems circular — you need a spanning tree to find cycles, but you're computing the MST. The escape: cycle detection doesn't require edge weights. Any spanning tree (from DFS or BFS, computed in O(m + n) without weight comparisons) reveals the cycle structure. Each non-tree edge closes exactly one fundamental cycle.

You compute the topological hierarchy weight-free, then apply it to the weighted MST computation. The hierarchy tells trimming where to stop.

This is the hardest subproblem. The construction is clear in outline, but the formal proof that it runs in O(m + n) for arbitrary graphs is incomplete.

How they compose

Subproblem 1                    Subproblem 3
(Approximate Union-Find)        (Cycle Hierarchy)
         \                          /
          v                        v
           Subproblem 2
           (Density Partition)
                  |
                  v
          Full MST Algorithm
              O(m + n)

Subproblems 1 and 3 are independent. Subproblem 2 combines their results. If all three are solved, they compose into a deterministic O(m + n) MST algorithm on the Word RAM.

On a pointer machine, this approach does not help. The disjoint-set operations without table lookup are bounded by O(m α(m,n)), and that factor cannot be removed without random access.

What's incomplete

Subproblem 1 has a theorem and proof sketch. The formal bounds on prediction quality are not tight. Subproblem 3 has a clear construction but the O(m + n) proof for arbitrary graphs is missing. Subproblem 2 depends on both.

Since 2018, each subproblem has connected to active research:

Approximate union-find → learning-augmented algorithms
Density partition → local graph algorithms and graph sparsification
Cycle hierarchy → dynamic connectivity and top trees

The three subproblems remain open. Whether O(m) MST is achievable deterministically, or whether α(m,n) is fundamental, is one of the oldest open problems in computer science.

The full technical writeup with proofs, definitions, and historical context is on my blog: The MST Problem: Three Subproblems to Linear Time.

The thesis was written at DIKU, University of Copenhagen, supervised by Jyrki Katajainen.

Easter egg time: Bernard Chazelle the CS professor from Toronto who has the best algorithm for the problem up to date is the father of Damien Chazelle, the director of Whiplash and LaLa Land.

Optimizing encrypted P2P file transfer - from 225 to 441 MB/s

Marius-Florin Cristian — Sun, 29 Mar 2026 08:36:21 +0000

Part of the KEIBIDROP development blog. KEIBIDROP is in active development. Release is coming soon.

KEIBIDROP transfers files between two peers over encrypted gRPC. The full stack:

Disk I/O -> FUSE kernel -> FUSE daemon -> gRPC framing -> ChaCha20-Poly1305 -> TCP -> Peer

We built micro-benchmarks for each layer and measured throughput with 1GB files on an Intel MacBook Pro.

Baseline numbers

Layer	Throughput	Overhead
Raw disk (SSD)	~5 GB/s	--
Raw gRPC (no encryption)	981 MB/s	5x vs disk
Encrypted gRPC (ChaCha20)	437 MB/s	2.2x vs raw gRPC
FUSE end-to-end	225 MB/s	1.9x vs encrypted gRPC

The encryption layer costs 2.2x. FUSE adds another 1.9x.

Six optimizations

1. Cache the AEAD cipher. The original code created a new ChaCha20-Poly1305 cipher for every message. Caching it in the constructor is safe because the nonce is a monotonic counter.

// Before: creating cipher per-message
aead, _ := chacha20poly1305.NewX(s.key)  // expensive!

// After: created once in constructor
type SecureWriter struct {
    aead cipher.AEAD  // created once
}

2. Single combined TCP write. Two syscalls per message (header + payload) became one. At 512KB chunks, this halves the write() call count.

3. In-place decryption. aead.Open(ciphertext[:0], nonce, ciphertext, nil) reuses the ciphertext buffer instead of allocating a new one. ~2000 fewer allocations per 1GB transfer.

4. Async cache writes in FUSE Read. FUSE needs the data returned to the kernel. Writing that data to the local cache file can happen in a background goroutine. A WaitGroup ensures everything completes before file close. Cache write overhead dropped from 16% to 1.8%.

n := copy(buff, data)
if cacheFD != nil {
    f.CacheWg.Add(1)
    go func() {
        defer f.CacheWg.Done()
        cacheFD.WriteAt(cacheData, cacheOffset)
    }()
}
return n

5. Push-based StreamFile RPC. The original streaming used request-response per chunk. A new server-streaming RPC pushes all chunks without waiting. On-demand reads (random access) still use bidirectional streaming.

6. Silencing hot-path logs. ~120 slog.Info/Debug/Warn calls in FUSE handlers. Even structured logging allocates. Commenting out non-error logs on Read/Write/Getattr gave 2-6% throughput improvement.

After optimizations

Layer	Before	After	Gain
Encrypted gRPC	437 MB/s	441 MB/s	+0.9%
FUSE end-to-end	225 MB/s	231 MB/s	+2.7%

The encryption tax: 1.49x

We built a controlled benchmark comparing identical gRPC transfers over plain TCP vs ChaCha20-Poly1305:

Transport	MB/s	Duration (100MB)
Plain TCP + gRPC	657	152ms
Encrypted (ChaCha20-Poly1305)	441	227ms

33% of transfer time goes to encryption. Our earlier 2.2x estimate was comparing across different test runs with different system load. The controlled A/B benchmark showed the real cost is much lower.

The irreducible 48%

The gap between encrypted gRPC (441 MB/s) and FUSE end-to-end (231 MB/s) is 48%. Every FUSE Read requires two kernel-to-userspace context switches. We confirmed this by measuring raw FUSE overhead (local file read through FUSE vs direct): ~48%, matching exactly.

Plain gRPC:     657 MB/s  (ceiling)
  -33% encryption
Encrypted gRPC: 441 MB/s
  -48% FUSE kernel overhead
FUSE E2E:       231 MB/s

The FUSE overhead is nearly twice the encryption cost. If we wanted to improve throughput, reducing FUSE transitions would have more impact than switching ciphers. But FUSE is the cost of running a userspace filesystem, and for secure P2P file sharing the tradeoff is worth it.

What we learned

Without isolating each layer, we would have optimized the wrong thing. The biggest win (async cache writes, 16% to 1.8%) was only visible when measured independently.
Caching the AEAD cipher and in-place decryption are tiny code changes with measurable impact at 2000+ messages per second.
Structured logging is not free on hot paths. Even checking the log level has overhead.
FUSE has an inherent 48% overhead from context switches. No amount of userspace optimization can fix this.
Server-streaming eliminates round-trip latency for sequential prefetch. Request-response is still needed for random access.
Our initial 2.2x encryption estimate was wrong. A controlled A/B benchmark (same test, same system) showed 1.49x. Always compare apples to apples.

More from the KEIBIDROP blog: full series | product page | FAQ

Make git clone work between encrypted P2P FUSE peers (7 bugs)

Marius-Florin Cristian — Sun, 29 Mar 2026 08:34:07 +0000

Part of the KEIBIDROP development blog. KEIBIDROP is in active development. Release is coming soon.

KEIBIDROP mounts a virtual FUSE filesystem for each peer. When Peer A creates or modifies files, the changes sync to Peer B's mount in real-time through encrypted gRPC. We already made git work on a single FUSE mount (previous post). The question was: what happens when Peer A clones a repo into their mount, and Peer B tries to use it through theirs?

Bob (FUSE mount)                     Alice (FUSE mount)
+-- .git/                            +-- .git/          <- synced from Bob
|   +-- HEAD                         |   +-- HEAD
|   +-- config                       |   +-- config
|   +-- objects/pack/                |   +-- objects/pack/
|   |   +-- pack-abc123.pack (2MB)   |   |   +-- pack-abc123.pack
|   +-- refs/heads/main              |   +-- refs/heads/main
+-- go.mod                           +-- go.mod
+-- README.md                        +-- README.md

Seven bugs appeared across three debugging sessions.

Bug 1: The rename race

Git uses atomic writes: create config.lock, write content, close, rename to config. KEIBIDROP sends ADD_FILE on close and RENAME_FILE on rename. The prefetch goroutine would download config.lock, finish before the RENAME_FILE arrived, and skip the disk rename because the in-memory path hadn't changed yet.

Fix: the RENAME_FILE handler now does os.Rename on disk before updating maps. The prefetch cleanup becomes a redundant safety net.

Bug 2: Pack file corruption (20 missing bytes)

Alice's pack file was 2,182,186 bytes. Bob's was 2,182,206 bytes. Git's index-pack appends a 20-byte SHA-1 checksum after the file is closed but before the rename. KEIBIDROP sent the ADD_FILE notification at close time with the pre-checksum size.

Fix: after renaming on disk, compare local file size with the size in the RENAME_FILE notification. If they differ, trigger a re-download.

Bug 3: macFUSE cache poisoning

git status showed deleted: go.mod even though the file existed on disk. macFUSE's negative_vncache option caches ENOENT results. macOS probed Alice's mount (Spotlight, fsevents) before the file arrived, the kernel cached "doesn't exist", and subsequent reads never reached our FUSE handler.

Fix: remove negative_vncache from mount options.

Bug 4: The 612-file hang

Cloning a large repo (612 files) hung after checkout completed. Every FUSE Release sent a gRPC notification. 612 files closing in rapid succession overwhelmed the connection.

Three-part fix:

BatchNotify RPC -- batch 64 notifications into a single gRPC call
Per-path debounce (200ms deadline, reset on each update) on the sender side
Prefetch semaphore (max 8 concurrent) on the receiver side

rpc BatchNotify (BatchNotifyRequest) returns (BatchNotifyResponse);

message BatchNotifyRequest {
  repeated NotifyRequest notifications = 1;
  uint64 seq = 2;
  uint64 timestamp = 3;
}

Bug 5: 190,000 log lines

After fixing the notification flood, the clone still hung for 30 seconds. Turns out we had 207,000 getattr calls and 190,000 ENOENT error logs, all written synchronously. macOS probes hundreds of paths per directory (Spotlight, fsevents, .DS_Store, resource forks) and without negative_vncache every probe hits our handler.

Fix: remove trace logs from hot-path handlers. Stop logging ENOENT as an error.

grep -c "FUSE getattr" Log_Bob.txt    207,137
grep -c "Failed to lstat" Log_Bob.txt  190,614

Bug 6: Hook permission denied

fatal: cannot exec '.git/hooks/post-checkout': Operation not permitted. macOS Gatekeeper blocks executing scripts from FUSE mounts.

Fix: add defer_permissions to mount options.

Bug 7: LFS corruption (intermediate sizes + stale notifications)

Git LFS downloads a 420MB XML file incrementally into .git/lfs/incomplete/<hash>. Each intermediate close triggers ADD_FILE with the current size. The peer starts prefetching at 100MB, restarts at 200MB, restarts at 300MB. Then LFS renames incomplete/<hash> to objects/<sha>/<hash>, but the debounced ADD_FILE for the old path fires after the rename, pointing to a file that no longer exists.

Fix: per-path debounce with RENAME retargeting. When a RENAME arrives, the pending ADD_FILE for the old path gets its path rewritten to the new path instead of being deleted or sent stale.

case bindings.NotifyType_RENAME_FILE:
    if old, exists := pending[req.OldPath]; exists {
        delete(pending, req.OldPath)
        old.req.Path = req.Path  // retarget to new path
        pending[req.Path] = old
    }
    immediate = append(immediate, req)

We tried three approaches: deleting the pending notification (broke content delivery), sending it stale (wrong path), retargeting (correct).

The result

After all fixes, the full git workflow works between encrypted P2P FUSE peers:

# Bob clones a repo
$ cd MountBob && git clone git@github.com:user/repo.git
Cloning into 'repo'... done.

# Alice sees it immediately
$ cd MountAlice/repo && git status
On branch main -- nothing to commit, working tree clean

# Alice creates a branch and commits
$ git checkout -b test_me
$ echo "test" > tst.txt && git add tst.txt && git commit -m "Commit for test"

# Bob sees the branch and commit
$ cd MountBob/repo && git log
commit 38e23bc (HEAD -> test_me)
    Commit for test

Tested with a small repo (24 objects, 2MB, 8 files) and a large repo (2339 objects, 257MB + 3.71 GiB LFS, 612 files).

What we learned

FUSE handlers must rename on disk, not just in memory. Goroutines race with notification handlers, especially for tiny files that download in milliseconds.
Git writes data, closes the file, then renames. Between close and rename, other processes can modify the file (index-pack appending checksums). The rename notification has the correct size. Use it.
macFUSE's negative_vncache silently caches ENOENT results. Our handler was never called. These bugs don't show up in FUSE logs because the kernel short-circuits the call.
612 files = 612 goroutines = 612 gRPC streams = connection collapse. A bounded channel + batching worker gives backpressure without blocking the caller.
190,000 synchronous log writes caused a 30-second hang. The logging seemed harmless with 20 files. 612 files with macOS probing turned it into 400,000 disk writes.
For debounce + rename: you can't delete the pending notification (peer never gets content) and you can't send it stale (wrong path). Retarget it to the new path. Three attempts to get this right.
None of these bugs are git-specific. Any application that creates files through one peer's mount and reads them through another would hit the same problems. Git just exercises every edge case.

More from the KEIBIDROP blog: full series | product page | FAQ

I built a peer-to-peer file transfer tool with post-quantum encryption

Marius-Florin Cristian — Thu, 26 Mar 2026 08:58:20 +0000

KEIBIDROP is in active development. Release is coming soon.

I've been building KEIBIDROP for the past year. It's a file sharing tool where files go directly from one computer to another. No cloud, no accounts, no middleman.

What it does

Two people run KEIBIDROP, exchange a short code over Signal or Telegram, and connect. Both sides can browse each other's shared files in real time. With FUSE mode, the peer's files show up as a virtual folder in your file manager. You can open, copy, and read files as if they were local.

The connection is encrypted with ChaCha20-Poly1305 over a hybrid ML-KEM + X25519 key exchange. ML-KEM is the NIST post-quantum standard (formerly Kyber). If quantum computers eventually break classical crypto, sessions established today are still protected.

How it's different from existing tools

WeTransfer and Dropbox upload your files to their servers. AirDrop only works between Apple devices. croc and magic-wormhole are one-shot transfers. Syncthing is for background sync across your own devices.

KeibiDrop is for two people who want to share files right now, see each other's files live, and not trust a third party with their data. Full comparison: KEIBIDROP vs alternatives

The stack

Go 1.24 for backend, networking, cryptography, FUSE
Rust + Slint for the native desktop GUI (no Electron, 20MB binary)
gRPC + Protocol Buffers for peer communication
cgofuse for cross-platform FUSE (macOS, Linux, Windows)
~11,000 lines of Go, ~5,000 lines of tests

Three interfaces

Desktop GUI for drag-and-drop. Interactive CLI for terminal users. Agent CLI (kd) with JSON output over Unix socket, built for AI agents and automation.

IPv6 only (for now)

Both peers need IPv6. This keeps the design simple and avoids leaking IP metadata to STUN/TURN servers. Most ISPs support IPv6 today. NAT traversal for IPv4 is planned.

DEV Community: Marius-Florin Cristian

KEIBIDROP: Romania to Singapore Benchmarks

Setup

Bulk throughput

Handshake

FUSE on-demand

What the numbers say

Defects identified

Planned changes

KEIBIDROP: Also on r/selfhosted

KEIBIDROP: Every tool just works

How the shared folder works between two machines

Build tools

Media tools

Data and databases

Languages

File operations

Integrity verification

Security tools

Collaborative workflows

Large files

What does not work

How it works

KeibiDrop: Instant share 600 MB video from phone to Destkop

Mobile apps coming soon!

KEIBIDROP: Secure File Transfer!

KEIBIDROP: Secure File Transfer!

KEIBIDROP v0.1.0 is out! https://github.com/KeibiSoft/KeibiDrop/releases/tag/v0.1.0 Now you can send files, or mount files in real time as a distributed system :D Direct peer to peer or fallback via datarelay. Open source, and no transfer limits!

Release KeibiDrop v0.1.0 · KeibiSoft/KeibiDrop · GitHub

Can We Compute the Minimum Spanning Tree in Linear Time?

Where the bottleneck is

Two computation models

Attack vector 1: Approximate union-find

Attack vector 2: Density partition

Attack vector 3: Cycle hierarchy

How they compose

What's incomplete

Optimizing encrypted P2P file transfer - from 225 to 441 MB/s

Baseline numbers

Six optimizations

After optimizations

The encryption tax: 1.49x

The irreducible 48%

What we learned

Make git clone work between encrypted P2P FUSE peers (7 bugs)

Bug 1: The rename race

Bug 2: Pack file corruption (20 missing bytes)

Bug 3: macFUSE cache poisoning

Bug 4: The 612-file hang

Bug 5: 190,000 log lines

Bug 6: Hook permission denied

Bug 7: LFS corruption (intermediate sizes + stale notifications)

The result

What we learned

I built a peer-to-peer file transfer tool with post-quantum encryption

What it does

How it's different from existing tools

The stack

Three interfaces

IPv6 only (for now)

Links