FATAL: crypto/rand: blocked on getrandom() syscall, or 100% CPU at the 25th Client
The title is catchy enough—now let’s get to the point.
This error is a classic side effect of the default Pion WebRTC setup in Go. It is easy to reproduce: you read the usual Pion usage guides, deploy a nice-looking WHEP handler, open a couple of—okay, okay, not literally a couple, just a figure of speech—browser tabs with the player, and suddenly the server starts choking.
Handshakes begin taking seconds, ICE times out, and half of the pprof flame graph is filled with crypto/elliptic.p256OrdSqr and map allocations somewhere inside the engine.
Surprised?
There is really nothing surprising about it if you read most WebRTC tutorials for Go a little more carefully. I get the impression that many of them were written by people who never load-tested their implementation with even fifty concurrent viewers—at most a couple of streams, and that was apparently enough for them.
Here is what they usually do: for every POST request containing an SDP offer, they create a new webrtc.NewAPI(), register the default codecs, call api.NewPeerConnection(), and happily return the response.
On localhost, with a couple of clients, it flies.
In production, however, it turns into a disaster.
The problem here is not WebRTC itself—and certainly not Pion, which is a great library—and it is not Go either.
The problem is that expensive global “infrastructure” is being created for every single request instead of simply being reused.
Let’s dig into what is actually happening
A small disclaimer: from this point on, I am not going to explain every single term in detail. If something is unclear, it is better to check the documentation or ask in the comments—otherwise, this would turn into an extremely long post.
The first expensive operation is generating cryptographic material for DTLS.
For this, ECDSA P-256 keys are generated and later used to create the DTLS certificate. It is important to understand that key generation is not just a couple of constructor calls and writing a few bytes into memory.
Under the hood, it uses crypto/ecdsa, heavy elliptic-curve mathematics, and cryptographically secure randomness from crypto/rand.
The second expensive operation is allocating hundreds of small structures on the heap for all possible codecs, RTCP interceptors, RTP headers, header extensions, and other internal components.
None of these allocations are exactly free.
The third expensive operation happens at the end: the implementation goes into the OS and opens a bunch of random UDP ports for ICE candidate gathering and binding.
In a typical configuration, this means allocating new network resources for every PeerConnection.
So, for every new connection, the pipeline looks roughly like this:
New viewer
│
├── ECDSA P-256
│ └── crypto/ecdsa + crypto/rand
│
├── Creating and initializing WebRTC structures
│ └── hundreds of small heap allocations
│
├── Registering codecs and interceptors
│
├── Creating the network transport
│
└── UDP sockets + ICE candidate gathering
And what is the result?
Thirty viewers connect simultaneously—the CPU gets hammered by key generation, the connection tracking table starts ballooning, and the garbage collector strikes a pose while trying to clean up tons of dead objects left behind after every closed connection.
There is no point in building workarounds here.
The architecture needs to be designed properly from the beginning:
everything that can be reused should be reused.
The solution we used
In RUSEON Core, all of the heavy work was moved into an isolated singleton engine in internal/webrtc/engine.go.
Certificate generation now happens exactly once through:
ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
when the server starts.
The browser does not care whether the certificate was created a second ago or has been sitting in RAM for months. All it needs is a cryptographically valid fingerprint in the SDP offer—and nothing more.
The codec registry is initialized at startup as well and no longer constantly touches the heap.
The network layer is also usually a bottleneck.
But instead of opening dozens of random ports, we attached webrtc.NewICEUDPMux to a single UDP socket.
One port is used by the entire system.
The OS kernel multiplexes all incoming STUN, DTLS, and SRTP traffic through a single file descriptor, while Pion distributes packets between ICE and WebRTC sessions in userspace.
No port exhaustion.
No need to dance around opening a 50000–60000 port range in the firewall.
On top of that, an engine pool is used through sync.Pool:
var WebRTCEnginePool = sync.Pool{
New: func() interface{} {
return newWebRTCEngine()
},
}
Important: sync.Pool is not used here as guaranteed storage for a predefined set of objects.
Its purpose is to allow temporary engine objects to be reused and reduce unnecessary allocations on the hot path. If the runtime clears the pool, New simply creates a new instance.
When a WHEP request arrives in internal/api/handler.go, the handler does not create anything unnecessary.
It retrieves an already warmed-up and ready instance, feeds it the pre-generated certificate through:
baseConfig.Certificates = []webrtc.Certificate{*e.certificate}
and immediately returns the SDP response.
In practice, the time from request to response is reduced to just a couple of system calls—not counting, of course, the creation of the PeerConnection itself, DTLS/ICE configuration, and SDP negotiation.
The end result is that a new viewer no longer forces the entire WebRTC stack to be created from scratch.
It creates only its own connection-specific state on top of an already prepared engine.
Pitfalls when using the H.264 codec
If you simply take NALU packets from the ring buffer and feed them directly into the stream, the video on the client will constantly freeze.
In our case, the browser’s hardware decoder requires clean Annex B with proper 0x00 0x00 0x00 0x01 delimiters, and every key IDR frame must include the stream’s SPS and PPS parameters.
One of the solutions we used was to allocate a pre-sized 100 KB byte buffer in internal/webrtc/muxer.go.
This makes it possible to reliably avoid slice reallocations while concatenating headers at 30 FPS.
Why does this matter at all?
Because append() on the hot path can be a scary thing.
As long as there is enough capacity, everything is fine.
But once there is not, Go allocates a new array, copies the old data into it, and the old array eventually ends up as dinner for the garbage collector.
At 30 frames per second, this can mean dozens of additional allocations for every stream.
And you never really know when the accumulated garbage will become significant enough to trigger the collector.
Multiply that by the number of streams and viewers, and I think the outcome is obvious.
The whole idea comes down to this: if the approximate size of a working buffer is known and bounded, allocate it once and reuse it.
Do not allow a slice to grow dynamically at every stage of the hot pipeline.
The numbers
For testing, we used our own load-testing utility, cmd/loadtest/main.go.
It simultaneously ran:
- 50 synthetic cameras
- the REST API
- 30 HLS clients
- 30 active WebRTC viewers
Results:
[WebRTC WHEP] Sessions OK: 30 (err: 0) | RTP Packets: 488 214 | Egress: 9.04 MB/s
Handshake Latency: p50=3.21ms | p95=6.84ms | max=11.20ms
HeapAlloc: 34MB | GC Pause Total: 3.82ms | Goroutines: 142
Almost half a million RTP packets, a stable 9 MB/s of outgoing traffic, and exactly zero negotiation errors.
Handshake latency dropped from 800+ ms to 3 milliseconds.
Final thoughts
The main takeaway from this article is simple:
If WebRTC is not “handling” real video workloads, do not blindly trust the tutorials.
Especially when those tutorials suggest solving such problems by pulling in heavy CGO bindings to C++.
Our experience shows that the bottleneck is not the runtime itself, but rather mindlessly regenerating cryptography for every little thing and constantly hitting the allocator on the hot path.
WebRTC is not slow. The architecture around it is slow.
Source code: https://github.com/RUSEGAL/ruseon-core
Top comments (0)