DEV Community

JKC
JKC

Posted on

Stop Pulling a NuGet Package for zstd. .NET 11 Has ZstandardStream

#performance #dotnet #csharp #softwaredevelopment

TL;DR: .NET 11 puts Zstandard (zstd) right next to GZipStream and BrotliStream. new ZstandardStream(output, CompressionLevel.Optimal) is the whole setup, and HttpClient can decompress zstd responses with one flag.

.NET 11 is at Release Candidate 2, which ships with a go-live support license, but it is not GA yet. Everything below was run on the .NET 11 RC 1 SDK.

The package nobody wanted to own

zstd is everywhere now. Log pipelines, databases, and HTTP all speak it.

.NET, meanwhile, made you pick a third-party wrapper, check that its native binary ships for every platform you deploy to, and hope it is still maintained next year. It is the dependency that shows up in every security review and that nobody remembers adding.

What .NET 11 adds

The new types live in System.IO.Compression:

  • ZstandardStream - a Stream, used exactly like GZipStream.
  • ZstandardEncoder and ZstandardDecoder - span-based, no stream needed.
  • ZstandardDictionary - train a dictionary for lots of small, similar payloads.
  • ZstandardCompressionOptions - quality, window size, checksums, long-distance matching.

On the HTTP side, DecompressionMethods.Zstandard and ZstandardCompressedContent are new too.

The minimal version

If you have ever written a GZipStream, you already know this API:

using System.IO.Compression;

byte[] raw = File.ReadAllBytes("app.log");

using var output = new MemoryStream();
using (var zstd = new ZstandardStream(output, CompressionLevel.Optimal))
{
    zstd.Write(raw);
}
byte[] compressed = output.ToArray();

using var input = new MemoryStream(compressed);
using var unzstd = new ZstandardStream(input, CompressionMode.Decompress);
using var back = new MemoryStream();
unzstd.CopyTo(back);
Enter fullscreen mode Exit fullscreen mode

Swap GZipStream for ZstandardStream and the rest of your code does not notice.

I ran all three built-in compressors over the same payload: 2,000 lines of fake JSON logs, 188,010 bytes, all at CompressionLevel.Optimal.

Format Compressed size 200 runs on my machine
GZip 13,499 bytes 218 ms
Brotli 11,936 bytes 299 ms
Zstandard 11,891 bytes 150 ms

Smallest and fastest in one row. It is a synthetic, very repetitive payload and one run on one machine, so treat the timings as a hint, not a benchmark. The .NET team's own numbers on the Canterbury Corpus are in the Preview 1 release notes linked below.

Want more squeeze? ZstandardCompressionOptions takes a Quality from -131072 to 22, and the default is 3.

var options = new ZstandardCompressionOptions
{
    Quality = 19,
    AppendChecksum = true
};
using var zstd = new ZstandardStream(output, options);
Enter fullscreen mode Exit fullscreen mode

At quality 19 with a checksum, the same log payload came out at 7,535 bytes.

The realistic one: thousands of tiny messages

Compression is bad at tiny payloads. A 150-byte JSON event has almost no repetition inside it, so there is little to squeeze. The repetition is between messages: the same keys, the same enum values, the same shape, every time.

That is what zstd dictionaries are for. You train one on sample messages, then both sides use it:

using System.IO.Compression;

// training: byte[][] of real messages, e.g. the last 1,000 events
// message: one new event as byte[]
byte[] samples = [.. training.SelectMany(b => b)];
int[] lengths = [.. training.Select(b => b.Length)];

using ZstandardDictionary dictionary =
    ZstandardDictionary.Train(samples, lengths, maxDictionarySize: 4096);

byte[] buffer = new byte[1024];
ZstandardEncoder.TryCompress(
    message, buffer, out int written,
    dictionary, ZstandardCompressionOptions.DefaultWindowLog2);

byte[] restored = new byte[message.Length];
ZstandardDecoder.TryDecompress(
    buffer.AsSpan(0, written), restored, out int read, dictionary);
Enter fullscreen mode Exit fullscreen mode

I trained a 4,096-byte dictionary on 1,000 order.created events, then compressed 200 new events it had never seen:

Average size
Raw JSON 152.2 bytes
zstd, no dictionary 134.7 bytes
zstd with dictionary 38.6 bytes

Without a dictionary, zstd barely dented them. With one, the average message went from 152.2 bytes to 38.6. That is the difference between "compression is not worth it for our queue" and "why did we not do this sooner".

HttpClient speaks zstd now

Automatic decompression is one flag:

using var client = new HttpClient(new SocketsHttpHandler
{
    AutomaticDecompression = DecompressionMethods.Zstandard | DecompressionMethods.GZip
});

string body = await client.GetStringAsync("https://example.com/api/report");
Enter fullscreen mode Exit fullscreen mode

With that handler, the client sent Accept-Encoding: gzip, zstd. My local test server answered 8,000 bytes of JSON as 33 bytes of zstd, and GetStringAsync handed back all 8,000 characters.

Going the other way, ZstandardCompressedContent compresses a request body while it is sent and sets Content-Encoding: zstd for you:

var json = new StringContent(payload, Encoding.UTF8, "application/json");
await client.PostAsync(url, new ZstandardCompressedContent(json));
Enter fullscreen mode Exit fullscreen mode

300 copies of a small JSON object reached my test server as 36 bytes. Make sure the server actually accepts zstd request bodies before you turn this on.

Can I use it?

  • .NET 11 only. The types ship in the shared framework; my test apps had no package references at all.
  • Status: .NET 11 is at RC 2 with a go-live license; GA is still ahead.

Gotchas

  1. The int after the dictionary is not quality. In TryCompress(source, dest, out written, dictionary, 3), that 3 is windowLog2, and the call threw ArgumentOutOfRangeException: windowLog2 ('3') must be greater than or equal to '10'. Pass ZstandardCompressionOptions.DefaultWindowLog2 (23), or set quality with ZstandardDictionary.Create(buffer, quality).

  2. Both sides need the same dictionary. Decoding a dictionary-compressed message without it did not throw; TryDecompress just returned false. Version your dictionaries and ship the id with the message.

  3. HttpClient does not opt in by itself. A default HttpClient sent no Accept-Encoding at all. When my test server sent zstd anyway, the default client got the 33 raw zstd bytes, not the JSON.

  4. Preview samples are already stale. The Preview 1 notes use WindowLog; the RC has WindowLog2, MinWindowLog2, and MaxWindowLog2. Trust the API docs over old blog snippets, this one included once GA lands.

Wrap-up

Zstandard is now just another Stream in the box, with dictionaries and HTTP support on top. Grab the .NET 11 RC SDK, swap one GZipStream for ZstandardStream, and see what your payloads say.

Sources

This article was written with the help of AI and checked against the sources above.

Top comments (0)