DEV Community: Peter Gebri

Go 1.25 Greentea GC vs Classic: HydrAIDE 1M Swamp Test Shows +22% CPU Efficiency, -8% Memory

Peter Gebri — Sun, 31 Aug 2025 20:05:20 +0000

Go 1.25 new Greentea GC vs. classic GC. 1 Million Swamp stress test with HydrAIDE: +22% CPU efficiency, -8% memory, but rarer long spikes

Introduction – Why now, why HydrAIDE?

With the release of Go 1.25, an exciting experimental feature has arrived: the Greentea GC, a new garbage collector mode that promises to reduce GC overhead by 10–40%. But what does this mean in practice? Does it bring noticeable speed-ups, or is it just theoretical fine-tuning?

This is where HydrAIDE comes into play: an open-source (Apache 2.0) data engine that simultaneously provides strongly typed storage, pub/sub, caching, and real-time data services — all without additional infrastructure. Data is always hydrated into memory, and references are automatically dropped when no longer needed, allowing the GC to immediately reclaim memory.

This model makes HydrAIDE uniquely suited to test the behavior of Go’s new GC under real, production-like loads. No need for synthetic microbenchmarks — we can just create 1 million Swamps and wait for the GC to do its job.

👉 Direct test code: hydra_gc_test.go
👉 Full project: github.com/hydraide/hydraide

Good to know – How HydrAIDE manages Swamps

HydrAIDE stores all data in Swamps (mini-databases). These appear as physical folders in the filesystem, with each Swamp fully isolated. When a Swamp is accessed for the first time, the system automatically hydrates it into memory — loading the data into RAM. If the Swamp remains idle for longer, HydrAIDE closes it and removes it from memory while keeping the content safely persisted on disk. This means memory always only contains what is needed, and only for as long as it is needed.

This logic ensures efficient memory usage, automatically keeping the GC environment clean.

Why is this test special?

Real allocation pressure: creating 1 million Swamps, each with its own Treasure. For those familiar with traditional databases, imagine opening 1 million separate mini-databases (tables or collections), each with at least one record. This generates an enormous number of objects on the heap and creates a true stress scenario for the GC.
Deterministic idle-close: after 30s of inactivity, HydrAIDE automatically clears references → all objects instantly become GC-eligible. Think of it like dropping all pointers to a table’s rows at once: the data remains on disk, but memory is free. This gives the GC a crystal-clear signal about what can be reclaimed.
GC-close metrics: we used the runtime/metrics API directly → precise CPU, heap, and pause times. Introduced in Go 1.20, this API provides stable, low-level runtime measurements without external profilers. It allows us to track GC cycles, CPU usage, heap size, and pause distributions.
Straightforward comparison: the same binary runs once with the legacy GC, once with Greentea GC.

Background – What does Greentea GC promise?

Greentea GC is an experimental collector introduced in Go 1.25 that improves marking/scanning of small objects with better locality and CPU scalability. The Go team expects 10–40% GC overhead reduction in real workloads. Enabling it: build with GOEXPERIMENT=greenteagc. Go 1.25 also introduced container-aware dynamic GOMAXPROCS and the runtime/trace Flight Recorder, improving reproducibility and diagnostics.

How we measured the test

We defined two distinct phases to measure GC behavior. For this we used the runtime/metrics API, which reports GC cycles, CPU usage, heap sizes, and pause time histograms.

Phase A – Create burst: we created 1,000,000 Swamps, each with at least one Treasure. For the test, we specifically used HydrAIDE’s in-memory Swamp mode instead of filesystem-based Swamps, ensuring no disk I/O influenced the results. Each creation included a small write and save in memory, simulating a real, high-allocation workload. This produced a massive number of heap objects, ideal for stress-testing the GC.
Phase B – Idle-close window: HydrAIDE’s built-in idle-close logic was used: after 30 seconds of inactivity, all Swamps automatically closed, dropping references in memory. We then waited another 5 seconds to ensure everything was GC-eligible. In this phase, we measured the GC’s behavior during reclamation and memory release.

Important: this phase also ran entirely in-memory, with no disk writes. HydrAIDE’s idle-close mechanism guarantees references vanish deterministically, giving the GC clear visibility. This ensures reproducibility and measurement accuracy.

What results did we get?

The following shows how the classic GC and the new Greentea GC performed across the two phases.

Summary table

Metric	Greentea GC	Classic GC	Notes
⏱️ Runtime (Phase A)	22.94s	24.30s	~5% faster with Greentea
⚙️ Total GC CPU	21.33s	27.35s	~22% less CPU → more efficient
⏸️ Total GC Pause	0.03s	0.02s	practically identical, negligible difference
🔍 Mark Assist	11.42s	14.05s	less burden during mutation phase
🔍 Mark Dedicated	7.6s	10.07s	shorter dedicated time
🔍 Mark Idle	2.28s	3.22s	less idle CPU
💾 Heap size (end)	3.80 GB	4.12 GB	~8% smaller heap, faster memory release
💤 Idle-phase GC cycles	0	0	no extra overhead
📊 Pause p50	0.06 ms	0.05 ms	nearly identical
📊 Pause p95	0.20 ms	0.23 ms	nearly identical
📊 Pause p99	1.84 ms	0.92 ms	Greentea shows rarer but longer pauses

Interpretation in words

Speed: Greentea GC delivered ~5% faster runtime under the create burst.
CPU efficiency: it used 22% less CPU time for GC — the biggest win.
Memory usage: it finished with 8% smaller heap → better memory reclamation.
Pause times: pause times represent short stop-the-world events where all goroutines are paused for GC. On average (p50, p95) they are the same, but p99 shows Greentea occasionally introduces longer pauses.
Idle phase: neither GC triggered extra cycles → stable operation.

Conclusion

👉 Greentea GC is clearly advantageous when CPU and memory efficiency are critical. However, in latency-sensitive applications (e.g., real-time APIs), one must consider the occasional longer p99 pauses.

For HydrAIDE, this has particular significance: handling massive numbers of Swamps and Treasures in memory means every GC improvement directly translates into speedups and lower resource usage. Running this test was a pleasure. It demonstrated how effective the Go GC already was, and how much this new Greentea enhancement benefits systems like HydrAIDE.

Special thanks go to the Go developers for this excellent improvement.

Links and contacts

👉 Direct test code: hydra_gc_test.go
👉 Full project: github.com/hydraide/hydraide
👉 Community & support: HydrAIDE Discord
👉 Contact: peter.gebri@hydraide.io

🚀 How I Replaced Pub/Sub, Cache, and Database with a Single Tool: HydrAIDE

Peter Gebri — Fri, 15 Aug 2025 07:32:08 +0000

🚀 How I Replaced Pub/Sub, Cache, and Database with a Single Tool: HydrAIDE

GitHub: https://github.com/hydraide/hydraide

The Brainf@ck

Do you know that feeling when, instead of writing code, you get dragged into playing full-time infrastructure admin because you have too many external systems to keep alive? And the frustration of making sure your application plays nicely with each of them, which means you basically have to become an expert in all three?
That’s exactly why I built HydrAIDE: a single, Go-based data engine that replaced my pub/sub broker, memory cache and database.

If you’re tired of juggling separate tools for messaging, caching, and storage, and just want to develop applications quickly and efficiently (without learning another language), read on. This will change how you think about data.

😬 The Problem

In the Trendizz ecosystem — a large-scale, real-time data processing and delivery platform — we process millions of records every single day. This isn’t a few queries or thousands of rows; it’s the kind of load that melts most databases if you expect them to handle it all on a single server.

Our requirements were crystal clear:

Real-time data updates across multiple services (pub/sub)
Lightning-fast queries (cache)
Persistent, structured storage (database)
Automatic record expiry — without extra cron jobs
All without separate services

We tried Redis + Mongo + Kafka. It didn’t work.
There was no way to index the entire European internet on one server with that stack. It was also fragile, full of glue code, schema mismatches, multiple serialization formats, and painful to install/scale.

Eventually, I realized I was spending more time babysitting infrastructure than building the actual product. We kept hitting multi-system issues with no clean fix: deadlocks, slow tables, and other headaches.

That’s when I decided: I’ll build HydrAIDE — one engine to do everything those three systems did, but simpler, faster, and type-safe.

In short: too much complexity for something that could be simple. I want to code in the language I love, not manage infra.

💡 The Solution: HydrAIDE

HydrAIDE is an Adaptive Intelligent Data Engine, written in Go over two years, with 3.5+ years of battle testing. One engine that:

Stores data like a database
Caches hot data automatically
Sends events on every change (no separate broker)
Handles per-record expiry with an expireAt field
Loads into memory only when needed

It’s strongly typed, stores in binary, and is reactive out of the box.

┌───────────┐ + ┌────────┐ + ┌──────────────┐ ❌
│ Database  │   │ Cache  │   │ Pub/Sub Bus  │
└───────────┘   └────────┘   └──────────────┘
       ⬇
┌───────────────────────────────┐
│         HydrAIDE Engine       │ ✅
└───────────────────────────────┘

🛠️ How It Works

Intent-first, struct-first. First, I think about what data I need, where, and how it will behave in the app (lifetime, access pattern). That intent is expressed in names (Sanctuary / Realm / Swamp) and Go structs.
The beauty: you just write structs — the SDK handles saving, loading, events, and expiry.

In Practice:

You write structs — they’re your data models.
You define a naming function (e.g., users/profiles/<id> or chat/room/<roomID>). These namespaces are like mini-databases: blazing fast, infinitely scalable.
You call SDK methods: ProfileSave/Read, CatalogSave/Read, Subscribe, CatalogShiftExpired, etc.

🧩 Profile Example — "One Entity = One Swamp"

// users/profiles/<UserID>
type UserProfile struct {
    UserID       string
    Email        string
    Username     string
    RegisteredAt time.Time `hydraide:"omitempty"`
}

func (p *UserProfile) Name() name.Name {
    return name.New().Sanctuary("users").Realm("profiles").Swamp(p.UserID)
}

func (p *UserProfile) Save(h hydraidego.Hydraidego) error { return h.ProfileSave(ctx(), p.Name(), p) }
func (p *UserProfile) Load(h hydraidego.Hydraidego) error { return h.ProfileRead(ctx(), p.Name(), p) }

// usage
in := &UserProfile{UserID: "user-123", Email: "hello@example.com", Username: "peter"}
_ = in.Save(h)

out := &UserProfile{UserID: "user-123"}
_ = out.Load(h)

Here, the entire profile is saved/loaded in a single call; no schema, no migrations.

📚 Catalog Example — "Many Key-Values in One Swamp"

// chat/room/<RoomID>
type ChatMessage struct {
    MessageID string    `hydraide:"key"`
    Text      string    `hydraide:"value"`
    ExpireAt  time.Time `hydraide:"expireAt"`
}

func room(roomID string) name.Name {
    return name.New().Sanctuary("chat").Realm("room").Swamp(roomID)
}

// save (upsert) with TTL
msg := &ChatMessage{MessageID: "m1", Text: "hello", ExpireAt: time.Now().Add(1*time.Minute).UTC()}
_, _ = h.CatalogSave(ctx(), room("42"), msg)

// read by key
var got ChatMessage
_ = h.CatalogRead(ctx(), room("42"), "m1", &got)

Catalogs are great for lists/queues/logs; records can have expireAt, and every write emits an event.

⏳ Why `expireAt` is a Game-Changer

In the catalog example, the expireAt field isn’t just about TTL. It lets you control processes at the database level.

We use it to orchestrate a web crawler network: a domain can only be recrawled if its expireAt has passed. Similarly, it can drive microservices without extra business logic.

expireAt works per record:

Set it on save/update
Fetch expired records with CatalogShiftExpired
Build scheduled workflows directly from data state

⚡ Built-in Pub/Sub

Every write/update/delete triggers a real-time event for subscribers.

Example subscription:

err := h.Subscribe(ctx(), name.New().Sanctuary("chat").Realm("room").Swamp("42"), func(e hydraidego.Event) {
    log.Printf("Event: %+v", e)
})

No polling, no extra broker, instant reaction.

📊 Why It Matters

HydrAIDE lets us focus on coding, not infrastructure. Anyone who knows Go can pick it up in a day. Once your mindset shifts, you’ll wonder: Why didn’t I always do it this way?

Less infrastructure
Type-safe
Reactive by default
Horizontally scalable
Open Source

🔗 Try It

Join the growing HydrAIDERS community and supercharge your development!

GitHub: https://github.com/hydraide/hydraide
Discord: https://discord.gg/xE2YSkzFRm

If you’re juggling too many data tools, HydrAIDE can help you throw them out and sleep better. :D

Péter Gebri
Creator of HydrAIDE

🐱 Schrödinger’s Index: It Only Exists When You Observe It and Only the Way You Observe It – in the HydrAIDE Engine

Peter Gebri — Sat, 09 Aug 2025 07:40:01 +0000

🔍 We have no indexes, but...

We have no indexes. Then, the moment you ask for them, they suddenly exist — but only for as long as needed, and only in the way you observe them. 🤯 WTF?! This is the real Schrödinger’s Index. It’s only there when you look at it, and only in the form you see it.

This isn’t clickbait — it’s quantum mechanics in action 😄

Previously, I wrote about how I made millions of European websites searchable from a single server using HydrAIDE.: https://dev.to/hydraide/how-i-made-europe-searchable-from-a-single-server-the-story-of-hydraide-432h

Now, I’ll show you how HydrAIDE solved memory management and SSD issues that most databases suffer from due to old-fashioned indexing strategies, thanks to Schrödinger’s Indexes.

❓ What Is Schrödinger’s Index?

Schrödinger’s Index is a completely unique development in the HydrAIDE data engine — born from a different way of thinking.

In most databases, indexes work like this:

Separate persistent index structures (B-Tree/GIN/Hash) are built alongside tables and stored on disk.
Every INSERT/UPDATE/DELETE updates the data and all related indexes.
Indexes require separate maintenance (rebuild/vacuum/analyze, updating statistics).
Maintenance consumes memory (buffer cache), storage, and CPU via background processes.

This causes problems like:

Write amplification: higher write latency and lower throughput with heavy writes.
Extra storage cost: data stored multiple times (data + multiple indexes) = bigger SSD bills.
Locks and contention: large table/index updates can cause blocking and wait times.
Operational overhead: index rebuilds, DDL changes, migrations reduce performance or require maintenance windows.
Over-indexing risk: poorly chosen or outdated indexes still consume resources without helping.

🕰 Looking Back: Why Always On-Disk Indexes?

I get why this was necessary in the early 2000s. Back then, with slow HDDs, file-based indexing solved a real problem. But it’s not the early 2000s anymore — it’s been 25 years. Today’s M.2 SSD speeds are insane, and RAM is on another level entirely. So why cling to the old ways?

When building HydrAIDE, I decided to start from scratch — focusing on modern hardware and possibilities — and lay entirely new foundations, including a new indexing strategy.

⚡ How It Works in HydrAIDE

Unlike traditional databases, HydrAIDE doesn’t create physical indexes for querying by ID, value, or metadata. No data duplication to SSDs — the design makes it unnecessary.

When HydrAIDE loads a swamp into memory (a small, targeted “mini-database”), it creates a map[key]value structure in Go. Go maps return values at brutal speed via direct lookups — if you know the key.

But what if you don’t know the key, and instead want sorted data from a swamp — say, by insertion time descending, or sorted by value?

This is where Schrödinger’s Index comes in: on the first such query, the HydrAIDE engine temporarily sorts the data from the base map into a slice in memory (with a pointer to the value and metadata) according to your requested criteria — and returns the results immediately.

So, the sorted key set ONLY EXISTS IF YOU ASK FOR IT — and only in the way you observe it.

The built “index” lives only in memory. When HydrAIDE — per your settings — clears the swamp from memory, the index disappears with it. No SSD duplication, and memory stays clean.

Sorting is lightning-fast and highly efficient in Go and HydrAIDE, so in practice, these temporary indexes won’t burden your system.

🚫 No Background Index Processes

In HydrAIDE, there are no background processes managing indexes — because there’s no traditional indexing at all. That’s why HydrAIDE has virtually no idle CPU or RAM usage, yet still performs its job effectively — just not via the “traditional” route.

✅ Why It’s Not Clickbait

The index truly exists only at the moment of observation — and only in the form you observe it. 😄 Before and after that? It could be anything 😂

This system has been a game-changer for me. Without Schrödinger’s Indexes, I couldn’t have handled Europe’s web content index on a single server and served searches from it. I’m convinced this technical approach works for everything from the smallest projects to the largest ones. For reactive projects where data changes rapidly (making constant indexing a heavy burden), it’s outright essential.

💬 Your Turn

What do you think? Do you like the idea of Schrödinger’s Index? Would you use HydrAIDE as your database? I’m happy to answer questions in the comments.

You can try the engine in action on our GitHub repo. A 2‑minute install and Schrödinger’s Index comes alive:
https://github.com/hydraide/hydraide

How I Made Europe Searchable From a Single Server - The Story of HydrAIDE

Peter Gebri — Fri, 01 Aug 2025 17:10:15 +0000

How I Made Europe Searchable From a Single Server — The Story of HydrAIDE

🚀 1. From One Server to Millions of Pages

I made millions of websites across Central and Western Europe searchable — from a single server, and I barely crossed 3% CPU load doing it. No Redis. No Kafka. No Elasticsearch. Not a trick. Not AI. Not a crawler hack. I built my own data engine.

👉 GitHub: https://github.com/hydraide/hydraide

🧭 2. Why I Had to Build It

I've been programming for 30 years. The last 10+ were deep in backend systems — high-concurrency, high-load Go services.

In 2021, we launched Trendizz.com, a startup to help businesses find their best B2B partners across Europe — through precise, micro-segmented search.

We needed to answer questions like:

"Which Hungarian companies sell bicycles, don’t use GLS shipping, and run an Unas webshop?"

To do that, we had to index millions of websites. Not just metadata — word-level matches, across layers of content.

And that’s when we realized: no database could handle it. And we didn’t have millions in infra budget. Just the need to make it work.

🧱 3. Hitting the Wall With Databases

We tried everything: SQL, NoSQL, document stores, graph DBs, even exotic stuff.

SQL? Slows down hard after a few million records. Query optimization is its own career.
NoSQL? Assumes everything fits in RAM. Funny, when your data hits terabytes.
Cloud? Egress costs would’ve buried us. We wanted control — not billing tiers.

We knew we had to think differently.

Most databases are still based on early-2000s assumptions: single-core CPUs, spinning disks, batch jobs.

But today?

SSDs are monsters.
RAM is fast and cheap.
CPUs love concurrency.

Why should everything live in memory? Why can’t I decide — from code — what to load, and when?

📦 4. The Power of Small Files

That’s when the idea came. What if I skipped the huge database files — and just used lots of small, precisely named binary files I could jump to instantly?

Classic databases jam everything into one massive file, causing brutal I/O overhead. In HydrAIDE, targeting a file is an O(1) operation, and SSDs return it near-instantly.

Files are minimal: no extra index, no cache, no journal. Just the data — exactly where it should be.

The beauty? You only load what you need, and it's blazing fast. And if you don’t need something anymore? Just drop it from memory. You can control all of this — directly from the SDK.

🛠️ 5. The First Prototype

I built a prototype. And it wasn’t just fast — it was shockingly memory-efficient. The engine didn’t strain — it flew.

Side note: HydrAIDE can insert or read millions of records on a single thread, near hardware speed.

📜 6. Core Principles I Never Want to Rewrite Again

I laid down a few non-negotiable rules for what this engine must be:

Everything must be defined in code (Go SDK, extensible to other languages)
No query language
Realtime by default — no pub/sub middleware needed
No persistent indexes, yet still fast as hell
Delete means delete — no background cleanup jobs
No orchestrator required, yet still horizontally scalable
Nodes must be stateless
And I have to enjoy working with it.

🐉 7. HydrAIDE Is Born

This became HydrAIDE — Hydra Adaptive Intelligent Data Engine. A system that followed my thinking, not a legacy abstraction.

Then came 2 years of focused development and testing, and the crawling of tens of millions of sites. Billions of keyword pairs, technical attributes, and signals made searchable.

Now, in July 2025, HydrAIDE is finally fully open-source — and ready.

💬 8. Early Reactions

The first public release went better than I imagined. We hit 80 GitHub stars in the first few days, and a 7-person contributor group formed almost instantly.

Brilliant devs. Smart feedback. Real commits. Huge thanks to everyone who jumped in early — you're part of this.

⏱️ 9. Where We Are Now

HydrAIDE runs on a gRPC server. With the Go reference SDK, getting started takes minutes. A Python SDK is in the works, and Node.js and Rust are next.

We’re moving fast — because you can’t chain a Hydra.

🌍 10. So How Did Europe Become Searchable?

How did I store billions of keyword relationships from tens of millions of sites?

Now you know. HydrAIDE made it possible.

I built a headless crawler fleet, powered by real browser sessions.
I built a blazing-fast engine that runs even in "one-server zero-footprint" setups.
I built a frontend in Angular, backed by Go, with HydrAIDE’s reactive flow underneath.

And I did it without huge infra, no investors, no excuses — just new thinking and time.

🤝 11. Now It’s Yours Too

This engine is now open. Not a SaaS. Not a product. A working developer’s dream — now yours too.

📌 Try it, contribute, or explore:

GitHub: https://github.com/hydraide/hydraide
Discord: https://discord.gg/aBfAuYjR

Got questions about how it works or how to build with it? Just ask. I’ll explain everything.

DEV Community: Peter Gebri

Go 1.25 Greentea GC vs Classic: HydrAIDE 1M Swamp Test Shows +22% CPU Efficiency, -8% Memory

Go 1.25 new Greentea GC vs. classic GC. 1 Million Swamp stress test with HydrAIDE: +22% CPU efficiency, -8% memory, but rarer long spikes

Introduction – Why now, why HydrAIDE?

Good to know – How HydrAIDE manages Swamps

Why is this test special?

Background – What does Greentea GC promise?

How we measured the test

What results did we get?

Summary table

Interpretation in words

Conclusion

Links and contacts

🚀 How I Replaced Pub/Sub, Cache, and Database with a Single Tool: HydrAIDE

🚀 How I Replaced Pub/Sub, Cache, and Database with a Single Tool: HydrAIDE

The Brainf@ck

😬 The Problem

💡 The Solution: HydrAIDE

🛠️ How It Works

In Practice:

🧩 Profile Example — "One Entity = One Swamp"

📚 Catalog Example — "Many Key-Values in One Swamp"

⏳ Why expireAt is a Game-Changer

⚡ Built-in Pub/Sub

📊 Why It Matters

🔗 Try It

🐱 Schrödinger’s Index: It Only Exists When You Observe It and Only the Way You Observe It – in the HydrAIDE Engine

🔍 We have no indexes, but...

❓ What Is Schrödinger’s Index?

🕰 Looking Back: Why Always On-Disk Indexes?

⚡ How It Works in HydrAIDE

🚫 No Background Index Processes

✅ Why It’s Not Clickbait

💬 Your Turn

How I Made Europe Searchable From a Single Server - The Story of HydrAIDE

How I Made Europe Searchable From a Single Server — The Story of HydrAIDE

🚀 1. From One Server to Millions of Pages

🧭 2. Why I Had to Build It

🧱 3. Hitting the Wall With Databases

📦 4. The Power of Small Files

🛠️ 5. The First Prototype

📜 6. Core Principles I Never Want to Rewrite Again

🐉 7. HydrAIDE Is Born

💬 8. Early Reactions

⏱️ 9. Where We Are Now

🌍 10. So How Did Europe Become Searchable?

🤝 11. Now It’s Yours Too

⏳ Why `expireAt` is a Game-Changer