🚀 How I Replaced Pub/Sub, Cache, and Database with a Single Tool: HydrAIDE

🚀 How I Replaced Pub/Sub, Cache, and Database with a Single Tool: HydrAIDE

GitHub: https://github.com/hydraide/hydraide

The Brainf@ck

Do you know that feeling when, instead of writing code, you get dragged into playing full-time infrastructure admin because you have too many external systems to keep alive? And the frustration of making sure your application plays nicely with each of them, which means you basically have to become an expert in all three?
That’s exactly why I built HydrAIDE: a single, Go-based data engine that replaced my pub/sub broker, memory cache and database.

If you’re tired of juggling separate tools for messaging, caching, and storage, and just want to develop applications quickly and efficiently (without learning another language), read on. This will change how you think about data.

---

😬 The Problem

In the Trendizz ecosystem — a large-scale, real-time data processing and delivery platform — we process millions of records every single day. This isn’t a few queries or thousands of rows; it’s the kind of load that melts most databases if you expect them to handle it all on a single server.

Our requirements were crystal clear:

• Real-time data updates across multiple services (pub/sub)
• Lightning-fast queries (cache)
• Persistent, structured storage (database)
• Automatic record expiry — without extra cron jobs
• All without separate services

We tried Redis + Mongo + Kafka. It didn’t work.
There was no way to index the entire European internet on one server with that stack. It was also fragile, full of glue code, schema mismatches, multiple serialization formats, and painful to install/scale.

Eventually, I realized I was spending more time babysitting infrastructure than building the actual product. We kept hitting multi-system issues with no clean fix: deadlocks, slow tables, and other headaches.

That’s when I decided: I’ll build HydrAIDE — one engine to do everything those three systems did, but simpler, faster, and type-safe.

In short: too much complexity for something that could be simple. I want to code in the language I love, not manage infra.

---

💡 The Solution: HydrAIDE

HydrAIDE is an Adaptive Intelligent Data Engine, written in Go over two years, with 3.5+ years of battle testing. One engine that:

• Stores data like a database
• Caches hot data automatically
• Sends events on every change (no separate broker)
• Handles per-record expiry with an expireAt field
• Loads into memory only when needed

It’s strongly typed, stores in binary, and is reactive out of the box.

┌───────────┐ + ┌────────┐ + ┌──────────────┐ ❌
│ Database  │   │ Cache  │   │ Pub/Sub Bus  │
└───────────┘   └────────┘   └──────────────┘
       ⬇
┌───────────────────────────────┐
│         HydrAIDE Engine       │ ✅
└───────────────────────────────┘

---

🛠️ How It Works

Intent-first, struct-first. First, I think about what data I need, where, and how it will behave in the app (lifetime, access pattern). That intent is expressed in names (Sanctuary / Realm / Swamp) and Go structs.
The beauty: you just write structs — the SDK handles saving, loading, events, and expiry.

In Practice:

• You write structs — they’re your data models.
• You define a naming function (e.g., users/profiles/<id> or chat/room/<roomID>). These namespaces are like mini-databases: blazing fast, infinitely scalable.
• You call SDK methods: ProfileSave/Read, CatalogSave/Read, Subscribe, CatalogShiftExpired, etc.

---

🧩 Profile Example — "One Entity = One Swamp"

// users/profiles/<UserID>
type UserProfile struct {
    UserID       string
    Email        string
    Username     string
    RegisteredAt time.Time `hydraide:"omitempty"`
}

func (p *UserProfile) Name() name.Name {
    return name.New().Sanctuary("users").Realm("profiles").Swamp(p.UserID)
}

func (p *UserProfile) Save(h hydraidego.Hydraidego) error { return h.ProfileSave(ctx(), p.Name(), p) }
func (p *UserProfile) Load(h hydraidego.Hydraidego) error { return h.ProfileRead(ctx(), p.Name(), p) }

// usage
in := &UserProfile{UserID: "user-123", Email: "[e-mail]", Username: "peter"}
_ = in.Save(h)

out := &UserProfile{UserID: "user-123"}
_ = out.Load(h)

Here, the entire profile is saved/loaded in a single call; no schema, no migrations.

---

📚 Catalog Example — "Many Key-Values in One Swamp"

// chat/room/<RoomID>
type ChatMessage struct {
    MessageID string    `hydraide:"key"`
    Text      string    `hydraide:"value"`
    ExpireAt  time.Time `hydraide:"expireAt"`
}

func room(roomID string) name.Name {
    return name.New().Sanctuary("chat").Realm("room").Swamp(roomID)
}

// save (upsert) with TTL
msg := &ChatMessage{MessageID: "m1", Text: "hello", ExpireAt: time.Now().Add(1*time.Minute).UTC()}
_, _ = h.CatalogSave(ctx(), room("42"), msg)

// read by key
var got ChatMessage
_ = h.CatalogRead(ctx(), room("42"), "m1", &got)

Catalogs are great for lists/queues/logs; records can have expireAt, and every write emits an event.

---

⏳ Why expireAt is a Game-Changer

In the catalog example, the expireAt field isn’t just about TTL. It lets you control processes at the database level.

We use it to orchestrate a web crawler network: a domain can only be recrawled if its expireAt has passed. Similarly, it can drive microservices without extra business logic.

expireAt works per record:

• Set it on save/update
• Fetch expired records with CatalogShiftExpired
• Build scheduled workflows directly from data state

---

⚡ Built-in Pub/Sub

Every write/update/delete triggers a real-time event for subscribers.

Example subscription:

err := h.Subscribe(ctx(), name.New().Sanctuary("chat").Realm("room").Swamp("42"), func(e hydraidego.Event) {
    log.Printf("Event: %+v", e)
})

No polling, no extra broker, instant reaction.

---

📊 Why It Matters

HydrAIDE lets us focus on coding, not infrastructure. Anyone who knows Go can pick it up in a day. Once your mindset shifts, you’ll wonder: Why didn’t I always do it this way?

• Less infrastructure
• Type-safe
• Reactive by default
• Horizontally scalable
• Open Source

---

🔗 Try It

Join the growing HydrAIDERS community and supercharge your development!

GitHub: https://github.com/hydraide/hydraide
Discord: https://discord.gg/xE2YSkzFRm

If you’re juggling too many data tools, HydrAIDE can help you throw them out and sleep better. :D

---

Péter Gebri
Creator of HydrAIDE