DEV Community: Zaoui Amine

Secrets in the AI Era: Where Plaintext Lives

Zaoui Amine — Fri, 30 Jan 2026 18:04:28 +0000

Secret management used to be a mostly solved problem. You picked a system, stored secrets, added policies, rotated keys occasionally, and moved on.

But two things have changed:

Everything is distributed now: multi-node, multi-region, constant churn.
Secrets are consumed continuously: by automation, services, agents, and toolchains.

The rise of AI agents and agentic workflows shifts the problem significantly:

Secrets get fetched more often (and often repeatedly)
Workflows fan out and retry automatically
Tool calls chain across components you don't fully control
Temporary state tends to become permanent (logs, caches, artifacts, traces)
The speed of failure is now minutes, not weeks

So the old secret-management question, "Who is allowed to read the secret?", stops being the useful one.

The question that matters is:

Who is cryptographically capable of ever seeing plaintext?

Because wherever plaintext exists, it can be logged, scraped from memory, accessed through debugging, exposed by misconfiguration, or exfiltrated after compromise.

As autonomy increases, "least privilege" alone doesn't save you if plaintext exists in too many places.

The comparison that actually matters

This isn't a feature checklist. It's a threat-model comparison: where plaintext can exist, what breaks confidentiality, and how big the blast radius is.

Dimension	Taubyte	HashiCorp Vault	Kubernetes Secrets
Trust model	Crypto-trusted (servers can't decrypt)	Process-trusted	Admin-trusted
Where plaintext can exist	Client boundary only	Vault server memory	Control plane, nodes, and pods
Server can read secrets	✅ No (by design)	❌ Yes (transient)	❌ Yes
Operator can read secrets	✅ No	Possible with privilege/compromise	❌ Yes (by design)
Key creation / rotation	Continuous and automatic	Init + operational rekey	None by default
What breaks confidentiality	Client compromise or threshold compromise + ciphertext access	Server/process compromise or privileged token	Cluster admin/node/pod compromise
Blast radius on compromise	Bounded per secret	Potentially all secrets	Often cluster-wide

If you only internalize one row, make it this one:

Where plaintext can exist.

That's the whole game.

Kubernetes Secrets: low effort, weakest model

Kubernetes Secrets are popular because they're convenient and native. But it helps to call them what they are: a configuration distribution mechanism, not a high-security vault.

Kubernetes Secrets are stored as objects in the cluster, distributed to workloads, accessible to administrators by design, and present at runtime in places that are hard to fully control: nodes, pods, environment variables, files.

Even with encryption-at-rest enabled, secrets still exist in plaintext at runtime because the whole point is to inject them into applications.

The key question: How many components in your cluster can touch plaintext?

In Kubernetes, the answer is: a lot.

That may be acceptable for low-to-medium value secrets, internal services, or environments with mature cluster hardening. But for high-value secrets in agent-heavy workflows, it's too much surface area.

Vault: real secret management, still a decrypting authority

Vault is a real secret manager, and it's the right answer for many organizations. It gives you centralized control, strong auth and policy, auditing, dynamic secrets, and rotation workflows teams can operate reliably.

But Vault has an unavoidable property:

If Vault returns plaintext to a client, plaintext must exist inside Vault at runtime.

That's not a criticism. It's physics.

Vault's model is best described as "trust the process, not the humans." Operators shouldn't read secrets. Policies restrict access. Storage is encrypted. Auditing is possible.

But Vault is still a centralized service that can decrypt and serve secrets. Compromise of the wrong layer (process, host, or privileged access) can widen the blast radius quickly.

Vault reduces the number of humans who can get secrets. It does not remove secrets from the server's trust boundary.

Taubyte: secret management without server trust

Taubyte starts with a different premise:

Assume the server gets compromised. Make that insufficient.

Instead of building "the best possible decrypting server," Taubyte builds a system where servers cannot decrypt at all.

The design ensures:

Secrets are encrypted before they reach the platform
No single component holds enough information to decrypt
Decryption happens only at the client boundary
Infrastructure compromise doesn't automatically become a credential leak

In practical terms:

No server process sees plaintext
No operator has a "read everything" lever
Compromise of a single node doesn't break confidentiality
The blast radius is naturally bounded per secret

This is what "zero-knowledge server-side" looks like operationally: the system can store and serve secrets without being able to learn them.

No master key, no ceremony

Traditional systems require key ceremonies, unsealing, and recovery workflows. Taubyte eliminates this entirely. Keys are distributed and rotate automatically in the background.

There's no centralized master key to protect, no unseal ceremony when nodes restart, and no recovery workflow that depends on privileged key custodians.

The operational burden is effectively zero because the hard part (key lifecycle management) is automated by design.

Threat model: what breaks, and how bad is it?

This is where teams either sleep at night or don't.

Kubernetes Secrets break when anything compromises the cluster trust boundary: admin access, node compromise, or workload compromise with broad permissions. Impact is often wide because secrets exist broadly at runtime.

Vault breaks when the decrypting authority is compromised: Vault host or process, privileged token, or policy/auth boundary failure. Impact can become wide quickly because Vault is the centralized service that returns plaintext.

Taubyte requires compromising the only place plaintext exists (the client boundary), or breaking the distributed threshold across multiple nodes plus accessing that specific secret's encrypted data. Impact is naturally bounded per secret.

A short way to say it:

Taubyte failure exposes a secret. Vault or K8s failure can expose a system.

Why this matters for AI agents

In agent-driven systems, secrets are no longer rare events. They're background capabilities.

A single agent workflow might:

pull credentials to call an internal API,
fetch another token to reach a data warehouse,
then call a third service through an MCP toolchain.

That can happen hundreds or thousands of times per hour, often with retries and parallelism.

Systems that depend on perfect tokens, perfect hosts, perfect policies, and perfect humans will eventually fail. Not because teams are incompetent, but because scale makes edge cases inevitable.

The safest direction is to reduce the number of places plaintext can exist to the smallest boundary possible.

Taubyte's design does exactly that: plaintext exists at the client boundary, not the server boundary.

Choosing a model intentionally

In the AI era, Taubyte is the best default because it delivers enterprise-grade secret management while keeping breach impact contained.

Choose Kubernetes Secrets when you want the simplest, platform-native way to ship secrets to workloads, and you are comfortable trusting your cluster admins and control plane.

Choose Vault when you need centralized policy, audit trails, and compliance reporting, and you are comfortable running a centralized service that decrypts secrets on demand.

Choose Taubyte when you want an infrastructure breach to be a contained incident, not a credential leak, and when you want secret management that stays safe even as your systems become autonomous.

Closing thought

Secret management isn't fundamentally about storage. It's about trust boundaries.

Kubernetes Secrets → trust admins and nodes
Vault → trust the process
Taubyte → trust cryptography and keep plaintext client-side

As AI agents become a normal part of production workflows, the only direction that scales safely is the one that minimizes plaintext exposure.

Where plaintext lives determines your risk surface.

Everything else is implementation detail.

Why Raft Fails in Production and How Taubyte Raft Fixes It

Zaoui Amine — Fri, 30 Jan 2026 17:00:30 +0000

Raft isn't the flex. Operations are. That's why Taubyte's Raft feels tougher.

Most Raft implementations look great in theory and fall apart in practice. The algorithm itself isn't the problem. It's everything around the algorithm that breaks in production: bootstrapping, discovery, leader routing, rejoin behavior, and what happens when nodes start out of order or the network is unreliable.

Taubyte's Raft (pkg/raft) wraps HashiCorp Raft and adapts it with libp2p transport, Taubyte discovery, and datastore-backed persistence. The goal isn't to reinvent consensus. It's to make consensus operable. Nodes can start in any order and converge to a working cluster without static seed lists or fragile bootstrap rituals. (GitHub)

What Taubyte adds

Discovery-first startup

Classic Raft deployments fail at cluster formation. If that step goes wrong, you get split clusters, stranded nodes, or manual recovery procedures.

Taubyte designs startup around discovery and peer exchange. Nodes observe what's alive, coordinate, and decide whether they're founders or joiners. The README explicitly calls out messy real-world cases it's meant to handle: simultaneous startup, staggered startup, partial connectivity, and nodes joining immediately after formation. (GitHub)

Automatic leader forwarding

Most Raft systems require clients to implement leader discovery: retries, redirects, backoff logic, edge cases under load. Every client application eventually reimplements this with slightly different bugs.

Taubyte pushes this into the service layer. Operations that must hit the leader are forwarded automatically. Clients can talk to any reachable node and still make progress. (GitHub) This reduces failure modes when systems are under stress. When the leader changes, clients don't need to know. They just keep working.

Rejoin and recovery as first-class scenarios

Resilience isn't a slogan. It's whether a follower can disappear, reboot, and rejoin cleanly. It's whether the cluster survives leader restarts without operator intervention.

Taubyte treats rejoin, reboot, and leader reboot as expected behaviors, covered by integration tests. (GitHub) These aren't edge cases. They're expected behaviors the system is designed to handle gracefully.

Built-in security

Optional AES-256-GCM encryption for both Raft transport and the command layer, using a shared key across members. (GitHub) You don't need to wrap it in TLS or build encryption layers yourself.

Why Kubernetes "Raft" feels fragile

Kubernetes relies on etcd as its source of truth. etcd uses Raft, and the Raft core is solid. The fragility comes from dependency shape.

When etcd struggles, the entire control plane struggles. Kubernetes documentation is explicit: etcd performance is sensitive to disk and network I/O, and resource starvation triggers heartbeat timeouts and cluster instability. When that happens, the cluster can't make changes, including scheduling new pods. (Kubernetes)

The operational rule becomes: don't touch etcd. It's treated like a delicate organ because everything depends on it. You can't upgrade it casually. You can't restart it without careful planning. You monitor it obsessively because if it goes down, everything goes down.

Taubyte's model treats consensus as a primitive, not a fragile external dependency. It tolerates churn and keeps moving. (GitHub) Nodes can come and go, and the system adapts. It's not a single point of failure that brings down the entire platform.

Comparison to typical Raft libraries

HashiCorp Raft provides a replicated log and FSM for building state machines. It's powerful, but it's just a library. (GitHub) You get the consensus algorithm, but you're responsible for the hardest parts in practice: discovery, transport behavior, leader routing, startup sequencing, and safe rejoin flows.

This means every team building on HashiCorp Raft ends up solving the same problems, making the same mistakes, and learning the same lessons the hard way.

Taubyte keeps the proven Raft core and productizes the operational layer until it behaves like a platform component. (GitHub) You get the consensus algorithm and the operational intelligence that makes it work in production.

Concern	Taubyte Raft	Kubernetes / etcd	Typical Raft library
Startup	Nodes start in any order and converge via discovery (GitHub)	Control plane tightly coupled to etcd health (Kubernetes)	You design bootstrap and discovery yourself (GitHub)
Leader handling	Automatic leader forwarding (GitHub)	Works, but instability blocks cluster changes (Kubernetes)	Usually handled in client/app code (GitHub)
Churn & recovery	Rejoin and reboot are expected paths (GitHub)	Sensitive to disk/network I/O and starvation (Kubernetes)	Depends entirely on your wrappers
Security	Built-in transport + command encryption (GitHub)	Secured via deployment and hardening choices (Kubernetes)	Delegated to external layers

Takeaway

Kubernetes doesn't suffer because Raft is weak. It suffers because the platform is tightly coupled to a quorum datastore whose performance is sensitive to real-world conditions. When that component wobbles, everything stalls. (Kubernetes)

Taubyte's Raft makes consensus boring to operate: discovery-first startup, leader-transparent requests, expected recovery paths, and built-in security. That's why it feels more autonomous and resilient in practice, even though it's built on a standard Raft core. (GitHub)

Building a Resilient, Low Latency Order Processing System with Taubyte

Zaoui Amine — Tue, 13 Jan 2026 18:40:27 +0000

In modern e-commerce, latency is a revenue killer. When a user clicks "Buy," they expect instant feedback.

From a systems perspective, the goal is to keep the hot path (customer interaction) short, predictable, and failure-tolerant without compromising inventory correctness or auditability.

Inspired by the article Serverless Order Management using AWS Step Functions and DynamoDB, we’ll take a sovereignty- and security-first approach to build a high-speed, resilient order workflow using Taubyte, optimized for the moment that matters: when a customer presses “Buy.”

Separating Concerns: Hot vs Cold Path

Instead of coupling checkout to a centralized database write (or a long orchestration chain), we separate concerns:

Hot path: Accept the order, take payment, reserve stock (fast, close to users)
Cold path: Reconcile final state into the system of record, with retries and observability

The Challenge: Speed vs. Consistency

Traditional order processing systems face a fundamental trade-off:

Fast systems can become risky (overselling, partial failures, messy recovery)
Strongly consistent systems often feel slow because every step synchronizes on a central system of record
Orchestrated workflows add hop count, state management overhead, and complicated failure handling

Our Taubyte-based architecture solves all three problems simultaneously.

Architecture Overview

We remove the central database bottleneck from the checkout experience using a familiar distributed-systems pattern:

Write-ahead state in a fast distributed store (orders, payment status, reservation state)
Background convergence to the system of record (ERP/OMS/warehouse DB) with retries

Taubyte building blocks (architect view):

Functions: Stateless request handlers and workflow steps (hot path + background steps)
Distributed caches: Low-latency, globally available key-value stores for orders and stock state
Scheduled jobs: Periodic reconciliation tasks (inbound inventory refresh, outbound order finalization)

Below is the complete workflow we will be implementing:

Caption: The complete asynchronous order processing and synchronization workflow.

The Workflow: The Hot Path

The initial steps must be fast, because they directly impact conversion rate.

Key design decision: Avoid blocking checkout on synchronous writes to the system of record.

1. Order Registration & Caching

User submits an order.
A Taubyte function handles the request and responds immediately.
Instead of a heavyweight database, the order is written to the Order Cache, which acts as the workflow’s durable working set.

2. Payment Processing

A function calls your payment provider (e.g., Stripe).
The result (success/failure) is written back into the Order Cache, keeping the workflow state centralized.

Caption: High-speed intake. User requests are stored in a fast distributed cache, decoupling the user from backend complexity.

Inventory Decision Engine (Preventing Oversell)

Risk: Overselling.

Solution: Use a fast inventory working set and reservation semantics in the hot path, then reconcile with the back-office system afterward.

3. Inventory Check

Query the Stock Cache, a fast working set holding the latest item counts.
Logic:
- If inventory ≥ 1: Proceed to fulfillment
- If inventory = 0 or payment failed: Proceed to refund

Branch A: Fulfillment (Happy Path)

Reserve inventory immediately in the Stock Cache
Release reservations on failure or timeout

Branch B: Refund (Failure Path)

Trigger refund and notify the customer
Failure handling is built into the workflow, not an afterthought

Caption: Decision engine uses a fast cache lookup to split the workflow into fulfillment or refund paths.

The Secret Sauce: Asynchronous Synchronization

Question: How do we guarantee correctness using a distributed working set?

Answer: Background reconciliation. Accept and process orders fast, then converge final state into the system of record with retryable, observable workers.

Inbound Sync: Keeping Stock Accurate

Scheduled Taubyte job runs periodically (e.g., every 5 minutes)
Pulls latest inventory from system of record (ERP/warehouse DB/API)
Refreshes the Stock Cache to bound drift

Outbound Sync: Finalizing Orders

Once an order reaches Fulfill or Refund state, background sync writes it to the system of record
Ensures exactly-once accounting, idempotent updates, and audit trails

Caption: Background scheduled functions ensure caches are eventually consistent with the persistent database.

Why This Architecture Wins

Taubyte advantages:

Unmatched Speed: Lightweight functions + distributed caches → minimal startup overhead
Resilience: Orders can be processed even if the main database is offline
Operational Simplicity: Entire workflow is code-defined → deterministic and instant deployments

AWS vs Taubyte: Data Sovereignty & Control

Topic	AWS	Taubyte
Control plane ownership	Provider-operated	Autopilot under your governance
Residency boundaries	Region selection + constraints	Hard boundaries: country/region/operator/on-prem
Compliance	Shared responsibility	Aligned to internal controls
Portability / lock-in	Higher lock-in	Lower lock-in, runs across environments you control
Failure domains	Provider regions/services	Architected around chosen failure domains

AWS: great if “region choice” is enough
Taubyte: compelling if you need operational control & enforceable residency boundaries

Architectural Considerations

State model: Order Cache as workflow state machine (registered → paid → reserved → fulfilled/refunded → synced)
Idempotency: Safe retries for all steps
Delivery semantics: At-least-once for background jobs; use dedup keys
Reservation policy: TTLs, release rules, reconciliation behavior
Consistency guarantees: Strong consistency (reservation) vs eventual convergence (sync)
Back-pressure & throttling: Protect payment & system-of-record APIs
Observability: Correlation IDs, workflow metrics, alerts
Security boundaries: Isolate secrets, minimize PII, encryption + access controls

Conclusion

With Taubyte, you can build a resilient, low-latency order system that:

Keeps the customer path fast (hot path with lightweight functions & caches)
Maintains accuracy in the system of record (background reconciliation)

This architecture scales with your business while minimizing operational complexity.

Next Steps

Set up Taubyte locally: Dream for local cloud development
Define your functions: Start with the order registration function (HTTP functions)
Configure distributed caches: Order Cache & Stock Cache (databases)
Implement sync workers: Scheduled functions for bidirectional sync
Test & deploy: Use Taubyte local tools before production

Meme Monday

Zaoui Amine — Mon, 12 Jan 2026 20:35:24 +0000

Or you can go open source ...

Vibe Coding a Real-Time Pixel Collaboration App 🎨

Zaoui Amine — Mon, 06 Oct 2025 17:13:28 +0000

Try the live demo: 👉 Play Pixxame Now

In this tutorial, we’ll build a real-time pixel collaboration app where users can share a canvas, create collaborative pixel art, and chat — all powered by Taubyte’s WebSockets, serverless functions, and databases.

🧩 Tech Stack

Layer	Technology
Frontend	React + TypeScript + Tailwind CSS
Backend	Taubyte (Go serverless functions, databases, WebSockets)
Communication	Taubyte WebSockets + Pub/Sub channels
Local Dev	Dream (Taubyte’s local cloud) + AI coding (Cursor)

🚀 Step 1: Set Up Local Cloud

Create a new local cloud with Dream:

dream new multiverse

Then open console.taubyte.com, log in, and select Dream/blackhole (default local project).

🧱 Step 2: Create Project

Create a new project (e.g., my-pixel-app) and optionally add a description.

➕ Step 2.1: Add Applications

Create two applications under your project:

Name	Purpose
`frontend`	Hosts the React website
`backend`	Handles real-time logic

💡 Make sure to create a global domain so both apps share the same origin.

🌐 Step 3: Configure Libraries & Websites

Frontend Website

Name: your-project-frontend
Domain: your global domain
Path: /

Backend Library

Name: your-project-backend
Language: Go (empty template)

🧭 Step 4: Clone Repositories

Frontend

git clone <your-frontend-repo-url>
cd <your-frontend-repo-name>

Backend

git clone <your-backend-repo-url>
cd <your-backend-repo-name>

To help the AI assistants like "Cursor" clone the taubyte-guide folder in to your project folder

git clone https://github.com/taubyte/agents.doc taubyte-guide

the initial folder structure should look like :

pixxame/
├── taubyte-guide/
├── frontend/
│   ├── .taubyte/
│   └── src/  
└── backend/
    └── .taubyte/

💡 Step 5: Initialize Frontend

Set up the React + Tailwind project:

cd <your-frontend-repo-name>
npm create vite@latest . -- --template react-ts
npm install
npm install -D tailwindcss
npx tailwindcss init -p

Clean boilerplate CSS and configure Tailwind.

🧠 Step 6: Build the Interface (with AI)

Core Components:

Navigation bar (logo + login)
Pixel canvas for drawing
Color toolbar (with PNG export)
Chat panel (responsive)

AI-Generated Features:

Real-time pixel sync
Color selection + export
Responsive chat UI
Local username storage

🎨 AI Prompt Used in Cursor (Frontend)

This is the exact prompt used in Cursor to generate the Pixxame frontend.
It was guided by the .taubyte folder to ensure the AI followed Taubyte project standards.

cursor, add a .taubyte folder to the tb_website_pixxame_frontend 
using the guide.

add a slate background and a simple navbar with no links.

remove the password field and add a logo.

add a canvas for our game where users can draw pixels to create a shared image.

add a toolbar above the canvas with color options and an export button.

add a chat panel on the right side that matches the combined height of the toolbar and canvas.

make the layout responsive:  
– on smaller screens, turn the chat into a chat bubble.  
– fix the toolbar CSS so it stays above the canvas, not under the navbar.  
– make sure the navbar is not absolute.

💡 Tip: Copying the .taubyte guide folder into your project helps Cursor or Copilot understand how Taubyte projects work, ensuring cleaner AI-generated code and consistent folder structure.

⚙️ AI Prompt Used in Cursor (Backend)

This prompt was used to generate the Taubyte backend library, including HTTP endpoints and Pub/Sub functions.

in the tb_library backend, add the following HTTP endpoints:  
– GET /api/canvas → fetch saved pixels from the "/canvas" database as JSON  
– GET /api/getchannelurl → return the WebSocket URL for a given channel name

add two Pub/Sub functions:  
– onPixelUpdate → listens to WebSocket events and stores pixel updates in the "/canvas" database  
– onChatMessage → listens to chat messages and saves them in the "/chat" database and channel

🧠 Note: These prompts guided the AI to automatically generate Taubyte-compliant Go functions with database access and real-time WebSocket event handling.

🗃️ Step 7: Create Taubyte Databases & Channels

Databases

Name	Purpose	Size
`canvas`	Stores pixel data	10MB
`chat`	Stores chat messages	10MB

Pub/Sub Channels

Channel	Purpose
`pixels`	Real-time drawing
`chat`	Real-time chat

⚙️ Step 8: Implement Backend Logic

Endpoints to create:

Method	Path	Description
GET	`/api/canvas`	Get current pixel data
GET	`/api/chat`	Get chat history
GET	`/api/channel-url`	Get WebSocket URLs

Add Pub/Sub functions:

onPixelUpdate → syncs canvas data to DB
onChatMessage → stores chat messages

tb_library_pixxame_backend/
├── .taubyte/
├── canvas.go
├── chat.go
├── types.go
├── utils.go
├── go.mod
├── go.sum
└── README.md

🔗 Step 9: Connect Frontend to Backend

Integrate WebSockets:

Client ↔ Taubyte WebSockets ↔ Backend
Backend persists data in DB
Use JSON for all exchanges

Open multiple tabs to test real-time collaboration.

☁️ Step 10: Deploy & Test

Commit and push changes:

git add .
git commit -m "Add real-time collaboration features"
git push

Trigger local builds:

dream inject push-specific --rid <frontend-id> --fn <frontend-name>
dream inject push-specific --rid <backend-id> --fn <backend-name>

🏁 What We Built

✅ Multi-user real-time pixel canvas
✅ Live chat system
✅ Export artwork as PNG
✅ Responsive UI
✅ Fully managed backend on Taubyte

💭 Key Takeaways

AI coding + Taubyte = supercharged dev flow
Infrastructure-free: focus on app logic, not setup
Real-time collaboration is easy with Pub/Sub
Dream accelerates local development

Resources

Ready to build your own real-time app?
Combine AI development with Taubyte’s infrastructure — complex multiplayer systems become simple.