DEV Community

Cover image for Your RAG Pipeline's Memory Will Disappear. Here's What We Built Instead.
Divine Evna Olong
Divine Evna Olong

Posted on

Your RAG Pipeline's Memory Will Disappear. Here's What We Built Instead.

If you've built a RAG pipeline, you've already rented your AI's memory from someone else.
Pinecone goes down — your embeddings are gone. You exceed your AWS budget — your vectors get deleted. The startup you're using pivots — your data disappears overnight. You have no cryptographic guarantee that anything you stored yesterday is still there today.
That's not a product complaint. That's a structural problem with how centralized vector storage works. There's no proof. There's no enforcement. There's just trust and an SLA that doesn't actually protect you.
We built Engram on Bittensor to fix this at the infrastructure layer.

What Engram actually does
Engram is a decentralized vector database on Bittensor subnet 450 (testnet). You send text or a pre-computed vector. Engram hashes it into a permanent content-addressed CID using SHA-256 — same input always produces the same identifier, forever. That embedding gets stored across competing miners on the Bittensor network, each of whom must cryptographically prove they still hold your data or get slashed.
The key word is prove.
When a validator challenges a miner, the miner doesn't just say "yes I have it." They compute an HMAC-SHA256 response that binds the nonce the validator sent, the index position of the embedding in the batch, and a hash of the embedding itself. If they can't produce a valid response, their score drops and their TAO emissions drop with it. The financial incentive to keep your data alive is baked into the protocol.
Here's the actual challenge-response in simplified form:

import engram_core

# Validator side — issue a challenge
challenge = engram_core.generate_challenge("v1::a3f2b1c4...", timeout_secs=30)

# Miner side — respond with proof
response = engram_core.generate_response(challenge, my_stored_embedding)

# Validator side — verify constant-time
valid = engram_core.verify_response(challenge, response, my_stored_embedding)
Enter fullscreen mode Exit fullscreen mode

The core is written in Rust via PyO3. Proof generation and verification run 10-50x faster than a pure Python equivalent. Constant-time comparison throughout — no timing oracle vulnerabilities.
Batch challenges are also supported — one nonce, N CIDs, one round trip. Each proof in the batch is index-bound, meaning a miner cannot shuffle valid proofs between positions to fake holding data they don't have. We wrote a specific test for this attack vector.

How miners score TAO
Miners on subnet 450 earn TAO emissions based on:

score = 0.50 × recall@K
+ 0.30 × latency_score (1.0 at ≤100ms, 0.0 at ≥500ms)
+ 0.20 × proof_success_rate

Validators score every miner every 120 seconds. Miners with a proof success rate below 50% receive weight zero — meaning zero emissions. The incentive structure explicitly rewards persistence, not just retrieval accuracy. This is the thing VectorStore SN14 doesn't have. They reward cosine similarity. We reward miners for still holding your data 30 days from now.

Try it right now
Subnet 450 is live on Bittensor testnet. You can query an actual running miner today:

pip install engram-subnet
Enter fullscreen mode Exit fullscreen mode
from engram.sdk import EngramClient

client = EngramClient(miner_url="http://72.62.2.34:8091")

# Store a memory
cid = client.ingest("The transformer architecture changed everything.")
print(cid)  # v1::a3f2b1c4...

# Semantic search
results = client.query("how does attention work?", top_k=5)
for r in results:
    print(r['score'], r['cid'])
Enter fullscreen mode Exit fullscreen mode

LangChain and LlamaIndex adapters are available if you want to swap Engram in as a drop-in for your existing vector store:

from engram.sdk.langchain import EngramVectorStore

store = EngramVectorStore(miner_url="http://72.62.2.34:8091", embeddings=your_embeddings)
retriever = store.as_retriever(search_kwargs={"k": 5})

Enter fullscreen mode Exit fullscreen mode

Run a miner and earn TAO
If you want to participate as a miner on testnet:

git clone https://github.com/Dipraise1/Engram.git
cd Engram
pip install -e .

# Configure your wallet
cp .env.miner.example .env.miner
# Set WALLET_NAME, WALLET_HOTKEY, EXTERNAL_IP

# Register and run
btcli subnet register --netuid 450 --subtensor.network test
python neurons/miner.py --netuid 450 --subtensor.network test
Enter fullscreen mode Exit fullscreen mode

Miners need 4GB RAM minimum and 100GB SSD. No stake required on testnet. Mainnet launches Q3 2026 — early miners who establish performance history on testnet will have a head start when emissions go live.

**
What's coming**

Testnet is running. Mainnet is Q3 2026. Between now and then: erasure coding for replication, anti-spam staking, Prometheus metrics, and Arweave archival as a permanent storage backstop.
The repo is open source and public. If you want to look at the Rust proof implementation, the scoring logic, or the SDK — it's all there.
GitHub: github.com/Dipraise1/-Engram-
Docs + Playground: theengram.space
Discord: discord.gg/ehpvsyTyJ
If you're building RAG pipelines, AI agents, or anything that needs persistent semantic memory and you're tired of trusting a centralized vector store with data you can't afford to lose — this is what we built for you.

Top comments (0)