DEV Community: sumeet saraf

Building an AI Runtime Operating System for Commodity Hardware (UGR)

sumeet saraf — Wed, 22 Jul 2026 07:46:47 +0000

Building an AI Runtime Operating System for Commodity Hardware

For the last few months I've been working on something that started as a simple question:

Why do we still treat AI inference as "load an entire model into GPU memory and hope it fits"?

My development machine certainly doesn't make life easy.

Dell Precision 5520
NVIDIA Quadro M1200 (4 GB VRAM)
32 GB RAM
Ubuntu

Like many developers experimenting with open-source LLMs, I quickly ran into the same wall everyone eventually hits:

CUDA Out of Memory

At first I assumed the answer was simply "buy a bigger GPU."

But after digging through projects like llama.cpp, Ollama, TensorRT-LLM, Colibri, and various memory streaming experiments, I started wondering whether we were solving the wrong problem.

The Problem Isn't Just Memory

Most inference engines think about execution like this:

Application
    ↓
Inference Engine
    ↓
GPU

If the model fits, great.

If it doesn't...

You're usually done.

Modern runtimes have become incredibly efficient, but they're still primarily focused on executing tensors as quickly as possible.

What if we instead treated inference more like an operating system treats processes and memory?

From Runtime to Runtime OS

That idea eventually became UGR.

Instead of replacing inference engines like llama.cpp, UGR sits above them as an orchestration layer.

Application
    │
OpenAI-Compatible API
    │
Planner
    │
Runtime Intelligence Engine
    │
Scheduler
    │
Virtual Resource Manager
    │
Adaptive Memory Fabric
    │
Driver Abstraction Layer
    │
llama.cpp / TensorRT / Vulkan

The execution backend becomes interchangeable.

The runtime becomes responsible for deciding how execution should happen.

Core Ideas

Rather than managing a single GGUF file, UGR treats AI workloads as collections of resources.

That includes things like:

Tensors
KV cache
MoE experts
Embeddings
LoRA adapters
Execution state

These resources can move between different memory tiers depending on runtime conditions.

VRAM
  ↓
RAM
  ↓
NVMe
  ↓
Remote Storage
  ↓
Archive

Instead of asking:

Can this model fit into VRAM?

The runtime asks:

How should these resources be orchestrated across available hardware?

Hardware Shouldn't Dictate Capability

One thing I kept running into was that modern AI software often assumes modern hardware.

Reality is different.

There are countless developers, researchers, universities, and small teams running inference on older GPUs or mixed hardware environments.

Rather than optimizing for a single GPU architecture, I wanted UGR to detect hardware capabilities and adapt its execution strategy accordingly.

Simulation Before Execution

One feature I'm particularly excited about is the simulation engine.

Instead of loading a model and hoping for the best:

ugr simulate gemma-4-31b.gguf

The runtime predicts:

Expected memory usage
Estimated throughput
Bottlenecks
Scheduler choice
Whether the model is even runnable

...before execution begins.

Current Status

UGR is still very much a research project.

Currently implemented:

Multi-backend runtime
GGUF parser
Adaptive Memory Fabric
Virtual Resource Manager
OpenAI-compatible API
Hardware capability detection
CLI tools
Model registry

Still under active development:

Runtime Intelligence Engine
Simulation engine improvements
MoE expert paging integration
TensorRT backend
Vulkan backend

Why Open Source?

I decided to open source the project because I don't believe architectures improve in isolation.

I also don't have access to high-end hardware.

Most of my development has been on a machine with a 4 GB Quadro GPU, so I would love feedback from people running larger GPUs or heterogeneous systems.

If nothing else, I hope some of the ideas around resource virtualization and runtime orchestration spark interesting discussions.

GitHub

⭐ UGR (AI Runtime Operating System)

https://github.com/UGEM-io/UGR

I'd genuinely appreciate feedback, bug reports, architectural criticism, or ideas for improving the runtime.

Final thoughts

When I started this project, I thought I was trying to solve a memory problem.

Today I think I'm trying to solve a runtime orchestration problem.

Whether UGR ultimately becomes a useful research platform or something more, I'm excited to see where the community takes it.

AMTP: HTTP for the Agentic Web — A New Markdown-First Protocol for AI Agents

sumeet saraf — Tue, 26 May 2026 19:06:53 +0000

The web was built for humans. But AI agents are struggling with it.

Traditional approaches — DOM parsing, headless browsers, or forcing LLMs to read raw HTML — are brittle, expensive, and token-heavy.

Today, I'm open-sourcing AMTP (Agent Markdown Transfer Protocol) — a new protocol designed as HTTP for the Agentic Web.

The Problem

AI agents today face several major challenges when interacting with websites:

Extremely high token usage when processing full HTML pages
Brittle scraping that breaks with any UI change
No native action support — agents must simulate clicks and forms
Poor session handling for authenticated actions
Lack of real-time updates

We need a better way.

Introducing AMTP

AMTP allows websites to serve clean, semantic Markdown optimized for Large Language Models and autonomous agents, alongside their normal HTML and JSON responses.

It uses content negotiation (Accept: text/amtp) so one backend can serve all three consumers: browsers, APIs, and agents.

Key Features

~90% fewer tokens compared to HTML
Native actions, forms, and navigation (hypermedia style)
Built-in session & authentication bridging (works with your existing Passport, JWT, etc.)
Streaming support for real-time updates
Security-first design with scopes and validation
Easy integration via Express/Fastify middleware

How It Works

Here's a quick example using the AMTP middleware:

import { amtp } from '@amtp/protocol';
import express from 'express';

const app = express();

app.use(amtp({
  // Define what agents can see and do
  routes: {
    '/dashboard': {
      title: "User Dashboard",
      content: "Welcome back, {{user.name}}",
      actions: [
        {
          name: "view_orders",
          label: "View My Orders",
          href: "/orders",
          method: "GET"
        }
      ]
    }
  }
}));

From the agent side, the response is clean Markdown with embedded action metadata — perfect for LLMs to understand and act upon.

Real-World Use Cases

Agent-native SaaS platforms (Let users say “cancel my subscription” to their AI assistant)
E-commerce agents that can reliably browse and purchase
Personal automation agents that work with your existing logged-in sessions
Multi-agent workflows across different websites
Efficient crawling and indexing

Comparison

Feature	Raw HTML	JSON API	AMTP
Token Efficiency	Poor	Good	Excellent
Action Support	None	Manual	Native
Session/Auth Bridging	Hard	Medium	Built-in
LLM Readability	Poor	Medium	Excellent
Streaming	Difficult	Possible	Native

Getting Started

npm install @amtp/protocol

Visit the repository for:

Full protocol specification
Grammar and examples
Client SDK
Reference implementations

GitHub: https://github.com/sumeetingenuity/amtp

The Vision

AMTP is an early but functional step toward a truly agent-ready web — where any website can become natively compatible with AI agents without sacrificing control or security.

I’d love your feedback, contributions, and real-world use cases.

What’s the biggest pain point you face when building agents that interact with websites?

"Building a Lightweight Blockchain in Go from Scratch"

sumeet saraf — Fri, 09 May 2025 16:52:19 +0000

Hey Gophers and blockchain enthusiasts! 👋

I recently built Lite-Blockchain, a minimal yet functional blockchain implementation in Go. The goal was to demystify how blockchains work under the hood while leveraging Go’s simplicity and concurrency features. Let me walk you through what I learned!

Why Build a Blockchain in Go?
Go’s native support for concurrency (goroutines, channels) and efficient performance makes it a great fit for distributed systems like blockchains. Plus, its simplicity helps keep the codebase clean and educational.

Key Features
Proof-of-Work (PoW) Consensus: A Simple mining mechanism to secure the chain.

Transaction System: Create and validate transactions.

Pure Go: No external dependencies—just the standard library.

Lightweight: Perfect for learning.

Code Snippets
Here’s a peek at how blocks are structured:

type Block struct {  
    Timestamp    int64  
    Transactions []*Transaction  
    PrevHash     []byte  
    Hash         []byte  
    Nonce        int  
}

And how transactions are added:

func (bc *Blockchain) AddTransaction(sender, recipient string, amount int) {  
    tx := NewTransaction(sender, recipient, amount)  
    bc.PendingTransactions = append(bc.PendingTransactions, tx)  
}

Why Go for Blockchain?
Concurrency: Goroutines make parallel mining or network synchronization easier to model.

Performance: Faster than interpreted languages like Python.

Simplicity: Explicit error handling and no inheritance headaches.

Try It Out!
Clone the repo:

bash
git clone https://github.com/sumeetingenuity/Lite-Blockchain.git

Run the example:

bash
go run main.go

Contribute or Learn
This project is open-source and meant to be a learning resource! Feel free to:
⭐ Star the repo if you find it useful.
🐞 Open issues for bugs or feature requests.
🔧 Submit PRs for improvements (e.g., adding a P2P network layer).

Check out the GitHub repo to dive deeper!

I'm looking for some feedback on my project

sumeet saraf — Wed, 07 May 2025 17:43:45 +0000

I'm new to programming and not sure if I'm heading in the right direction. I'd really appreciate any feedback or constructive criticism on my project. Thanks in advance for your help!

https://github.com/sumeetingenuity/Lite-Blockchain.git