Project EREN: My Journey to Building a CUDA-Powered LLM Inference Engine

Arunav Pathak — Thu, 22 Oct 2026 15:30:55 +0000

Hey DEV Community! 👋

This is my first post here, and I'm excited to share a project I've recently started working on.

I'm a Computer Science undergraduate interested in GPU programming, high-performance computing, and the systems behind large language models.

Over the past few months, I've been learning CUDA and exploring how GPUs execute parallel workloads. Eventually, that curiosity led me to a bigger question:

What would it take to build an LLM inference engine from scratch?

Not just load a model through an existing inference framework, but actually understand and implement the components responsible for running it.

That's how Project EREN started.

What is Project EREN?

EREN is my attempt to build a CUDA-accelerated LLM inference engine from the ground up, using C++ and CUDA.

The initial development target is my NVIDIA RTX 4060 Laptop GPU with 8 GB of VRAM.

I'll start with a small decoder-only transformer such as GPT-2, establish correctness, and gradually work toward more capable models and efficient inference.

My long-term goal is to support multiple concurrent inference requests with high throughput, while understanding the underlying GPU kernels, memory management, scheduling, and execution strategies.

Eventually, I want EREN to serve as the inference backend for a RAG-based AI agent.

This isn't about competing with mature inference engines right away. It's about learning how these systems work by building one.

Why build an inference engine from scratch?

Libraries like PyTorch and established inference runtimes make running language models surprisingly accessible.

But I wanted to understand what happens underneath those abstractions.

For example:

How are transformer operations mapped onto GPU kernels?

What happens to model weights and intermediate activations in GPU memory?

How does autoregressive token generation work internally?

Why do memory bandwidth and kernel launch overhead matter during inference?

How can an engine efficiently serve multiple requests on a resource-constrained GPU?

Instead of only reading about these concepts, I want to implement, test, benchmark, and optimize them myself.

Technology stack

My current development environment includes:

Language: C++17 and CUDA C++

GPU: NVIDIA GeForce RTX 4060 Laptop GPU (8 GB)

Environment: Ubuntu on WSL2

Build system: CMake

Testing: CTest

Reference implementation (upcoming): Python, PyTorch, and Hugging Face Transformers

The core inference runtime will be implemented in C++ and CUDA. Python and PyTorch will initially help validate correctness against a trusted reference.

Progress so far: Phase 0

My first milestone was establishing a reliable C++/CUDA development foundation.

So far, I've worked on the following components.

Project structure and build system

Organized the repository into components for kernels, source code, headers, tests, benchmarks, and documentation.

Configured CMake to build the C++ and CUDA targets.

Here's a look at the current EREN project structure in VS Code:

CUDA vector addition

Implemented a basic vector addition kernel to validate device memory allocation, host-to-device transfers, kernel launches, and result verification.

GPU reduction

Implemented a shared-memory, multi-stage reduction baseline.

This was an important exercise in understanding parallel reduction and coordinating work across thread blocks.

CUDA error handling

Created a reusable CUDA error-checking helper to make runtime failures easier to identify and debug.

Testing and benchmark foundation

Added smoke tests using CTest and an initial reduction benchmark target.

A clean build succeeded, and all four configured tests passed.

Here's the actual Ubuntu terminal output from the EREN development environment, showing the build and test results:

The inference engine itself isn't operational yet. This milestone establishes the foundation for the next stage.

What's next?

I'm developing EREN in phases rather than jumping straight into optimized kernels.

Phase 1 — Python/PyTorch reference implementation

I'll begin with GPT-2, study its forward pass, inspect its weights and tensor shapes, and generate reference outputs for correctness testing.

Phase 2 — First CUDA inference implementation

Implement the initial CUDA operators and connect them into a complete GPT-2 forward pass.

Later phases — Optimization and serving

Explore optimized matrix multiplication, quantization, GPU profiling, memory management, batching, concurrent requests, and inference scheduling.

The aim is to progress from a correct implementation toward a measurable, efficient inference runtime.

Why I'm documenting this publicly

I want this series to be more than a collection of project announcements.

As I progress, I'll share:

What I'm implementing and why

CUDA concepts and engineering decisions

Bugs, failed experiments, and lessons learned

Benchmarks and performance comparisons

Design tradeoffs and improvements

I'll also try to distinguish between what works, what I've measured, and what I'm still experimenting with.

Final thoughts

EREN is still at an early stage, and there's a lot left to learn.

But that's exactly why I want to document the process from the beginning.

If you're interested in CUDA, GPU computing, LLM inference, or systems programming, I'd love to exchange ideas and learn from your experiences.

Next up: Building a GPT-2 reference implementation in PyTorch before moving into CUDA inference.

Thanks for reading!

GitHub Repository: Project EREN

DEV Community: Arunav Pathak

Project EREN: My Journey to Building a CUDA-Powered LLM Inference Engine