AI inference is getting expensive.
For developers building AI products, the cost isn't just the model itself. Every request consumes GPU compute, and as usage grows, inference can quickly become one of the largest infrastructure expenses.
We ran into this problem ourselves while building AI products.
So we started building Kielo.
Kielo is an OpenRouter-like inference layer for open-source models, powered by a decentralized network of GPUs.
The problem with AI inference
Running AI models requires expensive compute.
This becomes especially painful when you're building and iterating on AI products. You're constantly testing prompts, comparing models, running evaluations, and experimenting with different workloads.
Then your users arrive.
More users → more inference → more GPU capacity → higher costs.
Most of today's inference infrastructure ultimately relies on centralized GPU providers. This makes sense operationally, but centralized infrastructure has to account for expensive hardware, data centers, electricity, cooling, networking, maintenance, redundancy, and idle capacity.
At the same time, there are thousands of GPUs sitting underutilized around the world.
Gaming PCs.
Developer workstations.
Research machines.
Small servers.
Unused GPUs in organizations.
We started asking:
What if that idle compute could become part of an inference network?
Decentralized inference
Instead of every developer sending requests to a centralized GPU cluster, Kielo connects developers to independent compute providers.
GPU Provider A ─┐
GPU Provider B ─┤
GPU Provider C ─┼──> Kielo Network ──> Developer
GPU Provider D ─┘
Providers contribute available GPU capacity.
Kielo handles the infrastructure layer that connects that compute with inference requests.
The developer doesn't need to know which physical machine processes their request.
They simply call an API.
How Kielo works
The architecture can be simplified into four layers:
Providers → Compute Network → Routing → Developers
1. Providers
A provider installs the Kielo node on their machine.
The node detects the available hardware and benchmarks it.
For example:
GPU: RTX 4090
VRAM: 24 GB
Available models:
- Llama
- Mistral
- Qwen
The provider can then contribute available compute to the network.
The goal is simple:
Turn idle GPU capacity into useful infrastructure.
2. Compute Network
Kielo keeps track of the available compute across providers.
The network needs to know things like:
- Which providers are online
- Which models they support
- GPU capabilities
- Available VRAM
- Current utilization
- Latency
- Provider health
- Capacity
This allows the network to make intelligent decisions about where inference should run.
3. Intelligent routing
When a developer sends an inference request, Kielo determines where it should execute.
Conceptually:
Developer Request
↓
Kielo API
↓
Routing Layer
↓
Find compatible providers
↓
Select healthy/optimal GPU
↓
Run inference
↓
Return response
Eventually, routing can consider multiple factors:
Model availability
+ GPU capability
+ Current load
+ Latency
+ Reliability
+ Cost
↓
Optimal provider
This routing layer is one of the most important parts of making decentralized inference actually work.
4. Developers
Developers shouldn't have to manage individual GPUs.
They should be able to use one API and access multiple open-source models.
Kielo is designed around an OpenAI-compatible API, making it easier to integrate into existing applications.
The idea is:
One API
↓
Multiple open-source models
↓
Distributed GPU network
Why open-source models?
We're focusing on open-source and open-weight models because they provide much more flexibility over where inference happens.
Models such as Llama, Mistral, Qwen, and other open models can be deployed across different hardware configurations.
That makes them particularly interesting for decentralized infrastructure.
Instead of building infrastructure around a single model provider, we're building toward a network where developers can access a growing collection of open models through one interface.
What we've built
Kielo is still early, but we've already been working on the core infrastructure required for the network.
We've built and experimented with:
- Developer API
- OpenAI-compatible endpoints
- Provider/node software
- Hardware detection
- GPU benchmarking
- Model detection
- Provider registration
- Model execution
- Compute allocation
- Request routing
- Provider health tracking
- Usage tracking
- Distributed inference infrastructure
The provider experience is designed around a simple flow:
Install Kielo
↓
Detect hardware
↓
Benchmark machine
↓
Configure models
↓
Start node
↓
Contribute compute
The developer experience should be even simpler:
Get API key
↓
Choose model
↓
Send API request
↓
Get inference
What we're testing
There are still some very important questions we need to answer.
Can decentralized inference actually be cheaper?
It's easy to say that using idle hardware should reduce costs.
But real economics include:
- Electricity
- Hardware depreciation
- Bandwidth
- Network overhead
- Failed requests
- Idle capacity
- Reliability
- Data transfer
- Platform infrastructure
We want to measure the actual economics rather than assume them.
Can decentralized inference be reliable?
A centralized GPU cluster is relatively predictable.
A decentralized network isn't.
Providers can disconnect.
Machines can become overloaded.
Networks can fail.
Hardware can disappear.
So reliability requires health checks, intelligent routing, retries, and failover.
Will people contribute their GPUs?
This may be the biggest question.
Having an idle GPU doesn't automatically mean someone wants to share it.
The provider experience needs to be:
- Easy
- Secure
- Transparent
- Economically worthwhile
We're still learning what that looks like.
What we're looking for
We're looking for both developers and GPU providers to test the idea.
If you're building with open-source models, we'd love to understand:
- What models are you using?
- What are you currently paying for inference?
- How important are latency and reliability?
- Would significantly cheaper inference change what you build?
And if you have a GPU sitting idle, we'd like to understand:
- What hardware do you have?
- How much is it actually idle?
- What would make you comfortable contributing it?
- What compensation would make it worthwhile?
- What security concerns would you have?
These answers will directly influence the product.
The bigger idea
We're not trying to decentralize AI simply because decentralization sounds interesting.
The goal is practical:
Make open-source AI inference cheaper and more accessible by turning unused compute into usable infrastructure.
There is already a huge amount of compute sitting idle.
We're building the infrastructure to connect that compute with the developers who need it.
Kielo is our attempt at building an OpenRouter for open-source models, powered by decentralized compute.
We're early.
If you're building with open-source models, running GPUs, or working on distributed infrastructure, we'd love your feedback.
Try Kielo: https://www.kielo.in/
Top comments (1)
The hard part for an open-source model router is usually less about listing models and more about reliability semantics. Developers need to know what happens when a node disappears, how retries are bounded, and whether latency/cost tradeoffs are visible enough to make production decisions.