Like many others, when implementing AI systems and local LLMs in a corporate environment, I encountered the issue that GPU costing $2,000 are used at only 5% load, while the company pays for cloud inference.
Many organizations already have several workstations equipped with GPUs that remain idle most of the time. Existing GPU sharing solutions typically rely on a centralized scheduler, Kubernetes, or AI gateways, which can be excessive for small and medium-sized GPU clusters. In addition, most existing solutions lack cross-platform support, decentralized architecture, and flexible policy-based routing.
That's when the idea arose to enable any office computer to poll available computers with GPUs and, based on dozens of parameters, choose the best one for local generation. I described all this in the PBDR architecture. You can read more in the scientific work. http://doi.org/10.17513/doi.26 and then in the PBDR v2.0 version http://doi.org/10.17513/doi.27 Added support for cloud models, and the new version began evaluating the speed, privacy, and cost of a request.
The idea was also that the software should be open, as compact as possible, and ready for rapid deployment on any OS from source or binary. After removing everything unnecessary, two main kernel modules and a supporting remote administration service remained.
Simply run a server or client on each machine, and your existing office network becomes a decentralized AI cluster. Routing decisions are made entirely on the client side, eliminating the need for a master node, load balancer, or any other single point of failure.
Prebuilt binaries for Windows and Linux are under 10 MB and can bring a machine online as a cluster node in under 30 seconds. An optional admin server provides centralized monitoring and management from any machine on the network.
GitHub (MIT License):
https://github.com/oqo-ai/OQOAI-PBDR
Under the hood is a lightweight and open-source core based on literally 3 source files of ~1000 lines each, with compiled versions for Windows and Linux supporting Nvidia and AMD GPUs -the entire binary takes about 10 MB. It allows, taking into account dozens of parameters.
The project intentionally avoids unnecessary abstractions. The core routing logic consists of only two source files, making it relatively easy to understand, audit, and extend.
The PBDR architecture (policy based decentralized routing) on which the project is based. Each routing decision evaluates multiple runtime metrics, GPU utilization, available VRAM, queue length, GPU temperature, network latency, inference throughput, and other node characteristics:
- GPU and CPU load,
- available VRAM,
- GPU temperature,
- request queue length,
- inference time,
- cold start of the model,
- currently loaded model,
- network latency,
- node idle time, to make intelligent routing decisions.
Now new criteria have been added to this list:
- economic efficiency — comparing the cost of local generation versus cloud costs,
- model availability in the cloud,
- data privacy - PII and medical data remain local,
- time of day - the cloud may be cheaper at night,
- model size - requests for large models are routed to the cloud.
The system automatically decides: send the request to the local GPU cluster or to one of 40+ cloud providers (OpenAI, Anthropic, Google, Azure, AWS, DeepSeek, Together AI, and others).
PBDR also supports client-group policies, allowing different workloads to optimize for different objectives. For example:
Internal RAG services can prioritize low latency.
Development workloads can prioritize output quality.
Batch inference jobs can prioritize throughput.
One practical benefit of this approach is that organizations can often reuse existing workstation GPUs for low- and medium-throughput workloads-such as internal RAG systems-instead of deploying dedicated inference infrastructure from scratch.
In many environments, having GPUs installed in only 10-15% of workstations can be sufficient to satisfy inference demand for lightweight workloads (for example, an internal knowledge-base RAG receiving approximately 3-5 requests per user per day) without deploying centralized AI infrastructure.
Getting started:
Install OQOAI-PBDR on each machine.
Connect server nodes to any OpenAI-compatible inference backend, including Ollama, llama.cpp, vLLM, or TGI.
Configure your applications to use the OpenAI-compatible client endpoint provided by OQOAI-PBDR.
Clients automatically discover available models on all nodes. From the application’s perspective, the cluster behaves like a distributed OpenAI API compatible system, while user requests are routed in a decentralized manner to the most suitable node according to the chosen policy.
Everything works through a unified OpenAI-compatible API - your applications (Open WebUI, N8N, LangChain, LlamaIndex) require no changes.
An interesting detail: to calculate the cost of local generation, the system collects the power_max parameter from the GPU (maximum power consumption) and uses it to compute electricity costs. This allows real-time comparison: what is cheaper - running generation locally or sending the request to the cloud.
What already works:
- OpenAI-compatible API (Ollama, vLLM, TGI, llama.cpp)
- Admin dashboard with real-time metrics
- Auto-discovery and network scanning
- Remote config management for nodes, groups, or the whole cluster
- Experimental binary builds for Windows and Linux
Roadmap highlights:
- ComfyUI/AUTOMATIC1111 image/video generation
- Grafana/Prometheus integration
- AI-assisted semantic routing (auto-model selection)
- Dynamic IP support
We're actively looking for feedback, and testers.
What features or integrations would you like to see next?
Top comments (0)