hey everyone 👋
so basically i was super annoyed with a classic problem:
i have a decent GPU at home running Ollama & vLLM, but whenever i wanted to use it from my laptop at a cafe or connect it to my side-projects hosted on Render/Vercel, things got messy real fast.
either you mess around w/ router port forwarding (which is annoying & unsafe), pay for ngrok/cloudflare tunnels that randomly drop when sessions timeout, or rent overpriced cloud GPUs on AWS/RunPod just for basic testing.
so over the past few weeks i built Viento — an ultra-lightweight distributed AI inference mesh SDK in Python.
github: https://github.com/abhinav00anand/viento
docs: https://viento.readthedocs.io/en/latest/
pypi: pip install viento
What it actually does (tl;dr)
instead of exposing your home IP or opening incoming ports, Viento runs as a tiny background node daemon on your machine.
it opens an outbound WebSocket tunnel (wss://) to a lightweight cloud gateway mesh. Whenever your app makes a standard OpenAI-format API call (https://viento.onrender.com/v1/chat/completions), the gateway routes the job down through teh websocket to your local engine, streams tokens back via SSE, and handles all backpressure / queueing automatically.
Basically turns any random laptop, homelab, or cloud spot GPU into a private serverless LLM endpoint.
Top comments (0)