Introduction
Recently, while exploring the new Zed IDE, I encountered a need to integrate it with a self-hosted Ollama model. Zed allows for custom LLM integrations, which is great, but I ran into a small hiccup.
Problem Description
The challenge arose because Ollama's configuration in Zed requires a connection to localhost. However, I have Ollama running on my GPU-enabled home server, not on my local machine.
Solution Approach
To bridge this gap, I devised a simple solution: a Rust script that acts as a proxy. This script forwards the Ollama port from my server to the local machine, allowing Zed to connect seamlessly.
Implementation Details
Here's a brief walkthrough of the Rust script:
use hyper::service::{make_service_fn, service_fn};
use hyper::{Body, Client, Request, Response, Server};
use std::convert::Infallible;
use std::net::SocketAddr;
async fn proxy(req: Request<Body>) -> Result<Response<Body>, hyper::Error> {
// Create a client to send the forwarded request.
let client = Client::new();
// Build a new URI using the hostname "ace" and the appropriate port.
// Here, we assume your Ollama server is also listening on port 11434.
// Adjust the port in the format string if it's different.
let uri_string = format!("http://ace:11434{}", req.uri());
let new_uri = uri_string.parse().expect("Failed to parse URI");
// Replace the URI in the incoming request with the new one.
// This effectively "redirects" the request to your remote server.
let mut proxied_request = req;
*proxied_request.uri_mut() = new_uri;
// Forward the request to the remote server.
client.request(proxied_request).await
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// Bind to local port 11434.
let addr = SocketAddr::from(([127, 0, 0, 1], 11434));
let make_svc = make_service_fn(|_conn| async {
// For each connection, we create a service to handle the request.
Ok::<_, Infallible>(service_fn(proxy))
});
let server = Server::bind(&addr).serve(make_svc);
println!("Reverse proxy running on http://{}", addr);
// Run the server until it's stopped.
server.await?;
Ok(())
}
The script efficiently handles the port forwarding with no noticeable performance penalty, making the integration smooth and reliable.
Conclusion
This quick Rust script solved the integration challenge, enabling me to use Zed with my self-hosted Ollama model effortlessly. It's a testament to how small, targeted scripts can effectively solve specific problems without overcomplicating the setup.
To me this is a very interesting implication of the new world of AI tooling. I just used some simple prompts and had a working solution to a problem in a few minutes. I'm sure I could have taken the time to make caddy or nginx or something else work, but custom built software for a specific need is also an option nowadays.
Top comments (0)