DEV Community: Sergio Pino

Calling GitHub Copilot Models from OpenHands using LiteLLM Proxy

Sergio Pino — Thu, 03 Apr 2025 19:16:06 +0000

Integrating GitHub Copilot models into your development workflow can significantly enhance productivity. By leveraging LiteLLM Proxy alongside OpenHands, you can seamlessly connect to various large language models (LLMs), including GitHub's offerings. This guide will walk you through setting up LiteLLM Proxy and OpenHands using Docker Compose, enabling you to call GitHub Copilot models effectively.openhands backends, litellm-proxy

Prerequisites

Before proceeding, ensure you have the following:

Docker and Docker Compose: Installed on your system.
GitHub Personal Access Token (PAT): Required for authenticating with GitHub's API.authenticating to the rest api

Step 1: Obtain a GitHub Personal Access Token

To authenticate with GitHub's API, you'll need to create a Personal Access Token (PAT):

Log in to your GitHub account.
Navigate to Settings > Developer settings.
In the left sidebar, click on Personal access tokens > Tokens (classic).
Click Generate new token (classic).
Provide a descriptive note, set an expiration, and select the necessary scopes.
Click Generate token and copy the token for later use.

Note: Treat your PAT like a password. Store it securely and do not share it.

Step 2: Configure LiteLLM Proxy

LiteLLM Proxy serves as a gateway to various LLM providers, including GitHub. To configure it:

Create a Configuration File: Save the following content as litellm_config.yaml:quick_start

   model_list:
     - model_name: github-gpt-4o-mini
       litellm_params:
         model: github/gpt-4o-mini
         api_key: "os.environ/GITHUB_API_KEY"

This configuration specifies the GitHub Copilot model and retrieves the API key from the environment variable GITHUB_API_KEY. There are several more models to choose from marketplace.

Set Environment Variables: Ensure the GITHUB_API_KEY environment variable is set with your GitHub PAT.

Step 3: Set Up Docker Compose

To orchestrate the deployment of LiteLLM Proxy and OpenHands, use Docker Compose with the following configuration:

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    ports:
      - "4000:4000"
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    environment:
      - GITHUB_API_KEY=${GITHUB_API_KEY}
      - LITELLM_MASTER_KEY=somekey
      - UI_USERNAME=someuser
      - UI_PASSWORD=somepassword
    command: --config /app/config.yaml --detailed_debug

  openhands:
    image: docker.all-hands.dev/all-hands-ai/openhands:0.23
    container_name: openhands-app
    ports:
      - "3000:3000"
    environment:
      - SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.24-nikolaik
      - WORKSPACE_MOUNT_PATH=${HOME}/OH
      - LOG_ALL_EVENTS=true
    volumes:
      - ${HOME}/OH:/opt/worspace_base
      - /var/run/docker.sock:/var/run/docker.sock
      - ${HOME}/.openhands-state:/.openhands-state
    extra_hosts:
      - "host.docker.internal:host-gateway"

Explanation:

LiteLLM Service:
- Mounts the litellm_config.yaml configuration file.
- Exposes port 4000.
- Sets environment variables, including GITHUB_API_KEY.
OpenHands Service:
- Depends on LiteLLM Proxy.
- Exposes port 3000.
- Configures necessary environment variables and volume mounts.

Step 4: Deploy the Services

With the configuration in place:

Ensure your GITHUB_API_KEY environment variable is set:

   export GITHUB_API_KEY=your_personal_access_token

Start the services using Docker Compose:

   docker-compose up

This command will pull the necessary images and start both services.

Step 5: Configure OpenHands to Use LiteLLM Proxy

Once the services are running:

Access the OpenHands UI at http://localhost:3000.
Navigate to Settings.
Enable Advanced options.
Set the following configurations:
- Custom Model: litellm_proxy/github-gpt-4o-mini
- Base URL: http://litellm:4000
- API Key: somekey (matches LITELLM_MASTER_KEY)

These settings direct OpenHands to route requests through LiteLLM Proxy to access the GitHub Copilot model.

Conclusion

By following these steps, you've integrated GitHub Copilot models into your development environment using LiteLLM Proxy and OpenHands.

Building a Raspberry Pi Cluster

Sergio Pino — Tue, 01 May 2018 17:00:00 +0000

I came across a presentation by Ray Tsang et al. that showcased a Raspberry Pi cluster with Docker and Kubernetes support ¹. It is very useful to have a personal Raspberry Pi cluster to explore concepts of distributed systems and create proof of concept prototypes. This post will be updated with my experiences on how to create and maintain a Raspberry Pi cluster, so I will add content over time.

1. Parts

My cluster is based on Raspberry Pi 3 Model B. I tried to follow the list of parts presented in ¹. Below I list the parts and links to each part in amazon (USA).

Part	URL
Raspberry Pi 3 Model B Motherboard	link
Anker 60W 6-Port USB Wall Charger	link
Samsung 32GB 95MB/s (U1) MicroSD EVO Select Memory Card	link
Cat 6 Ethernet Cable 5 ft (5 PACK) Flat Internet Network Cable	link
NETGEAR N300 Wi-Fi Router with High Power 5dBi External Antennas (WNR2020v2)	link
Kingston Digital USB 3.0 Portable Card Reader for SD, microSD.	link

I 3D printed the cluster rack using the files Raspberry Cluster Frame.

2. Operating System

I selected Hypriot OS since it already has support for docker. Also, with HypriotOS 1.7 and up, it is possible to use cloud-init to automatically change some settings on first boot. The version of cloud-init that it seems to support is v0.7.9.

Since my cluster is based on Raspberry PI 3 model B which includes a Quad Core 1.2GHz Broadcom BCM2837 64bit CPU ², then I use the releases of Hypriot OS provided by the github repo DieterReuter/image-builder-rpi64 (64 bit distribution) instead of the ones provided by the repo hypriot/image-builder-rpi (32-bit distribution) ³.

2.1. Flashing the OS

First, I decided to use the command line script developed by hypriot and hosted on github hypriot/flash ⁴. The advantage of this over a more simple method such as sudo dd if=image.img of=/dev/rdisk2 bs=1m. After installing the dependencies as described in the hypriot/flash repo, I installed the command line utility in $HOME/bin.

$ # get the release
$ curl -O https://raw.githubusercontent.com/hypriot/flash/master/flash
$ # add executable permissions
$ chmod +x flash
$ # move to ~/bin
$ mv flash $HOME/bin/
$ # export $HOME/bin to the path
$ export PATH=$HOME/bin:$PATH

Second, I got the OS image from DieterReuter/image-builder-rpi64⁵:

$ wget https://github.com/DieterReuter/image-builder-rpi64/releases/download/v20180429-184538/hypriotos-rpi64-v20180429-184538.img.zip
$ wget https://github.com/DieterReuter/image-builder-rpi64/releases/download/v20180429-184538/hypriotos-rpi64-v20180429-184538.img.zip.sha256
$ # Verify the image with sha-256
$ shasum -a 256 hypriotos-rpi64-v20180429-184538.img.zip

Flashing the sd-card:

$ # Setting nodeX and with the user-data.yml
$ flash --hostname nodeX --userdata ./user-data.yml hypriotos-rpi64-v20180429-184538.img

You can find an example of the user-data.yml

2.1. Network configuration (using a router)

I decided to configure the network interface using static IPs. The idea is to modify the file for eth0 which is located at sudo vim /etc/network/interfaces.d/eth0. As an example this is the setup for node1.

allow-hotplug eth0
iface eth0 inet static
    address 192.168.2.11
    network 192.168.2.0
    netmask 255.255.255.0
    broadcast 192.168.2.255
    gateway 192.168.2.1

I have a small Netgear WNR2020 wireless router dedicated to the Pi cluster, which is connected to another router that provides internet to it. I found useful to modify the DNS routes by adding the google DNS servers. Thus, I modified the file /etc/resolv.conf to look like (you need sudo to write to it):

$ cat /etc/resolv.conf
  nameserver 8.8.8.8
  nameserver 8.8.4.4
  nameserver 192.168.2.1

2.1.1 Network configuration (using a switch and head node with NAT)

Recently, I updated the cluster by removing the router. Thus, now the configuration uses a switch to communicate internally with the cluster's nodes.
Also, I configured the head node to be a NAT. Basically, the head node connects to my wireless network using wlan0 and serves as NAT through the ethernet interface eth0.

2.1.1.1 Head node

First, tell the kernel to enabled IP forwarding by either echo 1 > /proc/sys/net/ipv4/ip_forward or by changing the line net.ipv4.ip_forward=1 in the file /etc/sysctl.conf.

Second, make the head node to act as NAT:

$ sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
$ sudo iptables -A FORWARD -i wlan0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
$ sudo iptables -A FORWARD -i eth0 -o wlan0 -j ACCEPT

We can save the iptable rules and restore them at boot by:

$ sudo iptables-save > /etc/cluster_ip_rules.fw

Third, I added the configuration for the wireless interface as:

$ cat /etc/network/interfaces.d/wlan0 
auto wlan0
iface wlan0 inet dhcp
    wpa-ssid "Your ssid"
    wpa-psk "your password"
    gateway 192.168.1.1
    post-up iptables-restore < /etc/cluster_ip_rules.fw

2.1.1.2. Slave nodes

In the other nodes, I changed the eth0 configuration by adding the head node IP as the gateway. So it looks like:

$ cat /etc/network/interfaces.d/eth0
allow-hotplug eth0
#iface eth0 inet dhcp
iface eth0 inet static
    address 192.168.2.13
    network 192.168.2.0
    netmask 255.255.255.0
    broadcast 192.168.2.255
    gateway 192.168.2.11

make sure that the file /etc/resolv.conf.head has the line nameserver 192.168.1.1 or pointing to your wifi router's IP address.

2.2. Setting up passwordless

I followed the guide by Mathias Kettner ⁶ to setup ssh login without password. Basically, this is a two part process. First, create the key in your local machine.

$ # generating a key that has not the default filename
$ ssh-keygen -t rsa
 Enter file in which to save the key (<user_path>/.ssh/id_rsa): <user_path>/.ssh/pic_id_rsa
 Enter passphrase (empty for no passphrase):
 Enter same passphrase again:
 Your identification has been saved in <user_path>/.ssh/pic_id_rsa.
 Your public key has been saved in <user_path>/.ssh/pic_id_rsa.pub.
 The key fingerprint is:
 ...

Second, setup the public key on the remote host.

$ # Create .ssh folder in remote machine
$ ssh ruser@remote 'mkdir -p ~/.ssh'
 ruser@remote's password:
$ # Send key.pub to remote machine
$ cat ~/.ssh/pic_id_rsa.pub | ssh ruser@remote 'cat >> ~/.ssh/authorized_keys'
 ruser@remote's password:
$ # Changing permissions in the remote machine of both .ssh and .ssh/authorized_keys
$ ssh ruser@remote 'chmod 700 ~/.ssh'
$ ssh ruser@remote 'chmod 640 ~/.ssh/authorized_keys'

Now, you can ssh to remote without password. I'm using a Mac laptop so I had to add the key to the ssh agent by $ ssh-add $HOME/.ssh/pic_id_rsa.

3. Docker

There are several docker images for arm64v8 in the docker registry.

4. Kubernetes

In the following section, I assume that you're logged as root. If not, then you will need to insert sudo in the commands. The end result should be the same.

First, add the encryption key for the packages

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -

Second, add repository

echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" >> /etc/apt/sources.list.d/kubernetes.list

Third, in each node install kubernetes

apt-get install -y kubelet kubeadm kubectl kubernetes-cni

4.1. Configuring the master node

$ kubeadm init --pod-network-cidr 10.244.0.0/16 --apiserver-advertise-address 192.168.2.11

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

4.2. Configuring the network with Flannel

Flannel is a simple and easy way to configure a layer 3 network fabric designed for Kubernetes.\
You just need to apply it to your cluster using kubectl:

$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

4.3. Configuring the worker nodes

We just need to join our worker nodes using the kubeadm join command and passing the token and discovery-token-ca-cert-hash that
was provided by kubeadm init.

$ kubeadm join 192.168.2.11:6443 --token=<token> --discovery-token-ca-cert-hash <sha256:...>

4.4. Accessing the cluster

To access the cluster from a remote machine with kubectl you can use kubectl proxy. In my case, the master node in the raspberry pi
cluster has two network interfaces, and I wanted to connect from a machine that is in the external network (external network is 192.168.1.0/24 and the internal network is 192.168.2.0/24). Thus, running kubectl proxy in the master node and passing parameters to use the
external ip of the master node (in my case --address=192.168.1.32) and passing a list of allowed hosts for the IPs in the external network (--accept-hosts="^192.168.1"):

$ kubectl proxy --port=8080 --address=192.168.1.32 --accept-hosts="^192.168.1"

Then, you can create or add into your desired KUBECONFIG file the following information:

apiVersion: v1
clusters:
- cluster:
    server: http://192.168.1.32:8080
  name: rpi-cluster
contexts:
- context:
    cluster: rpi-cluster
  name: rpi
current-context: rpi
kind: Config
preferences: {}