DEV Community

fanioz
fanioz

Posted on

How to Configure a Custom LLM Proxy in OpenOPC (and Debug LiteLLM)

#ai #agents #python #opensource

(If you missed it, check out my previous post on Getting Started with OpenOPC to see why I'm using this agent framework in the first place!)

I recently started setting up OpenOPC to run my AI agents. To save on API costs, I decided to hook it up to a custom, OpenAI-compatible proxy at https://api.freetheai.xyz/v1.

I expected it to be a simple drop-in replacement. In other platforms, you just set your model to something like glm/glm-5.2 and it works perfectly. So, I updated my .opc/config/llm_config.yaml:

llm:
  default_model: "glm/glm-5.2"
  api_base: "https://api.freetheai.xyz/v1"
  api_key: "my_api_key"
Enter fullscreen mode Exit fullscreen mode

OpenOPC uses LiteLLM under the hood to handle model routing. Instead of my agents spinning up, LiteLLM crashed out with this error:

litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=glm/glm-5.2

The openai/ Prefix Trick

Because I was pointing to a custom api_base, LiteLLM didn't natively know how to route the glm/ provider prefix. It needs to know exactly which API format to use to talk to the endpoint.

For almost all custom endpoints, the format is OpenAI-compatible. To force LiteLLM to use the standard OpenAI request format, you have to prefix your model with openai/.

I updated my config:

llm:
  default_model: "openai/glm-5.2"
Enter fullscreen mode Exit fullscreen mode

The "Unknown Aliased Model" Trap

I restarted OpenOPC. LiteLLM knew how to format the request this time, but I hit a new error. This time, it was rejected by the proxy itself:

OpenAIException - unknown aliased model

When you use the openai/ prefix, LiteLLM strips it off before sending the request. That means it passed exactly glm-5.2 to the proxy. The proxy had no idea what that was.

To see what the proxy actually expected, I queried its models endpoint directly using curl:

curl -s https://api.freetheai.xyz/v1/models -H "Authorization: Bearer my_api_key"
Enter fullscreen mode Exit fullscreen mode

Buried in the massive JSON response, I found the literal ID registered on the server for that model: glm/glm-5.2.

The Final Working Config

To make this work, I needed LiteLLM to use the OpenAI API format, but I needed it to pass glm/glm-5.2 as the actual model name in the JSON payload. The fix was stacking them:

llm:
  default_model: "openai/glm/glm-5.2"
Enter fullscreen mode Exit fullscreen mode

Once I restarted OpenOPC with this stacked config, the routing worked perfectly. I did hit a ServiceUnavailableError - provider capacity temporarily unavailable immediately after, but that's just the reality of using busy proxy servers—the actual connection and routing were finally correct.

If you're hooking OpenOPC up to a custom proxy, always hit their /v1/models endpoint first to get the literal string they expect, and slap openai/ on the front.

Discussion: How are you handling custom API proxies and model routing for your agent stacks? Do you prefer configuring prefixes like this in LiteLLM, or do you rewrite the requests at a proxy level before they even hit your models? Let me know below!

Looking for a solid API proxy? You can join FreeTheAI.xyz here.

Top comments (0)