Connecting AnythingLLM to an OpenAI-Compatible Multi-Model Gateway

AnythingLLM already supports a Generic OpenAI provider, so it can connect to any service that exposes an OpenAI-style API. That makes it possible to keep AnythingLLM’s local UI, workspaces, RAG pipelines, and agent features while routing model requests through CometAPI, which provides access to 500+ models through one API surface.

The useful part is not the UI integration itself—it is the ability to change models without reworking the AnythingLLM setup.

What this setup provides

AnythingLLM is an open-source AI application for chat assistants, RAG workflows, and LLM-powered agents. It supports desktop and self-hosted deployments, exposes a developer API, and can work with both local and cloud models.

The Generic OpenAI provider is the relevant integration point. It lets AnythingLLM send requests to an OpenAI-compatible endpoint rather than requiring a provider-specific connector.

Using a gateway in that slot gives me:

• A single API key and billing path across multiple vendors
• The ability to switch models without changing AnythingLLM workflows
• Centralized cost control and model selection
• A consistent interface for chat completions and embeddings
• Easier experimentation with different models for RAG, agents, and multimodal tasks

The gateway endpoint used in this example is:

https://api.cometapi.com/v1

It exposes OpenAI-style routes such as /v1/chat/completions and /v1/embeddings.

Requirements and deployment notes

Before configuring AnythingLLM, I check the following:

• AnythingLLM is running on Windows, macOS, or Linux.
• The installed version exposes LLM Preferences or AI Providers.
• I have an API key in the sk-xxxxx format.
• The host can make outbound HTTPS requests to https://api.cometapi.com.
• For manual testing, I have curl, Python 3.10+, or Node 18+ available.
• An HTTP client such as Postman or HTTPie is optional but useful.

The Generic OpenAI provider is intended for developer-controlled endpoints, so its URL, model name, context window, and streaming behavior need to match the service being configured.

If the selected endpoint or model does not support streaming, disable streaming in AnythingLLM’s Generic OpenAI settings.

Secrets and limits

Treat the API key like any other production credential:

• Do not commit it to a repository.
• Prefer environment variables, an OS keychain, or a secret manager.
• Remember that entering it in the AnythingLLM UI stores it in the host’s local settings.
• If using private documents in RAG, verify the endpoint’s privacy and compliance requirements.
• Set sensible context-window and maximum-token limits to avoid accidental usage spikes.

Verify the endpoint before touching AnythingLLM

I prefer testing the API independently first. That separates authentication or networking problems from AnythingLLM configuration issues.

curl

curl -X POST "https://api.cometapi.com/v1/chat/completions" \
  -H "Authorization: Bearer sk-xxxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "user",
        "content": "Reply with one short sentence confirming the connection."
      }
    ],
    "max_tokens": 50
  }'

A successful request should return HTTP 200 and a JSON object containing a choices array.

Python

import requests

url = "https://api.cometapi.com/v1/chat/completions"
headers = {
    "Authorization": "Bearer sk-xxxxx",
    "Content-Type": "application/json",
}
payload = {
    "model": "gpt-4o",
    "messages": [
        {
            "role": "user",
            "content": "Reply with one short sentence confirming the connection.",
        }
    ],
    "max_tokens": 64,
}

response = requests.post(url, json=payload, headers=headers, timeout=15)
print(response.status_code, response.json())

If this fails, I fix the key, network access, or model name before investigating AnythingLLM.

Configure the Generic OpenAI provider

The exact menu labels vary slightly by AnythingLLM version, but the usual path is:

Settings → AI Providers → LLM Preferences

Select Generic OpenAI and configure it with values like these:

Setting	Value
Provider	Generic OpenAI
Base URL	https://api.cometapi.com/v1
API key	sk-xxxxx
Model	A model name published by the gateway
Token context window	The selected model’s supported context size
Max tokens	A value appropriate for the model and workload

Add the model names exactly as published. Examples include:

gpt-4o
minimax-m2
kimi-k2-thinking

Do not assume that a model name accepted by another provider is available here. A typo or unsupported model generally produces a 400 or 404 response.

After saving the provider:

1. Start a new chat or open an existing workspace.
2. Select Generic OpenAI if multiple providers are configured.
3. Choose one of the configured model names.
4. Send a simple prompt.
5. Confirm that the response is returned normally.

What AnythingLLM does with the configuration

AnythingLLM’s Generic OpenAI wrapper builds OpenAI-style requests for routes such as:

/v1/chat/completions
/v1/embeddings

Once the base URL and key are configured, chat requests, agent calls, and embedding requests use that provider transparently. AnythingLLM agent flows, including @agent workflows, inherit the same provider configuration.

That means the integration does not require changes to individual workspaces or agent code. The main operational responsibility is keeping model names and token settings aligned with the selected backend.

Operational issues worth checking

Context windows and output limits

The Token Context Window and Max Tokens values should reflect the actual model. If they are too low, responses may be truncated. If they exceed the model’s limits, requests can fail.

I usually start with conservative values, verify a normal RAG request, and increase them only when the workload requires it.

Streaming support

Streaming behavior differs between providers. If responses hang, render incorrectly, or fail during generation:

1. Confirm that the selected model supports streaming.
2. Confirm that the installed AnythingLLM version supports the expected streaming behavior.
3. Temporarily disable streaming in the Generic OpenAI provider and retry.

UI and .env conflicts

Self-hosted deployments can have configuration conflicts between UI settings and environment files. If a value reverts after a restart, inspect:

/app/server/.env

Community reports have included LLM_PROVIDER being reset. The important point is to identify which configuration source owns the setting in the current deployment rather than repeatedly changing the UI.

Usage and model selection

I use smaller or cheaper models while developing prompts, RAG retrieval, and agent tools, then reserve more capable models for production paths. Central model switching makes this practical without modifying AnythingLLM’s workflows.

Usage dashboards and budget alerts are also worth enabling. Agent loops and large document contexts can consume more tokens than a basic chat test suggests.

Where this integration is useful

RAG workspaces

AnythingLLM’s document loaders and vector database can provide retrieved context while the configured gateway supplies the chat and embedding models. This allows combinations such as a lower-cost embedding model with a more capable generation model.

Agents

AnythingLLM agents can browse pages, call tools, and run automations. Routing their interpretation and control steps through the same OpenAI-compatible provider makes it easy to test different models without changing the agent implementation.

Agent actions should still be tested with safe prompts and, preferably, against a staging instance before being connected to production systems.

Model comparisons

Different workspaces or features can use different models. For example:

gpt-4o       production answers
gpt-4o-mini  development and low-cost testing

This is useful for comparing response quality, latency, tool behavior, and cost using the same AnythingLLM interface.

Multimodal workflows

Depending on the provider and AnythingLLM capabilities in use, the same general setup can support image captioning, multimodal summarization, or audio transcription through compatible image, audio, and specialized models.

A practical configuration checklist

Before considering the integration complete, I verify:

• The standalone curl or Python request returns HTTP 200.
• The configured base URL ends at /v1.
• The API key is stored securely.
• The model name exactly matches an available model.
• The context window and maximum output tokens match that model.
• Streaming is enabled only when both sides support it.
• AnythingLLM can create a normal chat completion.
• RAG works with the selected embedding configuration.
• Agent actions are tested in a controlled environment.
• Usage limits or alerts are configured.

The integration is deliberately simple: AnythingLLM supplies the application layer, while the Generic OpenAI provider supplies the compatibility boundary. Once that boundary is configured correctly, changing models becomes a settings task rather than an application rewrite.

---

Originally published at cometapi.com