How to Connect AnythingLLM to APIVAI (Cheap Claude & GPT + RAG)

Connect AnythingLLM to APIVAI

AnythingLLM is a popular all-in-one app for chatting with your documents
(RAG). It supports a generic OpenAI-compatible LLM provider, so you can run it on APIVAI's Claude
and GPT models cheaply while your documents stay local.

Configure the LLM

1. Open Settings → LLM Preference.
2. Provider: Generic OpenAI (OpenAI-compatible).
3. Set:
   • Base URL: https://api.apivai.com/v1
   • API Key: your APIVAI key
   • Chat Model: a name APIVAI serves (e.g. claude-sonnet-4-6, gpt-5.5)
   • Token context window / max tokens: match the model.
4. Save.

Embeddings

AnythingLLM can use its built-in/local embeddings for retrieval — keep those, and use APIVAI for the
chat/answer model. (APIVAI focuses on the chat side: Claude and GPT.)

Confirm models

curl -s https://api.apivai.com/v1/models -H "Authorization: Bearer $APIVAI_API_KEY"

Use it

Create a workspace, upload documents, and ask questions — AnythingLLM retrieves relevant chunks and
answers with your chosen APIVAI model. Claude Sonnet is great for faithful long-context answers.

Troubleshooting

• Auth error — check the Base URL (/v1) and key.
• model_not_found — use a name from /v1/models.

FAQ

Does AnythingLLM work with APIVAI? Yes — pick the Generic OpenAI provider, set base URL
https://api.apivai.com/v1, your key, and a chat model.

Are my documents sent to APIVAI? Only the prompt with retrieved snippets is sent to the model;
your files stay in AnythingLLM. Use local embeddings for retrieval.

Which model for document Q&A? Claude Sonnet for faithful long-context answers; GPT-5.5 for fast
multilingual replies.

Get started

Set APIVAI as the Generic OpenAI LLM in AnythingLLM, upload docs, and chat. Examples:
APIVAI examples repo.