Bypass AI API Rate Limits & Achieve 100% Uptime with SmartCore LLM Proxy

#ai #llm #api #backend

🚀 Bypass AI API Rate Limits with SmartCore LLM Proxy

Hitting HTTP 429 Rate Limits on Google Gemini, DeepSeek, or OpenAI can crash your AI backend or assistant (like Hermes AI).

SmartCore LLM Proxy is a lightweight local gateway built with LiteLLM that automatically load-balances requests and routes to fallback models whenever your primary key hits a limit.

💡 How It Works

Instead of calling LLM APIs directly, your app points to http://127.0.0.1:4000:

Instead of calling LLM APIs directly, your app points to http://127.0.0.1:4000:
[ App / Hermes AI ] ──▶ [ SmartCore LLM Proxy ] │ ┌───────────────────────┴───────────────────────┐ ▼ ▼ Gemini 1.5 Pro ──(On 429)──▶ Gemini 1.5 Flash / DeepSeek

Primary Route: Sends requests to your main model.
Automatic Cooldown & Fallback: Put rate-limited endpoints on a 60s cooldown and failover instantly.

3. Zero Code Changes: Works with any OpenAI-compatible client.

🛠️ Quick Setup (4 Steps)

1. Clone & Install


bash
git clone https://github.com/aaserag1/SmartCore-LLM-Proxy.git
cd SmartCore-LLM-Proxy
pip install "litellm[proxy]"
2. Configure Keys (config.yaml)
Add your API keys and fallbacks in config.yaml:

yaml


model_list:
  - model_name: gemini-1.5-pro
    litellm_params:
      model: gemini/gemini-1.5-pro
      api_key: "YOUR_GOOGLE_KEY"
  - model_name: deepseek-coder
    litellm_params:
      model: deepseek/deepseek-chat
      api_key: "YOUR_DEEPSEEK_KEY"
router_settings:
  cooldown_time: 60
  fallbacks: 
    - gemini-1.5-pro: ["deepseek-coder"]
3. Run Gateway
Double-click Run_LiteLLM.bat or run:

bash


python -m litellm --config config.yaml --port 4000 --host 127.0.0.1
4. Connect Your App
Set your app's Base URL to http://127.0.0.1:4000.

🔗 Open Source & Contribution
Check out the project and leave a ⭐ on GitHub!

🐙 GitHub: https://github.com/aaserag1/SmartCore-LLM-Proxy
👨‍💻 Author: Ahmed Adel (Abo Adel)
5:09 AM