Bypass AI API Rate Limits & Achieve 100% Uptime with SmartCore LLM Proxy

🚀 Bypass AI API Rate Limits with SmartCore LLM Proxy

Hitting HTTP 429 Rate Limits on Google Gemini, DeepSeek, or OpenAI can crash your AI backend or assistant (like Hermes AI).

SmartCore LLM Proxy is a lightweight local gateway built with LiteLLM that automatically load-balances requests and routes to fallback models whenever your primary key hits a limit.

---

💡 How It Works

Instead of calling LLM APIs directly, your app points to http://127.0.0.1:4000:

Instead of calling LLM APIs directly, your app points to http://127.0.0.1:4000:
[ App / Hermes AI ] ──▶ [ SmartCore LLM Proxy ] │ ┌───────────────────────┴───────────────────────┐ ▼ ▼ Gemini 1.5 Pro ──(On 429)──▶ Gemini 1.5 Flash / DeepSeek

1. Primary Route: Sends requests to your main model.
2. Automatic Cooldown & Fallback: Put rate-limited endpoints on a 60s cooldown and failover instantly.

3. Zero Code Changes: Works with any OpenAI-compatible client.

🛠️ Quick Setup (4 Steps)

1. Clone & Install

bash
git clone https://github.com/aaserag1/SmartCore-LLM-Proxy.git
cd SmartCore-LLM-Proxy
pip install "litellm[proxy]"
2. Configure Keys (config.yaml)
Add your API keys and fallbacks in config.yaml:

yaml


model_list:
  - model_name: gemini-1.5-pro
    litellm_params:
      model: gemini/gemini-1.5-pro
      api_key: "YOUR_GOOGLE_KEY"
  - model_name: deepseek-coder
    litellm_params:
      model: deepseek/deepseek-chat
      api_key: "YOUR_DEEPSEEK_KEY"
router_settings:
  cooldown_time: 60
  fallbacks: 
    - gemini-1.5-pro: ["deepseek-coder"]
3. Run Gateway
Double-click Run_LiteLLM.bat or run:

bash


python -m litellm --config config.yaml --port 4000 --host 127.0.0.1
4. Connect Your App
Set your app's Base URL to http://127.0.0.1:4000.

🔗 Open Source & Contribution
Check out the project and leave a ⭐ on GitHub!

🐙 GitHub: https://github.com/aaserag1/SmartCore-LLM-Proxy
👨‍💻 Author: Ahmed Adel (Abo Adel)
5:09 AM

Comments

[CarbonLayer]

Nice practical approach, Ahmed. Putting the 429 handling and cooldown logic behind a LiteLLM-compatible endpoint seems like a clean way to keep that complexity out of application code.

I'm curious about one design choice: when the primary model is rate-limited, is your fallback order always fixed, or have you encountered cases where the backup model needed to be selected based on latency, cost, or compatibility with the original request?

Also, has the 60-second cooldown been sufficient in your testing, or have different providers required different recovery behavior?

Would be interested to hear what you've found while building and testing this.