While developing my RAG-LCC experimental RAG, I had multiple use cases:
RAG-LCC uses LLMs for:
- Prompt guard
- Inference (query answering)
- Reply judgement
- Query rewrite
- Query split
Although some of the tasks can be handled by the same LMM it is good practice to have other models then the inference one for prompt guard and judgement.
I use 2 sets of configurations, which I want to switch as convenient as possible.
Coding assistance:
I tried different models, which would be a separate discussion. But when I use local coding assistance, I cannot run the RAG tests because my Spark DGX can't provide it.
What it does
- Two configuration files
- Starts and stops models. Pre-flight memory check before startup
- Uses the NVIDIA provided vLLM docker runtime
- Integrates LiteLLM as central entry point (LLM proxy)
- Supports model groups, e.g. rag and rag_old
- Define aliases. So you must not remember the model name, only what it does, e.g. "judge"
- Status shows running and stopped models
- Log interaction allows you to attach and detach from logs
- Container creation and recreation. You can protect a models container from beeing recreated
Conclusion
Building a RAG system taught me that managing the LLMs I use is an important part of my development flow. What started as some lines of python code has become a GitHub repo on its own.
If you want to have a look: harinezumigel-llm-stack
Feedback and ideas are welcome.



Top comments (0)