If you want to become good at Generative AI/LLM engineering, watching tutorials is not enough.
The fastest way to understand these technologies is to build projects where you are forced to solve real problems: prompting, context management, retrieval, model adaptation, and evaluation.
Here is a practical project roadmap:
LLM → RAG → Fine-Tuning → Evals
1. Learn LLMs by Building a Lead Triaging Bot
🎯 Project: Lead Triaging Bot
Build an AI system that receives a new customer lead and decides:
- Is this a high-quality lead?
- What is the customer's intent?
- Which product/service are they interested in?
- How urgent is the lead?
- What should the sales team do next?
Example
Input:
"Hi, I'm looking for an enterprise plan for 200 employees. We need SSO and would like to schedule a demo next week."
Output:
Lead Quality: High
Intent: Enterprise Purchase
Company Size: 200 employees
Urgency: High
Recommended Action: Schedule Demo
What you'll learn
This project gives you a strong foundation in:
- LLM APIs
- Prompt engineering
- System prompts
- Structured outputs
- Few-shot prompting
- JSON responses
- Function/tool calling
- Temperature and model parameters
- Error handling
- LLM application architecture
Why this project?
Before jumping into RAG or fine-tuning, you should understand how an LLM behaves without external knowledge or model customization.
This gives you the baseline against which you can later compare RAG and fine-tuning.
2. Learn RAG by Building a Company Knowledge Assistant
🎯 Project: Company Knowledge Assistant
Now take the same LLM and give it access to your own knowledge base.
Upload documents such as:
Company Policies
Product Documentation
HR Policies
FAQs
Technical Documentation
Pricing Documents
Users should be able to ask questions about these documents.
Example
User:
"What is our work-from-home policy?"
RAG pipeline:
User Question
↓
Query Embedding
↓
Vector Database
↓
Retrieve Relevant Documents
↓
Context + Question
↓
LLM
↓
Grounded Answer
RAG works by retrieving relevant information from an external data source and providing that information to the LLM as context. (GitHub)
What you'll learn
Build the project in stages:
Level 1 — Basic RAG
- Document loading
- Chunking
- Embeddings
- Vector database
- Similarity search
- Context injection
Level 2 — Better RAG
- Metadata filtering
- Hybrid search
- Query rewriting
- Reranking
- Top-K retrieval
Level 3 — Production RAG
- Conversation history
- Citations
- Access control
- Streaming
- Caching
- Observability
Public GitHub project
A great reference is LangChain — RAG From Scratch.
It builds RAG progressively from indexing, retrieval and generation, making it particularly useful for understanding how RAG actually works rather than simply copying a framework implementation. (GitHub)
You can also explore LlamaIndex RAG example for a more application-oriented implementation. (GitHub)
3. Learn Fine-Tuning by Building a Medical Advisor
🎯 Project: Medical Advisor
Important: This should be treated as an educational AI project, not a real medical diagnostic system.
The goal is to take an open-source LLM and adapt it to produce responses in a particular domain and format.
For example, create a dataset containing:
Question
↓
Medical Context
↓
Expected Response
Then fine-tune an open model on your dataset.
Example
Input:
"What are common symptoms associated with iron deficiency?"
The model should learn to produce a response following your desired structure and style.
What you'll learn
This project teaches:
- Dataset preparation
- Instruction datasets
- Data cleaning
- Tokenization
- Training/validation splits
- Supervised Fine-Tuning (SFT)
- LoRA
- QLoRA
- Model checkpoints
- Training metrics
- Model comparison
- Inference with adapters
Instead of trying to fine-tune a huge model from scratch, start with LoRA/QLoRA. These techniques make experimentation much more practical.
Public GitHub projects
For a simple introduction:
Fine-Tuning LLMs with LoRA and QLoRA
This repository demonstrates LoRA and QLoRA fine-tuning using PyTorch and Hugging Face Transformers. (GitHub)
For a more complete implementation:
LLM Fine-Tuning — SFT, LoRA & QLoRA
It includes dataset loading, tokenization, SFT, LoRA and QLoRA examples. (GitHub)
The important lesson
Don't think:
Fine-tuning = giving the model more knowledge
Instead, think:
Fine-tuning = adapting model behavior, style, format or task performance.
For frequently changing factual knowledge, RAG is often a better solution.
4. Learn Evals by Building an LLM Evaluation System
🎯 Project: Evaluate Your AI Applications
This is the project most beginners skip.
And it is one of the most important.
Suppose your RAG system answers:
"What is the company's leave policy?"
How do you know whether the answer is actually good?
You need an evaluation system.
Build a Lead/RAG Evaluation Bot
Create a test dataset:
Question
Expected Answer
Retrieved Context
Generated Answer
Then evaluate the system automatically.
Evaluate:
Retrieval
- Did we retrieve the correct document?
- Was the relevant information present?
- Was irrelevant information retrieved?
Generation
- Is the answer correct?
- Is it relevant?
- Is it grounded in the retrieved context?
- Did the model hallucinate?
Example
Question:
What is our annual leave policy?
Expected:
Employees receive 24 days of annual leave.
Model Answer:
Employees receive 24 days of annual leave.
Evaluation:
Correctness: 1.0
Faithfulness: 1.0
Relevance: 1.0
Now intentionally introduce a bad answer:
Model Answer:
Employees receive 30 days of annual leave.
Evaluation:
Correctness: 0.0
Faithfulness: 0.0
You have now started building an LLM evaluation pipeline.
Public GitHub projects
A good reference is RAG Evaluation Framework.
It separates evaluation into retrieval quality and generation quality and uses LLM-based evaluation with LangChain. (GitHub)
Another useful project is Ragas, which provides metrics and test-data generation for evaluating LLM applications and RAG systems. (GitHub)
You can also study LLM RAG Eval, which focuses specifically on evaluating RAG pipelines. (GitHub)
The Complete Learning Roadmap
Instead of building four unrelated projects, build them as a progression:
GENERATIVE AI
│
▼
┌─────────────────────┐
│ 1. Lead Triaging Bot│
│ LLM │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ 2. Knowledge │
│ Assistant │
│ RAG │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ 3. Medical Advisor │
│ Fine-Tuning │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ 4. Evaluation │
│ Framework │
│ Evals │
└─────────────────────┘
What You Should Know After Building All 4
| Project | Technology | You Learn |
|---|---|---|
| Lead Triaging Bot | LLM | Prompting, structured output, tools |
| Knowledge Assistant | RAG | Embeddings, retrieval, vector DB, RAG |
| Medical Advisor | Fine-Tuning | SFT, LoRA, QLoRA, datasets |
| Evaluation Framework | Evals | Metrics, test datasets, hallucination detection |
But the real value comes when you connect them.
Your final architecture can look like:
User
│
▼
Lead / Query
│
▼
┌──────────────┐
│ LLM │
└──────┬───────┘
│
┌─────────┴─────────┐
▼ ▼
RAG Fine-Tuned
Knowledge Model
│ │
└─────────┬─────────┘
▼
Final Answer
│
▼
EVALUATION
│
┌─────────┼─────────┐
▼ ▼ ▼
Correct? Relevant? Grounded?
The key takeaway
Don't learn these technologies as isolated topics.
Build progressively.
Start with an LLM application → add your own knowledge with RAG → adapt the model with fine-tuning → finally build an evaluation layer to measure whether your system actually improved.
That progression takes you from "I know how to call an LLM API" to "I can design, improve and evaluate production-style LLM systems."
Top comments (0)