Fine-Tuning with a Handful of Examples

Sumanth Kumar Kotagudem — Fri, 06 Nov 2026 08:51:22 +0000

Introduction

It was my first day of the internship, and I was given a brief about my new project.

It was a solo project to fine-tune a model on the organization's data; more specifically, on Jira tickets related to the most common database issues in the organization's cloud database services, including their investigations.

Seems easy, right?

After digging into the data for 2–3 days, I found these issues: the total dataset contained around 900+ issues. After categorizing the data, I was able to identify 10 broad categories (9 + miscellaneous).

The Problem

The total data size is very small (around 800).
The domain is not niche, nor is the product. As a result, you get a zero-shot response that is almost right but generic and not precise.
There is a very high class imbalance, which makes the data-sparsity issue worse.

My honest reaction was that I had no hope at the start because the problem did not make much sense to me. I felt that there was not much uniqueness in the domain, nor was there enough data. I had no hope that I would make any meaningful contribution. My internship was at stake, but I still had to make it work, as there was no other option.

The Journey

So, the journey of the Desert Wandering begins.

Collect: For the initial two weeks, I spent my time collecting, processing, and preparing the data.

Why two weeks? You know, organizational data is generally not in one place. You need permissions to access the data, understand the available methods for retrieving it, and determine where the required information is stored.

I tried using MCPs and different skill files to retrieve the organization's Jira data. Am I good? No. Most of the evidence and logs needed to be accessed from other data providers, such as Splunk, Dynatrace, database admin views, stack calls, etc., to collect all the data.

I also created an alternative. Since I had little hope in fine-tuning, I created a RAG alternative with hybrid retrieval, rerankers, and different prompting techniques as a fallback.

Then, Prepare: I summarized the evidence using rules and LLM calls to provide an overview, specified the irregularities or error traces, bound the data to the tickets, cleaned the comments, and organized the data.

The model needs to predict the root cause and the fix, which are present in the final comments.

Using LLMs for preprocessing was fast and more accurate. Worried about tokens? The data itself is small.

Data Augmentation (The Main Part): I spent the next 3–4 weeks mostly building different training datasets, as the model-training setup is easy nowadays. Obviously, Claude Code just needs a few prompts, and it is done.

I went with the synthetic approach first, using an LLM to produce multiple, diverse data entries across the categories, inversely proportional to the size of each label category.

Did it work? No.

The fine-tuned results and the zero-shot results looked the same. Why?

Because the new data entries contained no useful information to learn; they were just repetitions of the model's parametric knowledge.

Attempt 2: I tried different models, including larger models. The results were better, but this still was not a very useful direction. Was the model able to learn the most from the data? No.

That's when I understood that this was not the way forward. So, what did I do?

I tried to impute the data from the data itself. How?

The Jira tickets are basically conversations between the consumer and the engineering team. I used these conversations and tried to create data in turns, resulting in more data without introducing any noise.

This also helped the model find the early signals of the root cause.

Hurray! I finally saw some learning, and the model was working.

Optimizing: During the later weeks (7–8), I tried different models, made various attempts to make the data richer, tuned the hyperparameters, performed error analysis, and optimized the pipeline further.

I was able to produce better-quality results than the RAG approach and larger models in zero-shot settings.

I gave a presentation to the team, and it went well.

Optional (Later Weeks 9–11): I spent time serving the model, building an agentic version, setting up the local environment, and creating a model harness. I was also made part of a new project to create rule validation in the agentic system.

I used a combination of Knowledge Graphs, Rule Matching, and RAG to validate the agent's response and suggest the potential impacts of the agent's actions.

It was a good, quick brainstorming experience.

Lessons Learned

Your model is only as good as your data.

AI still needs humans for new directions; otherwise, it can get stuck in the same loop.

No matter how unusual the problem may be, there is always a solution. After all, we are engineers; we make it work.

That's the end.

Let me know your thoughts on where I could improve.

It's my first blog. I would never have written it if @georgekobaidze hadn't pushed me.

DEV Community: Sumanth Kumar Kotagudem

Fine-Tuning with a Handful of Examples

Introduction

The Problem

The Journey

Lessons Learned