DEV Community: Janet Wafula

RAG for Dummies: Making AI Smarter With Information Retrieval

Janet Wafula — Tue, 23 Sep 2025 07:04:44 +0000

Artificial intelligence tools like ChatGPT are great at generating text, but they don’t always “know” everything. Their knowledge is limited to what they were trained on, and that training data stops at a certain date. So, what happens if you want your AI to answer questions about new research, company documents, or yesterday’s news?

That’s where RAG (Retrieval-Augmented Generation) comes in.

What is RAG?

RAG stands for Retrieval-Augmented Generation. It’s a technique that makes large language models (LLMs) smarter by connecting them to an external knowledge source.

In simple terms:

Retrieval = finding relevant information from a database, documents, or the internet.

Generation = using that information to create a human-like answer.

Think of it like this:
Without RAG → The AI answers from memory only (like a student who studied months ago).
With RAG → The AI can also look things up first before answering (like a student with access to textbooks and notes).

How RAG Works (Step by Step)

User asks a question → e.g., “What are the key points of the Kenyan Constitution 2010?”

Retriever searches → The system looks through a collection of documents, knowledge bases, or even the web.

Relevant data is pulled out → Example: passages from the Constitution are retrieved.

Generator creates an answer → The language model reads both the question and the retrieved passages, then produces a clear response.

So instead of guessing, the AI grounds its answer in actual information.

Why is RAG Important?

Keeps AI updated → The model can access new information without retraining.

Custom knowledge → You can feed it your company manuals, research papers, or private datasets.

More accurate answers → Since the AI relies less on “hallucinations” (making things up).

Flexible use cases → From customer support chatbots to legal assistants, healthcare tools, or research helpers.

Simple Analogy

Imagine you’re writing a school essay:

Without RAG → You rely only on what’s in your head.

With RAG → You open a library, find the right book, and then write a much better essay.

That’s exactly how RAG boosts AI performance.

Real-World Examples of RAG

Search-powered chatbots: A company chatbot that pulls answers from product FAQs.

Legal assistants: Lawyers ask questions, and the AI retrieves relevant laws and past cases.

Healthcare: Doctors query medical guidelines, and the AI provides evidence-based summaries.

Research: Students ask about recent papers, and the AI retrieves the latest publications.

My Take on Supervised Learning: Focus on Classification

Janet Wafula — Sun, 24 Aug 2025 09:39:39 +0000

Supervised learning is essentially about learning from examples. It’s called “supervised” because the data already contains the answers, guiding the model step by step until it can make predictions on its own.Supervised learning, to me, mirrors how humans make decisions. We don’t make choices in a vacuum; we rely on past examples. A doctor diagnoses a patient based on years of cases they’ve studied,machines do the same with labeled data.

When we talk about classification, we’re dealing with problems where outcomes fall into distinct groups or classes. A classic case is fraud detection: the system must decide if a transaction is fraudulent or legitimate. The challenge lies in finding patterns in the features that point to one class over another.

Models We Use for Classification

KNN – classifies by “looking at neighbors.”
Logistic Regression — Works well for binary classification (yes/no, spam/not spam).
Decision Trees –A tree-like model where data is split into smaller groups it is clear, visual, and beginner-friendly.
Random Forests – an ensemble that builds many decision trees and improves accuracy.
SVMs – effective for complex, high-dimensional datasets.
Neural Networks – best for large-scale, deep feature learning.

Personal views, insights and challenges I have faced while working with classification

The biggest difficulty for me has been feature engineering. Sometimes the raw data isn’t enough, and the choice of features makes all the difference. Based on my last project on Car price prediction, I noticed that adding more features could improve the model's accuracy.What I love about classification is how it simplifies the messiness of life into categories.

Navigating the Trade-Off Between Type I and Type II Errors: A Medical Scenario

Janet Wafula — Mon, 11 Aug 2025 12:55:19 +0000

In the world of statistics and decision-making, particularly in critical fields like medicine, understanding the trade-off between Type I and Type II errors is essential. These errors, which occur in hypothesis testing, can have vastly different implications depending on the context. This article explores the trade-off between the two using a real-world medical example: screening for a life-threatening disease such as cancer.

Understanding Type I and Type II Errors
Before diving into the medical scenario, let’s clarify what these errors are:

Type I Error (False Positive): Rejecting a true null hypothesis. In medicine, this means diagnosing someone as having a disease when they actually don’t.

Type II Error (False Negative): Failing to reject a false null hypothesis. In this case, a patient with the disease is mistakenly told they are healthy.

Medical Scenario: Cancer Screening
Imagine a new test for early-stage pancreatic cancer, a disease that is notoriously difficult to detect and deadly when caught late.

Null Hypothesis (H₀): The patient does not have cancer.

Alternative Hypothesis (H₁): The patient does have cancer.

The diagnostic test must decide whether to reject or accept the null hypothesis based on the test result.

The Consequences of Each Error
Type I Error (False Positive)
Outcome: The test indicates the patient has cancer when they don’t.

Consequence: Emotional distress, costly and invasive follow-up procedures (like biopsies), and potential side effects from unnecessary treatments (e.g., chemotherapy).

Systemic impact: Overburdened healthcare system due to treating non-cancer cases as cancer.

Type II Error (False Negative)
Outcome: The test fails to detect cancer in a patient who actually has it.

Consequence: Delayed treatment, disease progression to an untreatable stage, and significantly reduced survival rate.

Systemic impact: Higher mortality, public distrust in screening programs.

Where to Trade Off: Prioritizing Based on Context
In medicine, the acceptable balance between these errors depends on the severity of the disease and the consequences of the errors.

In life-threatening, fast-progressing diseases:
Minimize Type II errors (i.e., false negatives).

It is often better to err on the side of caution and accept a higher number of false positives (Type I errors).

Why? Missing a true cancer diagnosis can be fatal. Early detection is key, even if it means further testing for some healthy patients.

In our scenario, it would be wise to design the cancer screening test to be highly sensitive, even if that reduces its specificity slightly. This ensures that most true cancer cases are caught (low Type II error), though some healthy individuals may receive false alarms (higher Type I error).

The Statistical Perspective
From a statistical design point of view:

Alpha (α) is the probability of a Type I error.

Beta (β) is the probability of a Type II error.

The power of a test = 1 - β, representing the ability to correctly detect true positives.

Tuning the test involves a trade-off:

Reducing α often increases β, and vice versa.

In critical cases like cancer screening, it's acceptable to set α at a slightly higher level (e.g., 0.05 or 0.10), to reduce β and increase the power of the test.

Real-World Implementation
PSA tests for prostate cancer and mammograms for breast cancer have historically faced this trade-off.

Modern approaches, like machine learning in diagnostics, aim to optimize both sensitivity and specificity using large datasets and risk stratification.

Conclusion
Type I and Type II errors are more than academic concepts,they can mean life or death in medical decisions. In high-stakes scenarios like cancer diagnosis, the cost of a false negative is often far greater than that of a false positive. Hence, medical testing typically prioritizes reducing Type II errors, accepting a tolerable increase in Type I errors. Balancing these errors requires not only statistical precision but also ethical consideration, patient safety, and healthcare resource management.

The Core Crew of Statistics: Meet Mean, Median, and Mode

Janet Wafula — Wed, 23 Jul 2025 07:38:01 +0000

When you first step into the world of data science, whether you’re analyzing the English Premier League table or Love Island voting patterns and also cleaning messy Excel sheets, you’ll come across one of the most fundamental ideas in statistics: measures of central tendency.

These measures are mean, median and mode and are like the compass of data analysis. They help us understand where the "center" of the data lies, giving us a summary of a dataset in just a single number. As simple as they seem, they hold massive importance when it comes to building models, making predictions, and communicating findings.

What Are the Measures of Central Tendency?

Mean (Average)

Add all the values and divide by the count.
Useful for: datasets without extreme outliers.
Example: If you have the ages of 5 islanders 21, 23, 24, 25, 70 the mean age is pulled up by that 70-year-old bombshell.

Median (Middle value)

Sort the values, pick the middle one (or average of two middle values if even number of items).
Useful when: your data has outliers or is skewed.
Example: In the same age data, the median is 24, which better reflects the typical age of the islanders.

Mode (Most frequent value)

The number that occurs most often.
Useful when: you're working with categorical data or seeing popularity.
Example: In Love Island votes, if most people vote for “Andreina”, then she’s the mode.

Why They Matter in Data Science

As someone interested in working with raw data, cleaning it, and drawing insight, you’ll realize these three measures are often your first checkpoint. Here’s why:

1. They summarize big datasets quickly.
Whether it’s health data or financial transactions, instead of going through every row, a quick average or median gives you a starting point.

2.They help detect skewness or outliers.
If the mean is much higher than the median, you likely have a few large outliers. That’s a sign to investigate further before jumping to conclusions.

3.They guide model building.
When preparing data for machine learning models, measures of central tendency often play a role in feature scaling, imputing missing values (like you asked with fillna()), and choosing thresholds.

4.They make data storytelling easier.
Whether you're presenting to a team or writing a LinkedIn post about your latest Power BI dashboard, saying "the average customer age is 32" communicates way more than showing a list of 500 numbers

Just like every story has a main character, every dataset has a center,a point that helps you make sense of everything else. Mean, median, and mode may seem like basic stats, but in data science, they are your first clues, your sanity check, and sometimes even your problem solvers

Power BI:How relationships make your data make sense

Janet Wafula — Wed, 18 Jun 2025 13:22:22 +0000

When I first started using Power BI, I was excited about building dashboards and making data look presentable.But after a few times working with Power BI,I realized at the heart of every data model there are relationships.

What Are Relationships in Power BI?

A relationship connects two tables based on a common column, allowing Power BI to combine data and filter it correctly.They act as links between tables, allowing you to filter data in one table based on selections in another

Let’s say you have:

A Sales table with CustomerID and order details
A Customers table with CustomerID and names

By creating a relationship between these two tables on CustomerID, you can analyze sales by customer name instead of just by ID.

Types of Relationships (and When I Use Them)

Here are the three main types I’ve worked with:

1. One-to-Many (1:*)

Most common. One customer, many sales. This is the most common type I use. One customer can have many sales, so their ID appears once in the Customers table but multiple times in the Sales table.

2. Many-to-Many

Rare,but useful when both tables have duplicate values.

3. Many-to-One

Many rows in the first table are related to one row in the second table.

4.One-to-one

Each row in the first table is related to only one row in the second table.

Cross Filtering: One-Way or Both?

When setting up a relationship, Power BI asks for the cross-filter direction:

Single: Filters flow from one table to another (safe for most use cases).

Both: Allows two-way filtering, great for complex models, but can slow things down or create loops.

I stick with single filtering unless I know I need both directions.

Key Takeaways

Always check Model view to see how your tables connect
Use unique IDs in your lookup tables (like Customers or Products).
Take time to learn relationships. They’re the foundation of everything else.
Learning Power BI isn’t just about dragging visuals onto a canvas,it’s about structuring your data so the story makes sense. Relationships are the quiet engine behind every great dashboard. They connect your tables, your insights, and your understanding of the bigger picture

How Excel is Used in Real-World Data Analysis

Janet Wafula — Tue, 10 Jun 2025 12:35:56 +0000

Introduction

Microsoft Excel is a tool used to analyze, interpret and visualize data.By organizing data using software like Excel, data analysts and other users can make information easier to view as data is added or changed.

Ways in which Excel is used in real-world data analysis

Data entry and storage -Microsoft Excel is widely used for data entry and storage due to its user-friendly interface, flexibility, and powerful organizational features.Most businesses use excel for storing employee records and other information.
Business Decision-Making
Businesses often rely on Excel to track sales data, monitor inventory levels, and forecast future trends. By analyzing historical data in Excel, companies can make more strategic decisions—such as when to launch a product or adjust pricing models.

3.Reporting on Finances
Excel is frequently used in accounting and finance to generate cash flow reports, income statements, and balance sheets. Financial analysts can assess a company's financial health and make budgetary decisions by using Excel to create models that forecast revenue and expenses.

Excel Features and how they can be applied

PivotTables
PivotTables are a powerful feature that summarize large datasets into easy-to-read tables. Businesses use PivotTables to break down sales by region, product category, or customer segment, making it easier to identify trends and outliers.

Charts (Column, Line, Pie, etc.)
Charts in Excel are used to visualize data, making it easier to understand trends, patterns, and comparisons.

Power Query
Cleans, transforms, and merges data from multiple sources.it is also an ETL tool

Personal Reflection: How Learning Excel Changed the Way I See Data

Before learning Excel, I saw data as just confusing numbers. But Excel helped me understand that data can tell a story if it's organized and presented well. With tools like sorting, formulas, and charts, I learned how to quickly find patterns and make sense of information to derive insights and solve problems in real life.

Charts, in particular, showed me how visuals can make data clearer and more impactful. Now, I use Excel not just to manage data, but to make better decisions in my daily life.