DEV Community: MindsDB

How Web Apps Like Midday Gain a Competitive Edge with Supabase and MindsDB

MindsDB Team — Wed, 04 Feb 2026 16:42:45 +0000

Written by Chandre Van Der Westhuizen, Community & Marketing Co-ordinator at MindsDB

SaaS companies live and die by their metrics. Data is collected in operational systems, copied into warehouses, transformed through pipelines, and finally visualized in dashboards. Finance teams learn where to click. Analysts learn how to maintain the machinery. Leadership waits for answers.

This approach worked when businesses were simpler and questions were predictable. But modern SaaS companies don't operate that way anymore. Metrics change weekly. Pricing models evolve. Usage patterns shift quickly. And the most important questions are often the ones no dashboard was designed to answer.

This is why finance analytics is reaching an inflection point.
Monthly Recurring Revenue (MRR), Annual Recurring Revenue (ARR), active users, churn risk - these numbers drive decisions across finance, product, and leadership. Tools like Midday.ai exist to make those metrics visible and actionable.

But there's a growing challenge beneath the surface: As data grows more complex, dashboards alone are no longer enough.
This is where MindsDB comes in - especially for teams already using Supabase as their primary data store.

The Real Bottleneck Isn't Just Data - It’s Distance

Finance teams using Web Applications already have the data they need. What they lack is proximity to insight.

Every additional layer between the question and the answer introduces friction. Data must be replicated. Dashboards must be updated. Someone must interpret the results. By the time insight arrives, the moment has often passed.

Worse, trust erodes along the way. Numbers differ across tools. Reports disagree. Teams argue about which system is the source of truth.

In finance, uncertainty isn't just inconvenient. It's dangerous.

Supabase Has Become the System of Record for SaaS Finance

Platforms like Midday.ai, alongside Rally and Deriv, reflect a broader shift in how modern SaaS products are built.

Supabase is no longer just a backend convenience. It increasingly holds the most critical business data: subscriptions, invoices, usage, customer activity, and even pre-aggregated metrics like MRR and ARR.

For many teams, Supabase is the financial system of record ,especially with its row-level security and all-in-one back-end solutions.

And that raises an uncomfortable question. If the authoritative data already lives in Postgres, why do we keep copying it elsewhere just to understand it?

When Intelligence Lives Where the Data Does

MindsDB challenges the assumption that analytics must live outside operational systems.

Instead of moving financial and user data into separate analytics stacks, MindsDB allows AI to run directly on Supabase. Queries execute where the data lives. Answers reflect the current state of the business. Every result can be traced back to real rows.

This is not just an architectural preference. It fundamentally changes how finance teams interact with their data.

Many AI analytics tools fall short for finance teams because their answers feel disconnected from reality. When numbers lack context, can't be audited, or appear probabilistic, trust breaks down in a domain that demands precision. MindsDB addresses this by grounding every answer directly in live Supabase data, with no hidden transformations or black-box aggregation, making each result fully traceable to its source.

When intelligence lives next to the source of truth, freshness and trust stop being tradeoffs.

MindsDB and Supabase: Bringing AI Intelligence Directly to Your Web Apps’ Data

Supabase provides a reliable Postgres-based system of record, while MindsDB turns that data into a queryable intelligence layer, enabling natural-language analytics, hybrid search, and AI agents that stay grounded in real rows. The result is faster insights, fewer pipelines to maintain, and analytics teams can trust—because every answer remains fresh, explainable, and tied back to the source of truth.

For web apps like Midday, MindsDB’s federated query engine allows Supabase to make financial intelligence a native part of the product rather than a separate analytics layer. This reduces infrastructure complexity for the product team while delivering a more flexible, differentiated experience for finance teams who need accurate insights they can act on immediately.

Let’s explore a use case for Midday where we connect web application data hosted in Supabase to MindsDB, build Knowledge Bases and perform Hybrid Search.

Please note, we will make use of synthetic data for Midday hosted in Supabase.

Pre-requisites:

Access MindsDB’s GUI via Docker locally or MindsDB’s extension on Docker Desktop.
Configure your default models in the MindsDB GUI by navigating to Settings → Models.
Navigate to Manage Integrations in Settings and install the dependencies for Supabase.
Once you have installed the dependencies for Supabase, you can connect to it via our SQL Editor.

CREATE DATABASE display_name  --- display name for database.
WITH ENGINE = 'supabase',     --- name of the mindsdb handler
PARAMETERS = {
   "user": "postgres.gyjaxewghgebqavchydf",              --- the user to authenticate
   "password": "Supabase2026!",          --- the password to authenticate the user
   "host": "aws-1-us-west-1.pooler.supabase.com",              --- the host name of the Supabase connection
   "port": "5432",           --- the port to use when connecting
   "database": "postgres"           --- database name
};

PGVector will be used as storage, which you can also connect to following these docs.

We will make use of two tables:

midday_user_data which stores details about organizations.
midday_metrics which starts metrics like MRR, ARR, NRR and churn risk.

Querying Supabase Data with MindsDB Knowledge Bases and Hybrid Search

This section walks through how to use MindsDB Knowledge Bases and Hybrid Search on top of Supabase data to power AI-native analytics inside a web application. Using a Midday-style dataset, we’ll show how operational and financial data stored in Supabase can be transformed into a queryable knowledge layer that supports questions without moving or duplicating data. By the end of this tutorial, you’ll see how combining semantic search with structured filters enables accurate, explainable insights directly from your live Postgres source with the precision of SQL.

Now that you have connected your Supabase data to MindsDB, you can create a Knowledge Base using the CREATE KNOWLEDGE_BASE syntax:

CREATE KNOWLEDGE_BASE user_data_kb
USING
    storage = heroku.supabase_1,
    metadata_columns = [region, org_created_at, org_is_active, user_id, email, role, user_created_at, last_login_at, user_is_active, subscription_id, plan_name, billing_interval, price, subscription_status, subscription_start, subscription_cancel, seats_included, seats_purchased, invoice_id, invoice_date, invoice_amount, invoice_status, usage_date, active_minutes, sessions, events_ingested, reports_generated],
    content_columns = [org_name, full_name, industry],
    id_column = org_id;

Insert your data using the INSERT INTO syntax:

INSERT INTO user_data_kb
SELECT region, org_created_at, org_is_active, user_id, email, role, user_created_at, last_login_at, user_is_active, subscription_id, plan_name, billing_interval, price, subscription_status, subscription_start, subscription_cancel, seats_included, seats_purchased, invoice_id, invoice_date, invoice_amount, invoice_status, usage_date, active_minutes, sessions, events_ingested, reports_generated, org_name, full_name, industry, org_id
FROM supabase.public.midday_user_data;

You can select the Knowledge Base to see the data inserted using the SELECT statement:

SELECT * FROM user_data_kb

You can follow the same steps for the midday_metrics table:

--Create Knowledge Base
CREATE KNOWLEDGE_BASE midday_metrics_kb
USING
    storage = heroku.supbase_metrics,
    metadata_columns = ['mrr', 'arr', 'active_users', 'net_revenue_retention', 'churn_risk_score'],
    content_columns = ['metric_month'],
    id_column = 'org_id';

--Insert Data
INSERT INTO midday_metrics_kb
SELECT metric_month, org_id, mrr, arr, active_users, net_revenue_retention, churn_risk_score
FROM supabase.midday_metrics;

--Query the Knowledge Base
SELECT * FROM midday_metrics_kb

MindsDB’s Hybrid search allows you to query data with keyword and semantic search using SQL. Let’s see what insights we can obtain from the Midday data stored in Supabase.

Identify which paying customers are most at risk right now and immediately see whether low engagement is a plausible driver.

--Find high churn-risk customers and pull their engagement signals
SELECT * FROM midday_metrics_kb
JOIN user_data_kb
ON midday_metrics_kb.id = user_data_kb.id
WHERE
 midday_metrics_kb.content LIKE '2025-10'
 AND midday_metrics_kb.churn_risk_score >= 0.70
 AND user_data_kb.org_is_active = 'true';

This is the fastest “save revenue” query—finance and Customer Service can prioritize outreach based on risk and product behavior in one view.

Surface customers who contribute meaningful recurring revenue but aren’t adopting the product.

--“High MRR, low usage” accounts (classic expansion + retention signal)
SELECT * FROM midday_metrics_kb
JOIN user_data_kb
ON midday_metrics_kb.id = user_data_kb.id
WHERE midday_metrics_kb.mrr >= '249'
 AND user_data_kb.active_minutes <= 20
 AND user_data_kb.sessions <= 2
 AND user_data_kb.subscription_status = 'active';

Low adoption is one of the strongest leading indicators of churn; catching it early protects ARR.

You can detect revenue leakage and billing failures without waiting for finance tooling or delayed reports.

--Accounts with payment issues + current subscription context
SELECT * FROM user_data_kb
WHERE
 invoice_status = 'failed'
 AND subscription_status = 'active'
 AND org_is_active = 'true'
 AND user_is_active = 'true';

Failed invoices are a direct threat to cash flow and can also signal account distress or impending churn.

Find customers who have outgrown their current plan footprint.

--Seat overages: customers growing beyond plan assumptions
SELECT *
FROM user_data_kb
JOIN midday_metrics_kb
 ON user_data_kb.id = midday_metrics_kb.id
WHERE
 user_data_kb.subscription_status = 'active'
 AND user_data_kb.seats_purchased > user_data_kb.seats_included
 AND midday_metrics_kb.content LIKE '2025-12';

Seat overages are a clean expansion motion—great for upsell workflows and improving NRR.

Identify customers still paying but no longer engaged—often a precursor to cancellation at renewal time.

--“Silent churn” candidates: active subscription, but no recent logins
SELECT * FROM user_data_kb
JOIN midday_metrics_kb
 ON user_data_kb.id = midday_metrics_kb.id
WHERE
 user_data_kb.subscription_status = 'active'
 AND user_data_kb.last_login_at <= '2025-12-15'
 AND midday_metrics_kb.content LIKE '2025-12'

This is a proactive retention list for finance and customer service, especially ahead of renewals.

Flag accounts contributing recurring revenue but shrinking or at risk of contraction.

--MRR present but retention weak
SELECT *
FROM midday_metrics_kb
JOIN user_data_kb
ON midday_metrics_kb.id = user_data_kb.id
WHERE midday_metrics_kb.content = ('2025-01' and 2025)
 AND midday_metrics_kb.mrr >= '99'
 AND midday_metrics_kb.net_revenue_retention < '1.00'
 AND user_data_kb.org_is_active = 'true';

Contraction reduces ARR quietly—catching it early helps protect revenue.

Find cases where revenue retention looks good but the subscription shows cancellation signals.

--NRR strong but subscription status at risk
SELECT *
FROM midday_metrics_kb
JOIN user_data_kb
 ON midday_metrics_kb.id = user_data_kb.id
WHERE midday_metrics_kb.content = '2025-12'
 AND midday_metrics_kb.net_revenue_retention >= 1.05
 AND user_data_kb.subscription_status = 'canceled';

This helps detect “NRR lag” (metrics still look fine, but renewal risk is real).

Find the accounts that could cause the biggest revenue loss if they churn.

```sql--Revenue risk: high MRR + high churn risk
SELECT *
FROM midday_metrics_kb
JOIN user_data_kb
ON midday_metrics_kb.id = user_data_kb.id
WHERE midday_metrics_kb.content = '2026-01'
AND midday_metrics_kb.mrr >= '249'
AND midday_metrics_kb.churn_risk_score >= 0.70
AND user_data_kb.org_is_active = 'true';




This allows finance and customer service teams prioritize saves by “revenue at risk,” not just customer count. 

![mindsdb](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0f90b2xdi40xxwbl0c63.png)

Find accounts expanding and engaging broadly across users.



```sql--High NRR + high active_users (product-market fit signal)
SELECT *
FROM midday_metrics_kb
JOIN user_data_kb
ON midday_metrics_kb.id = user_data_kb.id
WHERE midday_metrics_kb.content = '2026-01'
 AND midday_metrics_kb.net_revenue_retention >= '1.10'
 AND midday_metrics_kb.active_users >= '10'
 AND user_data_kb.org_is_active = 'true';

This is a high-confidence “sticky + growing” segment, useful for forecasting and expansion playbooks.

Validate contraction risk using real usage behavior.

--NRR weak + low engagement (risk validation)
SELECT *
FROM midday_metrics_kb
JOIN user_data_kb
 ON midday_metrics_kb.id = user_data_kb.id
WHERE midday_metrics_kb.content = ('2025-01' and '2025-12')
 AND midday_metrics_kb.net_revenue_retention < 1.00
 AND user_data_kb.active_minutes <= 10
 AND user_data_kb.sessions <= 1
 AND user_data_kb.user_is_active = 'true';

Confirms whether retention issues are product adoption issues versus pricing/billing factors.

What This Means for Web Apps Like Midday.ai Using Supabase

Web apps like Midday.ai are evolving from static dashboards into systems that help teams reason about financial health. By pairing Supabase as the source of truth with MindsDB Knowledge Bases and Hybrid Search, these platforms can deliver real-time, conversational financial insights that remain accurate and explainable. This represents a shift toward AI-native analytics that finance teams can trust and act on.

At MindsDB, we believe in simplifying getting real-time answers. If your AI Analytics needs are more complex, we’re here to help. Contact our team that understands AI analytics demand precision and scale.

Conclusion: From Dashboards to Dialogue

The future of finance analytics isn't more charts or denser dashboards.

It’s dialogue.

A continuous conversation between the business, its data, and the people responsible for decisions. Supabase provides the foundation. MindsDB provides the intelligence. Knowledge Bases and Hybrid Search ensure that intelligence operates safely.

Together, they point toward a world where finance teams spend less time navigating tools and more time understanding what's really happening in their business.

And that's where meaningful decisions begin.

Building a Semantic Search Knowledge Base with MindsDB

MindsDB Team — Wed, 28 Jan 2026 16:30:26 +0000

Written by Andriy Burkov, Ph.D. & Author, MindsDB Advisor

What happens when a developer searches for "how to make async HTTP calls" but your documentation says "asynchronous network requests"? Traditional keyword search fails—even though the content is exactly what they need.

This is the fundamental limitation of keyword search: it matches words, not meaning.

In this tutorial, we'll build a semantic search system using MindsDB that understands user intent. Using 2 million Stack Overflow posts, we'll create knowledge bases with two different vector storage backends—pgvector and FAISS—and compare their performance.

What You'll Learn:

How MindsDB knowledge bases convert text into searchable vectors
Setting up pgvector (PostgreSQL-based) and FAISS (Facebook AI Similarity Search) storage
Combining semantic search with metadata filters
Building an AI agent that uses your knowledge base to answer questions

Prerequisites:

A MindsDB account (cloud or self-hosted)
PostgreSQL database with the Stack Overflow dataset
An OpenAI API key for embeddings

How Semantic Search Works

Before we dive in, let's understand the key difference between keyword and semantic search:

Aspect	Keyword Search	Semantic Search
Matching	Exact words	Meaning/intent
Query: "async HTTP"	Misses "asynchronous requests"	Finds both
Handles synonyms	❌	✅
Understands context	❌	✅

Semantic search works by:

Embedding: Converting text into numerical vectors using an embedding model
Storing: Saving these vectors in a vector database
Querying: Converting the search query to a vector and finding the closest matches

MindsDB handles all of this through its Knowledge Base abstraction.

Installing Dependencies

We need two packages:

mindsdb_sdk: Python client for interacting with MindsDB servers
pandas: For working with query results as DataFrames

!pip install mindsdb_sdk pandas

3. Connecting to the MindsDB Cloud Instance

import mindsdb_sdk

# Connect to your MindsDB instance
server = mindsdb_sdk.connect(
    'YOUR_MINDSDB_URL',  # e.g., 'https://cloud.mindsdb.com' for MindsDB Cloud
    login='YOUR_USERNAME',
    password='YOUR_PASSWORD'
)
print("Connected to MindsDB server")

Connected to MindsDB server

4. Connecting to the Data Source

def run_query(sql, success_msg="Query executed successfully"):
    """Execute a SQL query and handle 'already exists' errors gracefully."""
    try:
        result = server.query(sql).fetch()
        print(success_msg)
        return result
    except RuntimeError as e:
        if "already exists" in str(e).lower():
            print("Resource already exists - skipping")
        else:
            raise
        return None

# Connect to your PostgreSQL database containing Stack Overflow data
run_query("""
    CREATE DATABASE pg_sample
    WITH ENGINE = "postgres",
    PARAMETERS = {
        "user": "YOUR_PG_USER",
        "password": "YOUR_PG_PASSWORD",
        "host": "YOUR_PG_HOST",
        "port": "5432",
        "database": "sample"
    }
""", "Created pg_sample database connection")

Created pg_sample database connection

Let's verify the connection by exploring the data. Check the dataset size:

# Get total row count
count = server.query("SELECT COUNT(*) as cnt FROM pg_sample.stackoverflow_2m").fetch()
print(f"Dataset size: {count['cnt'].iloc[0]:,} rows")

Dataset size: 2,000,000 rows

Show 10 records:

# Test sample data
df = server.query("SELECT * FROM pg_sample.stackoverflow_2m LIMIT 10").fetch()

# Display as a nice table (in Jupyter notebooks)
from IPython.display import display
display(df)

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	id	PostTypeId	AcceptedAnswerId	ParentId	Score	ViewCount	Body	Title	ContentLicense	FavoriteCount	CreationDate	LastActivityDate	LastEditDate	LastEditorUserId	OwnerUserId	Tags
0	7	2	NaN	4.0	522	NaN	An explicit cast to `double` like this isn't n...	None	CC BY-SA 4.0	NaN	2008-07-31T22:17:57.883	2019-10-21T14:03:54.607	2019-10-21T14:03:54.607	5496973.0	9.0	None
1	9	1	1404.0	NaN	2199	784860.0	Given a `DateTime` representing a person's bir...	How do I calculate someone's age based on a Da...	CC BY-SA 4.0	0.0	2008-07-31T23:40:59.743	2023-02-02T18:38:32.613	2022-07-27T22:34:36.320	3524942.0	1.0	c#,.net,datetime
2	11	1	1248.0	NaN	1644	197314.0	Given a specific `DateTime` value, how do I di...	Calculate relative time in C#	CC BY-SA 4.0	0.0	2008-07-31T23:55:37.967	2022-09-05T11:26:30.187	2022-07-10T00:19:55.237	16790137.0	1.0	c#,datetime,time,datediff,relative-time-span
3	14	1	NaN	NaN	491	173083.0	What is the difference between [Math.Floor()](...	Difference between Math.Floor() and Math.Trunc...	CC BY-SA 3.0	0.0	2008-08-01T00:59:11.177	2022-04-22T08:59:43.817	2017-02-25T17:42:17.810	6495084.0	11.0	.net,math
4	6	1	31.0	NaN	319	23465.0	I have an absolutely positioned `div` containi...	Why did the width collapse in the percentage w...	CC BY-SA 4.0	0.0	2008-07-31T22:08:08.620	2021-01-29T18:46:45.963	2021-01-29T18:46:45.963	9134576.0	9.0	html,css,internet-explorer-7
5	12	2	NaN	11.0	347	NaN	Here's how I do it\n\n ```\nvar ts = new TimeSp...	None	CC BY-SA 4.0	NaN	2008-07-31T23:56:41.303	2020-06-13T10:30:44.397	2020-06-13T10:30:44.397	238419.0	1.0	None
6	13	1	NaN	NaN	701	277780.0	Is there a standard way for a web server to be...	Determine a user's timezone	CC BY-SA 4.0	0.0	2008-08-01T00:42:38.903	2022-03-29T07:31:31.320	2020-12-03T03:37:56.313	584192.0	9.0	html,browser,timezone,user-agent,timezone-offset
7	4	1	7.0	NaN	794	70633.0	I want to assign the decimal variable "trans" ...	How to convert Decimal to Double in C#?	CC BY-SA 4.0	0.0	2008-07-31T21:42:52.667	2022-09-08T05:07:26.033	2022-09-08T05:07:26.033	16124033.0	8.0	c#,floating-point,type-conversion,double,decimal
8	17	1	26.0	NaN	198	85547.0	How do I store binary data in [MySQL](http://e...	Binary Data in MySQL	CC BY-SA 3.0	0.0	2008-08-01T05:09:55.993	2020-12-03T03:37:51.763	2020-12-03T03:37:51.763	584192.0	2.0	mysql,database,binary-data,data-storage
9	24	1	49.0	NaN	193	101180.0	If I have a trigger before the update on a tab...	Throw an error preventing a table update in a ...	CC BY-SA 4.0	0.0	2008-08-01T12:12:19.350	2021-01-29T12:57:17.153	2021-01-29T12:57:17.153	14152908.0	22.0	mysql,database,triggers

The Stack Overflow dataset contains 2 million posts—both questions (PostTypeId=1) and answers (PostTypeId=2). Key columns include:

Id - Unique identifier for each post
Body - The content we'll make semantically searchable
Title - The title of the post (questions only)
Tags - Programming language and topic tags (e.g., python, javascript)
Score - Community voting score—useful for prioritizing high-quality content
ViewCount - Popularity metric for filtering
PostTypeId - Type of post (1=question, 2=answer)
AcceptedAnswerId - ID of the accepted answer (for questions)
CreationDate, LastActivityDate, LastEditDate - Timestamps

This rich metadata allows us to combine semantic understanding with traditional filters—for example, finding Python questions about async programming with a score above 10.

4. Setting Up Vector Storage Backends

MindsDB supports multiple vector storage options. We'll set up both pgvector and a recently added FAISS and will compare how quick they are.

pgvector (PostgreSQL Extension)

pgvector is a PostgreSQL extension for vector similarity search. It's ideal when you want to keep vectors alongside your relational data.


python
# Create pgvector database connection
run_query("""
    CREATE DATABASE pg_vector
    WITH ENGINE = "pgvector",
    PARAMETERS = {
        "user": "YOUR_PG_USER",
        "password": "YOUR_PG_PASSWORD",
        "host": "YOUR_PG_HOST",
        "port": "5432",
        "database": "vector"
    }
""", "Created pg_vector database connection")

Created pg_vector database connection

FAISS (Facebook AI Similarity Search)

FAISS is a library for efficient similarity search developed by Facebook AI Research. It's optimized for fast similarity search on large datasets.


python
# Create FAISS database connection
run_query("""
    CREATE DATABASE db_faiss
    WITH ENGINE = 'duckdb_faiss',
    PARAMETERS = {
        "persist_directory": "/home/ubuntu/faiss"
    }
""", "Created db_faiss database connection")

Created db_faiss database connection

Choosing Between pgvector and FAISS

Feature	pgvector	FAISS
Best for	Integration with existing PostgreSQL	Maximum query speed
Persistence	Native PostgreSQL storage	File-based
Scalability	Good (PostgreSQL limits)	Excellent (billions of vectors)
Setup complexity	Requires PostgreSQL extension	Standalone
Query speed	Good (~19s for 2M vectors)	Excellent (~5s for 2M vectors)

For this tutorial, we'll implement both so you can see the performance difference firsthand.

5. Creating Knowledge Bases

Now we have a table with relational data and two vector stores to keep the embedding vectors. We are ready to create knowledge bases using both storage backends.

The knowledge base will:

Use OpenAI's text-embedding-3-small model for generating embeddings
Store the post Body as searchable content
Include metadata fields for filtering results

Knowledge Base with pgvector Storage


python
def kb_exists(kb_name):
    """Check if a knowledge base already exists."""
    try:
        result = server.query("SELECT name FROM information_schema.knowledge_bases").fetch()
        return kb_name in result['name'].values
    except Exception:
        return False

# Create pgvector knowledge base
if kb_exists("kb_stack_vector"):
    print("kb_stack_vector already exists - skipping creation")
else:
    run_query("""
        CREATE KNOWLEDGE_BASE kb_stack_vector
        USING
            storage = pg_vector.stack,
            embedding_model = {
                "provider": "openai",
                "model_name": "text-embedding-3-small"
            },
            content_columns = ['Body'],
            metadata_columns = [
                "PostTypeId",
                "AcceptedAnswerId",
                "ParentId",
                "Score",
                "ViewCount",
                "Title",
                "ContentLicense",
                "FavoriteCount",
                "CreationDate",
                "LastActivityDate",
                "LastEditDate",
                "LastEditorUserId",
                "OwnerUserId",
                "Tags"
            ]
    """, "Created kb_stack_vector knowledge base")

Created kb_stack_vector knowledge base

Knowledge Base with FAISS Storage


python
# Create FAISS knowledge base
if kb_exists("kb_stack_faiss"):
    print("kb_stack_faiss already exists - skipping creation")
else:
    run_query("""
        CREATE KNOWLEDGE_BASE kb_stack_faiss
        USING
            storage = db_faiss.stack,
            embedding_model = {
                "provider": "openai",
                "model_name": "text-embedding-3-small"
            },
            content_columns = ['Body'],
            metadata_columns = [
                "PostTypeId",
                "AcceptedAnswerId",
                "ParentId",
                "Score",
                "ViewCount",
                "Title",
                "ContentLicense",
                "FavoriteCount",
                "CreationDate",
                "LastActivityDate",
                "LastEditDate",
                "LastEditorUserId",
                "OwnerUserId",
                "Tags"
            ]
    """, "Created kb_stack_faiss knowledge base")

Created kb_stack_faiss knowledge base

Understanding the Parameters

Parameter	Description
`storage`	Specifies the vector database connection and table name
`embedding_model`	Configuration for the embedding model (provider and model name)
`content_columns`	Columns to embed and make semantically searchable
`metadata_columns`	Columns available for filtering (not embedded, but stored)

6. Loading Data into Knowledge Bases

Now we'll insert the Stack Overflow data into our knowledge bases. This process:

Fetches data from the source table in batches
Generates embeddings for content columns using the OpenAI API
Stores vectors and metadata in the vector database

Loading Data into pgvector Knowledge Base


python
def is_kb_empty(kb_name):
    """Check if a knowledge base is empty (fast - only fetches 1 row)."""
    result = server.query(f"SELECT id FROM {kb_name} LIMIT 1").fetch()
    return len(result) == 0

if is_kb_empty("kb_stack_vector"):
    print("kb_stack_vector is empty - starting data insertion...")
    server.query("""
        INSERT INTO kb_stack_vector
        SELECT * FROM pg_sample.stackoverflow_2m 
        USING 
            batch_size = 1000, 
            track_column = id
    """).fetch()
    print("Data insertion started for kb_stack_vector")
else:
    print("kb_stack_vector is not empty - skipping data insertion")

Data insertion started for kb_stack_vector

Loading Data into FAISS Knowledge Base


python
if is_kb_empty("kb_stack_faiss"):
    print("kb_stack_faiss is empty - starting data insertion...")
    server.query("""
        INSERT INTO kb_stack_faiss
        SELECT * FROM pg_sample.stackoverflow_2m 
        USING 
            batch_size = 1000, 
            track_column = id
    """).fetch()
    print("Data insertion started for kb_stack_faiss")
else:
    print("kb_stack_faiss is not empty - skipping data insertion")

Data insertion started for kb_stack_faiss

Wait until the data insertion is complete.

7. Querying the Knowledge Bases

Once data is loaded, you can perform semantic searches combined with metadata filtering.

Basic Semantic Search

Search for content related to "8-bit music" (finds semantically similar content):


python
import time

# Semantic search on pgvector KB
start = time.time()
results_vector = server.query("""
    SELECT * FROM kb_stack_vector 
    WHERE content = '8-bit music'
    AND Tags LIKE '%python%'
    LIMIT 10
""").fetch()
elapsed_vector = time.time() - start
print(f"pgvector query time: {elapsed_vector:.2f} seconds")
display(results_vector)

# Semantic search on FAISS KB
start = time.time()
results_faiss = server.query("""
    SELECT * FROM kb_stack_faiss 
    WHERE content = '8-bit music'
    AND Tags LIKE '%python%'
    LIMIT 10
""").fetch()
elapsed_faiss = time.time() - start
print(f"FAISS query time: {elapsed_faiss:.2f} seconds")
display(results_faiss)

pgvector query time: 19.21 seconds

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	id	chunk_id	chunk_content	distance	relevance	ContentLicense	ViewCount	LastEditDate	Score	AcceptedAnswerId	OwnerUserId	LastActivityDate	Tags	LastEditorUserId	PostTypeId	ParentId	Title	FavoriteCount	CreationDate	metadata
0	1118266	1118266:Body:1of2:0to971	Im trying to engineer in python a way of trans...	0.605447	0.622879	CC BY-SA 2.5	1694.0	2009-07-13T08:32:20.797	0	NaN	NaN	2010-03-17T15:16:17.060	python,audio	12855.0	1	None	List of values to a sound file	NaN	2009-07-13T08:27:25.393	{'Tags': 'python,audio', 'Score': 0, 'Title': ...
1	974071	974071:Body:1of1:0to791	I have a mosquito problem in my house. This wo...	0.615257	0.619097	CC BY-SA 2.5	55695.0	2017-05-23T12:32:21.507	44	974291.0	51197.0	2020-02-12T22:24:39.977	python,audio,mp3,frequency	-1.0	1	None	Python library for playing fixed-frequency sound	0.0	2009-06-10T07:05:02.037	{'Tags': 'python,audio,mp3,frequency', 'Score'...
2	1967040	1967040:Body:1of1:0to224	I am confused because there are a lot of progr...	0.626904	0.614665	CC BY-SA 2.5	6615.0	None	7	1968691.0	237934.0	2021-08-10T10:40:59.217	python,audio	NaN	1	None	How can i create a melody? Is there any sound-...	0.0	2009-12-27T21:04:34.243	{'Tags': 'python,audio', 'Score': 7, 'Title': ...
3	1118266	1118266:Body:2of2:972to1430	The current solution I'm thinking of involves ...	0.627442	0.614461	CC BY-SA 2.5	1694.0	2009-07-13T08:32:20.797	0	NaN	NaN	2010-03-17T15:16:17.060	python,audio	12855.0	1	None	List of values to a sound file	NaN	2009-07-13T08:27:25.393	{'Tags': 'python,audio', 'Score': 0, 'Title': ...
4	1344884	1344884:Body:1of1:0to327	I want to learn how to program a music applica...	0.643957	0.608289	CC BY-SA 2.5	2205.0	2017-05-23T12:11:22.607	7	1346272.0	164623.0	2022-04-14T09:12:07.197	python,perl,waveform	-1.0	1	None	Programming a Self Learning Music Maker	0.0	2009-08-28T03:28:03.937	{'Tags': 'python,perl,waveform', 'Score': 7, '...
5	2376505	2376505:Body:1of2:0to968	Write a function called listenToPicture that t...	0.645214	0.607824	CC BY-SA 2.5	3058.0	2010-03-04T02:28:26.703	0	NaN	285922.0	2010-03-06T05:27:48.017	python,image,audio	34397.0	1	None	How do I loop through every 4th pixel in every...	NaN	2010-03-04T02:26:22.603	{'Tags': 'python,image,audio', 'Score': 0, 'Ti...
6	2226853	2226853:Body:1of1:0to877	I'm trying to write a program to display PCM d...	0.654162	0.604536	CC BY-SA 2.5	12425.0	None	7	2226907.0	210920.0	2015-07-25T11:16:16.747	python,audio,pcm	NaN	1	None	Interpreting WAV Data	0.0	2010-02-09T05:01:25.703	{'Tags': 'python,audio,pcm', 'Score': 7, 'Titl...
7	1561104	1561104:Body:1of1:0to306	Is there a way to do this? Also, I need this t...	0.668074	0.599494	CC BY-SA 2.5	1303.0	2020-06-20T09:12:55.060	1	1561314.0	151377.0	2012-01-29T00:01:18.230	python,pygame,pitch	-1.0	1	None	Playing sounds with python and changing their ...	NaN	2009-10-13T15:44:54.267	{'Tags': 'python,pygame,pitch', 'Score': 1, 'T...
8	1382998	1382998:Body:4of4:2649to3382	``` \n¼ éíñ§ÐÌëÑ » ¼ ö ® © ’\n0 1\n2 10\n3 10\n...	0.670654	0.598568	CC BY-SA 3.0	12497.0	2011-06-09T06:00:51.243	18	1383721.0	6946.0	2015-06-04T17:13:43.323	python,unicode	6946.0	1	None	latin-1 to ascii	0.0	2009-09-05T10:44:40.167	{'Tags': 'python,unicode', 'Score': 18, 'Title...
9	1837686	1837686:Body:1of2:0to950	I wish to take a file encoded in UTF-8 that do...	0.675999	0.596659	CC BY-SA 3.0	3016.0	2011-10-15T13:17:24.520	2	NaN	NaN	2011-10-15T13:17:24.520	python,c,utf-8,compression	12113.0	1	None	Compressing UTF-8(or other 8-bit encoding) to ...	0.0	2009-12-03T04:43:05.963	{'Tags': 'python,c,utf-8,compression', 'Score'...

FAISS query time: 5.04 seconds

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	id	distance	chunk_id	chunk_content	relevance	ContentLicense	ViewCount	LastEditDate	Score	AcceptedAnswerId	OwnerUserId	ParentId	LastEditorUserId	LastActivityDate	Tags	PostTypeId	FavoriteCount	Title	CreationDate	metadata
0	1118266	0.605468	1118266:Body:1of2:0to971	Im trying to engineer in python a way of trans...	0.622871	CC BY-SA 2.5	1694.0	2009-07-13T08:32:20.797	0	NaN	NaN	None	12855.0	2010-03-17T15:16:17.060	python,audio	1	NaN	List of values to a sound file	2009-07-13T08:27:25.393	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...
1	974071	0.615225	974071:Body:1of1:0to791	I have a mosquito problem in my house. This wo...	0.619109	CC BY-SA 2.5	55695.0	2017-05-23T12:32:21.507	44	974291.0	51197.0	None	-1.0	2020-02-12T22:24:39.977	python,audio,mp3,frequency	1	0.0	Python library for playing fixed-frequency sound	2009-06-10T07:05:02.037	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...
2	1967040	0.626923	1967040:Body:1of1:0to224	I am confused because there are a lot of progr...	0.614657	CC BY-SA 2.5	6615.0	None	7	1968691.0	237934.0	None	NaN	2021-08-10T10:40:59.217	python,audio	1	0.0	How can i create a melody? Is there any sound-...	2009-12-27T21:04:34.243	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...
3	1118266	0.627461	1118266:Body:2of2:972to1430	The current solution I'm thinking of involves ...	0.614454	CC BY-SA 2.5	1694.0	2009-07-13T08:32:20.797	0	NaN	NaN	None	12855.0	2010-03-17T15:16:17.060	python,audio	1	NaN	List of values to a sound file	2009-07-13T08:27:25.393	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...
4	1344884	0.643955	1344884:Body:1of1:0to327	I want to learn how to program a music applica...	0.608289	CC BY-SA 2.5	2205.0	2017-05-23T12:11:22.607	7	1346272.0	164623.0	None	-1.0	2022-04-14T09:12:07.197	python,perl,waveform	1	0.0	Programming a Self Learning Music Maker	2009-08-28T03:28:03.937	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...
5	2376505	0.645192	2376505:Body:1of2:0to968	Write a function called listenToPicture that t...	0.607832	CC BY-SA 2.5	3058.0	2010-03-04T02:28:26.703	0	NaN	285922.0	None	34397.0	2010-03-06T05:27:48.017	python,image,audio	1	NaN	How do I loop through every 4th pixel in every...	2010-03-04T02:26:22.603	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...
6	2226853	0.654112	2226853:Body:1of1:0to877	I'm trying to write a program to display PCM d...	0.604554	CC BY-SA 2.5	12425.0	None	7	2226907.0	210920.0	None	NaN	2015-07-25T11:16:16.747	python,audio,pcm	1	0.0	Interpreting WAV Data	2010-02-09T05:01:25.703	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...
7	1561104	0.668055	1561104:Body:1of1:0to306	Is there a way to do this? Also, I need this t...	0.599501	CC BY-SA 2.5	1303.0	2020-06-20T09:12:55.060	1	1561314.0	151377.0	None	-1.0	2012-01-29T00:01:18.230	python,pygame,pitch	1	NaN	Playing sounds with python and changing their ...	2009-10-13T15:44:54.267	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...
8	1382998	0.670668	1382998:Body:4of4:2649to3382	```\n¼ éíñ§ÐÌëÑ » ¼ ö ® © ’\n0 1\n2 10\n3 10\n...	0.598563	CC BY-SA 3.0	12497.0	2011-06-09T06:00:51.243	18	1383721.0	6946.0	None	6946.0	2015-06-04T17:13:43.323	python,unicode	1	0.0	latin-1 to ascii	2009-09-05T10:44:40.167	{'ContentLicense': 'CC BY-SA 3.0', 'LastActivi...
9	1837686	0.675986	1837686:Body:1of2:0to950	I wish to take a file encoded in UTF-8 that do...	0.596664	CC BY-SA 3.0	3016.0	2011-10-15T13:17:24.520	2	NaN	NaN	None	12113.0	2011-10-15T13:17:24.520	python,c,utf-8,compression	1	0.0	Compressing UTF-8(or other 8-bit encoding) to ...	2009-12-03T04:43:05.963	{'ContentLicense': 'CC BY-SA 3.0', 'LastActivi...

Analyzing the Results

Notice how the search for "8-bit music" returned posts about:

Converting values to sound files
Playing fixed-frequency sounds
Creating melodies programmatically

None of these posts contain the exact phrase "8-bit music," yet they're all semantically relevant to chiptune/retro audio generation. This is the power of semantic search.

Also note the 4x speed improvement with FAISS (5 seconds vs 19 seconds for pgvector). For production systems with high query volumes, this difference is significant.

Combined Semantic and Metadata Filtering

Find AJAX-related posts tagged with jQuery that have high view counts:


python
# pgvector: Semantic search with metadata filters
start = time.time()
results = server.query("""
    SELECT * FROM kb_stack_vector 
    WHERE content = 'ajax'
        AND Tags LIKE '%jquery%'
        AND ViewCount > 1000.0
        AND relevance > 0.6
    LIMIT 10
""").fetch()
print(f"pgvector query time: {time.time() - start:.2f} seconds")
display(results)

# FAISS: Semantic search with metadata filters
start = time.time()
results = server.query("""
    SELECT * FROM kb_stack_faiss 
    WHERE content = 'ajax'
        AND Tags LIKE '%jquery%'
        AND ViewCount > 1000.0
        AND relevance > 0.6
    LIMIT 10
""").fetch()
print(f"FAISS query time: {time.time() - start:.2f} seconds")
display(results)

pgvector query time: 5.76 seconds

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	id	chunk_id	chunk_content	distance	relevance	ContentLicense	ViewCount	LastEditDate	Score	AcceptedAnswerId	OwnerUserId	LastActivityDate	Tags	LastEditorUserId	PostTypeId	ParentId	Title	FavoriteCount	CreationDate	metadata
0	1400637	1400637:Body:28of32:25627to26627	o.ajax({type:"POST",url:E,data:G,success:H,dat...	0.427265	0.700641	CC BY-SA 2.5	2741.0	2009-09-09T16:16:59.430	2	1400656.0	107129.0	2013-08-05T16:07:54.400	javascript,jquery	8590.0	1	None	Stop reload for ajax submitted form	NaN	2009-09-09T16:12:46.057	{'Tags': 'javascript,jquery', 'Score': 2, 'Tit...
1	1400637	1400637:Body:30of32:27488to28356	O=false;T.onload=T.onreadystatechange=function...	0.453764	0.687870	CC BY-SA 2.5	2741.0	2009-09-09T16:16:59.430	2	1400656.0	107129.0	2013-08-05T16:07:54.400	javascript,jquery	8590.0	1	None	Stop reload for ajax submitted form	NaN	2009-09-09T16:12:46.057	{'Tags': 'javascript,jquery', 'Score': 2, 'Tit...
2	1400637	1400637:Body:27of32:24691to25626	rn this},serialize:function(){return o.param(t...	0.454629	0.687460	CC BY-SA 2.5	2741.0	2009-09-09T16:16:59.430	2	1400656.0	107129.0	2013-08-05T16:07:54.400	javascript,jquery	8590.0	1	None	Stop reload for ajax submitted form	NaN	2009-09-09T16:12:46.057	{'Tags': 'javascript,jquery', 'Score': 2, 'Tit...
3	1424774	1424774:Body:2of2:934to1745	var self = this;\n $.ajax({\n ...	0.461486	0.684235	CC BY-SA 2.5	3601.0	None	1	1426940.0	173350.0	2020-06-08T10:43:45.037	jquery,loops	NaN	1	None	Loop with 8 times	NaN	2009-09-15T02:02:58.927	{'Tags': 'jquery,loops', 'Score': 1, 'Title': ...
4	1400637	1400637:Body:31of32:28357to29238	N=function(X){if(J.readyState==0){if(P){clearI...	0.462191	0.683905	CC BY-SA 2.5	2741.0	2009-09-09T16:16:59.430	2	1400656.0	107129.0	2013-08-05T16:07:54.400	javascript,jquery	8590.0	1	None	Stop reload for ajax submitted form	NaN	2009-09-09T16:12:46.057	{'Tags': 'javascript,jquery', 'Score': 2, 'Tit...
5	546344	546344:Body:2of3:902to1764	var before = function() { $(loading).show() ;...	0.463258	0.683407	CC BY-SA 2.5	1463.0	2009-02-13T16:17:38.170	0	546642.0	2755.0	2009-02-13T16:37:59.867	javascript,jquery,ajax	2755.0	1	None	Using jQuery, how can I store the result of a ...	0.0	2009-02-13T15:25:00.963	{'Tags': 'javascript,jquery,ajax', 'Score': 0,...
6	1279625	1279625:Body:2of3:782to1754	``` \n<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML ...	0.468882	0.680790	CC BY-SA 3.0	1130.0	2016-12-03T07:00:58.213	0	1279881.0	58375.0	2016-12-03T07:00:58.213	events,jquery,getjson	6637668.0	1	None	Trouble with jQuery Ajax timing	NaN	2009-08-14T19:06:28.043	{'Tags': 'events,jquery,getjson', 'Score': 0, ...
7	1400637	1400637:Body:32of32:29239to30048	L(){if(M.complete){M.complete(J,R)}if(M.global...	0.468944	0.680761	CC BY-SA 2.5	2741.0	2009-09-09T16:16:59.430	2	1400656.0	107129.0	2013-08-05T16:07:54.400	javascript,jquery	8590.0	1	None	Stop reload for ajax submitted form	NaN	2009-09-09T16:12:46.057	{'Tags': 'javascript,jquery', 'Score': 2, 'Tit...
8	1775625	1775625:Body:5of9:3144to4049	}\n\n}\n</script>\n\n\n\n<script type=...	0.472723	0.679014	CC BY-SA 2.5	2100.0	2009-11-21T14:46:00.250	1	1776406.0	212889.0	2009-11-21T19:03:52.070	jquery,form-submit	212889.0	1	None	jQuery - Multiple form submission trigger unre...	0.0	2009-11-21T14:32:41.383	{'Tags': 'jquery,form-submit', 'Score': 1, 'Ti...
9	1400637	1400637:Body:26of32:23690to24690	nclick")}o(function(){var L=document.createEle...	0.477784	0.676689	CC BY-SA 2.5	2741.0	2009-09-09T16:16:59.430	2	1400656.0	107129.0	2013-08-05T16:07:54.400	javascript,jquery	8590.0	1	None	Stop reload for ajax submitted form	NaN	2009-09-09T16:12:46.057	{'Tags': 'javascript,jquery', 'Score': 2, 'Tit...

FAISS query time: 2.50 seconds

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	id	distance	chunk_id	chunk_content	relevance	ContentLicense	ViewCount	LastEditDate	Score	AcceptedAnswerId	OwnerUserId	ParentId	LastEditorUserId	LastActivityDate	Tags	PostTypeId	FavoriteCount	Title	CreationDate	metadata
0	1400637	0.427243	1400637:Body:28of32:25627to26627	o.ajax({type:"POST",url:E,data:G,success:H,dat...	0.700651	CC BY-SA 2.5	2741.0	2009-09-09T16:16:59.430	2	1400656.0	107129.0	None	8590.0	2013-08-05T16:07:54.400	javascript,jquery	1	NaN	Stop reload for ajax submitted form	2009-09-09T16:12:46.057	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...
1	1400637	0.453769	1400637:Body:30of32:27488to28356	O=false;T.onload=T.onreadystatechange=function...	0.687867	CC BY-SA 2.5	2741.0	2009-09-09T16:16:59.430	2	1400656.0	107129.0	None	8590.0	2013-08-05T16:07:54.400	javascript,jquery	1	NaN	Stop reload for ajax submitted form	2009-09-09T16:12:46.057	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...
2	1400637	0.454589	1400637:Body:27of32:24691to25626	rn this},serialize:function(){return o.param(t...	0.687479	CC BY-SA 2.5	2741.0	2009-09-09T16:16:59.430	2	1400656.0	107129.0	None	8590.0	2013-08-05T16:07:54.400	javascript,jquery	1	NaN	Stop reload for ajax submitted form	2009-09-09T16:12:46.057	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...
3	1424774	0.461469	1424774:Body:2of2:934to1745	var self = this;\n $.ajax({\n ...	0.684243	CC BY-SA 2.5	3601.0	None	1	1426940.0	173350.0	None	NaN	2020-06-08T10:43:45.037	jquery,loops	1	NaN	Loop with 8 times	2009-09-15T02:02:58.927	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...
4	1400637	0.462233	1400637:Body:31of32:28357to29238	N=function(X){if(J.readyState==0){if(P){clearI...	0.683886	CC BY-SA 2.5	2741.0	2009-09-09T16:16:59.430	2	1400656.0	107129.0	None	8590.0	2013-08-05T16:07:54.400	javascript,jquery	1	NaN	Stop reload for ajax submitted form	2009-09-09T16:12:46.057	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...
5	546344	0.463237	546344:Body:2of3:902to1764	var before = function() { $(loading).show() ;...	0.683416	CC BY-SA 2.5	1463.0	2009-02-13T16:17:38.170	0	546642.0	2755.0	None	2755.0	2009-02-13T16:37:59.867	javascript,jquery,ajax	1	0.0	Using jQuery, how can I store the result of a ...	2009-02-13T15:25:00.963	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...
6	1279625	0.468854	1279625:Body:2of3:782to1754	```\n<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML ...	0.680803	CC BY-SA 3.0	1130.0	2016-12-03T07:00:58.213	0	1279881.0	58375.0	None	6637668.0	2016-12-03T07:00:58.213	events,jquery,getjson	1	NaN	Trouble with jQuery Ajax timing	2009-08-14T19:06:28.043	{'ContentLicense': 'CC BY-SA 3.0', 'LastActivi...
7	1400637	0.468931	1400637:Body:32of32:29239to30048	L(){if(M.complete){M.complete(J,R)}if(M.global...	0.680767	CC BY-SA 2.5	2741.0	2009-09-09T16:16:59.430	2	1400656.0	107129.0	None	8590.0	2013-08-05T16:07:54.400	javascript,jquery	1	NaN	Stop reload for ajax submitted form	2009-09-09T16:12:46.057	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...
8	1775625	0.472740	1775625:Body:5of9:3144to4049	}\n\n}\n</script>\n\n\n\n<script type=...	0.679007	CC BY-SA 2.5	2100.0	2009-11-21T14:46:00.250	1	1776406.0	212889.0	None	212889.0	2009-11-21T19:03:52.070	jquery,form-submit	1	0.0	jQuery - Multiple form submission trigger unre...	2009-11-21T14:32:41.383	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...
9	1400637	0.477785	1400637:Body:26of32:23690to24690	nclick")}o(function(){var L=document.createEle...	0.676688	CC BY-SA 2.5	2741.0	2009-09-09T16:16:59.430	2	1400656.0	107129.0	None	8590.0	2013-08-05T16:07:54.400	javascript,jquery	1	NaN	Stop reload for ajax submitted form	2009-09-09T16:12:46.057	{'ContentLicense': 'CC BY-SA 2.5', 'LastActivi...

Understanding Query Results

The query returns these columns:

Column	Description
`id`	Original document ID
`chunk_id`	Identifier for the text chunk
`chunk_content`	The actual text content
`metadata`	JSON object with all metadata fields
`distance`	Vector distance (lower = more similar)
`relevance`	Relevance score (higher = more relevant, 0-1)

Filtering by Relevance

Get only highly relevant results:

The Power of Combined Filtering

The query we just ran demonstrates MindsDB's hybrid search capability:


sql
SELECT * FROM kb_stack_faiss 
WHERE content = 'ajax'              -- Semantic match
    AND Tags LIKE '%jquery%'        -- Metadata filter
    AND ViewCount > 1000            -- Popularity threshold
    AND relevance > 0.6             -- Quality threshold

This finds posts that:

Are semantically similar to "ajax" (not just keyword matches)
Are tagged with jQuery
Have significant engagement (>1000 views)
Meet a minimum relevance score

This combination is impossible with traditional search and would require complex custom code with raw vector databases.


python
def run_query_ignore_exists(sql, success_msg="Query executed successfully"):
    """Execute a query, silently ignoring 'already exists' errors."""
    try:
        result = server.query(sql).fetch()
        print(success_msg)
        return result
    except RuntimeError as e:
        return None  # Silently ignore
# Create MindsDB Agent
run_query_ignore_exists("""
    drop agent stackoverflow_agent
""", "Dropped stackoverflow_agent")

run_query("""
    CREATE AGENT stackoverflow_agent
    USING
        model = {
            "provider": "openai",
            "model_name": "gpt-4.1"
        },
        data = {
            "knowledge_bases": ["mindsdb.kb_stack_faiss"]
        },
        prompt_template = '
            You are a helpful programming assistant. 
            mindsdb.kb_stack_faiss is a knowledge base that contains Stack Overflow questions and answers.
            Use this knowledge to provide accurate, helpful responses to programming questions.
            Include code examples when relevant.
            You must base your answer on the Stack Overflow questions and answers extracted from mindsdb.kb_stack_faiss.
            If you failed to get the results from mindsdb.kb_stack_faiss, answer I could not get the results from mindsdb.kb_stack_faiss.
            Print the chunk ID for each question and answer you based your answer on.
            IMPORTANT: Use a limit of 100 in your query to the knowledge base.
        '
""", "Created stackoverflow_agent")

Dropped stackoverflow_agent
Created stackoverflow_agent


python
# Query the agent
start = time.time()
response = server.query("""
    SELECT answer
    FROM stackoverflow_agent 
    WHERE question = 'Compare JavaScript to TypeScript for building web services'
""").fetch()
print(f"Agent response time: {time.time() - start:.2f} seconds\n")
print(response['answer'].iloc[0])

Agent response time: 63.44 seconds

To compare JavaScript and TypeScript for building web services, let's look at insights from Stack Overflow posts (see chunk IDs for reference):

JavaScript:


JavaScript is a dynamic, weakly typed, prototype-based language with first-class functions (1253285:Body:1of1:0to384).
It is the default language for web development, both on the client (browser) and, with Node.js, on the server (870980:Body:1of1:0to133).
JavaScript is flexible and widely supported, but its lack of static typing can lead to runtime errors and makes large codebases harder to maintain.


TypeScript:


While not directly mentioned in the top results, TypeScript is a superset of JavaScript that adds static typing and modern language features. It compiles to JavaScript, so it runs anywhere JavaScript does.
TypeScript helps catch errors at compile time, improves code readability, and is especially beneficial for large projects or teams.


Web Services:


JavaScript (with Node.js) is commonly used to build RESTful APIs and web services (208051:Body:1of1:0to147).
TypeScript is increasingly popular for the same purpose, as it provides all the benefits of JavaScript plus type safety and better tooling (e.g., autocompletion, refactoring).


Summary Table:




Feature
JavaScript
TypeScript




Typing
Dynamic, weakly typed
Static typing (optional)


Tooling
Good, but less type-aware
Excellent (autocompletion, refactor)


Learning Curve
Lower
Slightly higher (due to types)


Error Checking
Runtime
Compile-time + runtime


Ecosystem
Huge, universal
Same as JS, plus TS-specific tools


Maintainability
Can be challenging in large code
Easier in large codebases




Conclusion:  


For small projects or rapid prototyping, JavaScript is sufficient and easy to start with.
For larger projects, teams, or when maintainability and reliability are priorities, TypeScript is generally preferred.


References:  



1253285:Body:1of1:0to384


870980:Body:1of1:0to133


208051:Body:1of1:0to147



If you want more specific code examples or a deeper dive into either technology, let me know!

Feature	JavaScript	TypeScript
Typing	Dynamic, weakly typed	Static typing (optional)
Tooling	Good, but less type-aware	Excellent (autocompletion, refactor)
Learning Curve	Lower	Slightly higher (due to types)
Error Checking	Runtime	Compile-time + runtime
Ecosystem	Huge, universal	Same as JS, plus TS-specific tools
Maintainability	Can be challenging in large code	Easier in large codebases

Conclusion

We've built a complete semantic search system that:

Processes 2 million Stack Overflow posts
Supports both pgvector and FAISS backends
Combines semantic search with metadata filtering
Powers an AI agent for natural language queries

Key Takeaways

FAISS is much faster than pgvector for pure search queries
Metadata filtering lets you narrow results by tags, scores, dates
Knowledge bases abstract complexity—no need to manage embeddings manually
Agents can leverage knowledge bases for RAG-style applications

Next Steps

Try different embedding models
Add more data sources
Build a chat interface
Explore different chunking strategies

W.I.S.H – Whatever I Say Happens – Programming

MindsDB Team — Thu, 22 Jan 2026 16:37:25 +0000

Written by Jorge Torres, Co-founder & CEO at MindsDB

When Anyone Can Build Software, What Will CIOs Actually Invest In?

Every two weeks, I find myself sitting pretty on an airplane; with just enough connectivity to triage emails and Slack notifications, but not enough to do real work. These flights have become my thinking time; moments to step back and consider how the forces reshaping technology will affect our business, our customers, and the decisions that technology leaders face. Some of these thought experiments feel worth sharing. Here's the first one of 2026.

The End of Proprietary as a Moat

We're entering a software era where claiming "proprietary technology" as a competitive advantage is starting to feel not quaint but naive, like written in crayons. Engineering roadmaps as differentiators? Increasingly questionable. "Competitive moats" built purely on code? More like suggestions.

What we’ve been calling "vibe coding"; the casual ability to describe what you want and have AI generate working code, is rapidly evolving into something more significant. Call it W.I.S.H programming: Whatever I Say, Happens. And I don't mean entertaining but useless demos or toy applications. I mean functional backends, polished interfaces, scalable cloud infrastructure, tested and deployable. The whole stack.

Today, if you understand software architecture at the level of a competent project manager—someone who grasps how the components of a SaaS application connect, even if you've never written a line of code—you can build genuinely useful applications by describing what you need in plain English. Yes, there's still an interpretation dance between your intent and what the AI produces. Yes, quality and reliability still depend on your technical judgment about what needs testing and how components should integrate. But these systems are improving rapidly. Soon, they'll embed every best practice known for every layer of the stack automatically, bringing a level of attention to detail that currently requires years of engineering experience. All there, ready for you to just say it and see it magically happen before your eyes.

We're approaching a threshold where functional software can be thought into existence rather than coded into reality in the traditional sense. Even if that is a far-fetched exaggeration, I am inviting you to imagine a future where: software is a commodity, in which if we do not have our ears glued to the ground, we will all just be selling or buying reheated lasagna at different price points.

The Gmail Paradox

Follow this trajectory to its logical conclusion, and you arrive at an uncomfortable question: In a world where software becomes a commodity, where does durable value actually reside?

Consider Gmail. In the software-as-commodity future, cloning Gmail will be a weekend project. Building better versions of Gmail—with improved spam filtering, smarter categorization, better search—will be similarly trivial. The technical barriers that once protected established products are dissolving.

And yet.

No one is going to successfully storm that castle with an overnight Gmail clone, no matter how technically superior. Why? Because Gmail by itself isn't the product. Google Workspace is. And Google Workspace isn't just software—it's organizational infrastructure. It's the accumulated context of every conversation your company has ever had, every calendar invite, every shared document, every nested folder that someone has been curating since 2014.

Companies don't use Google Workspace because it's the theoretically optimal solution. They use it because now they are full people that have 47 nested folders that they’ve been curating since 2014, and if you try to migrate them to anything else, they will burn the place to the ground. They are recurrent customers, because migration would require organizational consensus that simply doesn't exist. That accumulated context isn't a feature. It's gravity. It's the business.

The principles that governed SaaS when the industry began—network effects, switching costs, data gravity—will still apply a decade from now. But they'll apply to different things.

What Actually Survives

In a future where anyone can build anything, value doesn't reside solely in what you build, even if it elegantly solves a clearly defined problem. If it solves that problem for a single team in isolation, it may be displaced by internally vibed tools that fit their specific workflow perfectly.

CIOs are going to face a constant calibration: Is this externally-sourced solution preventing technical debt, or is it creating friction we could eliminate by building something custom? When the cost of building drops dramatically, the calculus changes.

So where does sustained value live? I keep returning to a deceptively simple question: What do humans still need to agree on?

The answer points toward systems that can't simply be vibed into existence by any individual team, because their value depends on coordination across organizational boundaries. These are the systems where building in isolation creates more problems than it solves.

Trust as the Scarce Resource

Here's the realization that keeps crystallizing for me: AI is racing to make building software dramatically cheaper than it's ever been. But the faster that race progresses, the more expensive trust becomes.

Think about the implications. When every team can spin up internal tools overnight, who decides which source of truth actually matters? When you can clone any workflow application, who ensures the clone doesn't harbor compliance violations or security vulnerabilities that no one thought to check?

The winners in this landscape won't simply be the builders of workflow applications. They'll be the providers of what I've started calling "agreement infrastructure"—the systems of record where decisions get made, documented, and recognized by other parties. Approvals. Reviews. Planning. Compliance attestation. The unglamorous connective tissue between "I built a thing" and "other people can rely on it."

Wherever humans still need to make decisions together—and have those decisions carry weight beyond their immediate team—there's a durable business.

The Synthesis

The question I keep wrestling with: Can you deliver both? Can you provide the creative velocity that WISH programming enables while also building genuine agreement infrastructure—the kind that creates organizational gravity rather than technical debt?

That intersection—between radical accessibility and institutional trust—may be where the gravity rules of traditional SaaS and the new era of software commoditization find equilibrium. Where sustainable businesses will actually be built.

For CIOs navigating investment decisions, I'd suggest the filter isn't "what problems does this solve?" It's "what agreements does this enable, and how hard would those agreements be to recreate?"

The lasagna might be getting commoditized. The kitchen where everyone agrees to eat together? That's still worth building.

Building AI-Powered Data Analytics with MindsDB Enterprise: From Natural Language to Charts

MindsDB Team — Mon, 19 Jan 2026 10:48:58 +0000

Data analytics traditionally requires writing SQL queries, understanding database schemas, and manually creating visualizations. This creates a significant barrier for business users who need insights but lack technical expertise. With MindsDB Minds you could simply ask questions in plain English and receive both answers and charts automatically.

In this tutorial, I will show how to build an AI-powered analytics system using MindsDB Minds that can:

Understand natural language questions about your data
Automatically generate and execute SQL queries
Return formatted answers with dynamically generated charts

By the end of this tutorial, you'll have a working system that transforms questions like "What's the total sales revenue by product category?" into actionable insights complete with visualizations.

What You'll Learn

Setting up a MindsDB Mind with database connectivity
Asking data questions to MindsDB Mind and getting answers programmatically

Prerequisites

Before starting, ensure you have:

Python 3.8 or higher installed
A MindsDB API key (please contact MindsDB to get one)
Basic familiarity with Python and SQL concepts
The following Python packages installed:
- pandas
- openai
- minds-sdk (MindsDB Python client)

You can install the required packages with the following command:

pip install --upgrade pandas openai minds-sdk

The Dataset: Web Sales Analytics

This tutorial uses a web sales dataset stored in PostgreSQL available in your Minds dashboard. The dataset consists of four related tables that model an e-commerce business:

Table	Description	Key Columns
`websales_orders`	Order information	`order_id`, `order_date`, `ship_date`, `ship_mode`
`websales_sales`	Sales transactions	`order_id`, `product_id`, `customer_id`, `sales`, `quantity`, `discount`, `profit`
`websales_products`	Product catalog	`product_id`, `category`, `sub_category`, `product_name`
`websales_customers`	Customer information	`customer_id`, `customer_name`, `segment`, `country`, `city`, `state`, `region`

Table Relationships

websales_sales.product_id  → websales_products.product_id
websales_sales.order_id    → websales_orders.order_id
websales_sales.customer_id → websales_customers.customer_id

This schema supports a wide range of analytical questions — from product performance and customer segmentation to shipping analysis and regional trends.

For simplicity and to keep the focus on the Minds features, in this tutorial we used an existing dataset. Readers not familiar with the notion of data sources in MindsDB are invited to read this piece of documentation to learn how to make existing databases available for querying in MindsDB.

Step 1: Import Required Libraries

To start coding, let's import the necessary libraries for API communication, data handling, and MindsDB client operations:

import os, re, json
import pandas as pd
from openai import OpenAI
from minds.client import Client

The OpenAI library will be used as the API for communicating with a Mind. It's a popular choice of API supported by many chatbot and LLM providers. MindsDB supports it as well.

The Client is a MindsDB client we will use for Mind management.

Step 2: Configure API Credentials

Next, we set the MindsDB API URL, API key, and Mind name:

BASE_URL      = "https://mdb.ai/api/v1"
MINDS_API_KEY = "CREATE_YOUR_KEY" # "CREATE_YOUR_KEY at https://mindsdb.com/contact"
MIND_NAME     = "web_sales_demo_mind"
DATA_SOURCES  = [{
    'name': 'postgres_web_sales_datasource',
    'tables': [
        'websales_orders',
        'websales_sales',
        'websales_products',
        'websales_customers',
    ]
}]

Let's see what each parameter defines:

BASE_URL is the MindsDB API endpoint.
MINDS_API_KEY is your personal API key for authentication. Create one at the MindsDB portal.
MIND_NAME is a unique identifier for your Mind. Choose a descriptive name that reflects its purpose.
DATA_SOURCES specify which database tables the Mind should have access to:
- name is a reference name for this datasource configuration. This should match an existing datasource in your MindsDB environment.
- tables is a list of table names the Mind is allowed to query. Restricting access helps the AI focus on relevant data.

Step 3: Initialize the MindsDB Client

With credentials configured, we now can create a client instance that handles all communication with the MindsDB platform:

client = Client(api_key=MINDS_API_KEY, base_url=BASE_URL)

The Client object provides methods for creating, updating, and managing Minds. It handles authentication automatically using the provided API key.

Step 4: Create the Prompt Template

The prompt template is the heart of your Mind's behavior. It instructs the AI on how to interpret questions, generate SQL, and format responses:

PROMPT_TEMPLATE = """
# ROLE AND TASK

You are a precise data analyst with access to a SQL execution tool named **sql_db_query**.
You MUST (a) generate SQL, (b) EXECUTE it via sql_db_query, and (c) based ONLY on the returned rows,
return the answer in markdown format and always create a chart whenever possible.
Generate URL-encoded charts via quickchart.io like this:

https://quickchart.io/chart?c=%7Btype%3A%27line%27%2Cdata%3A%7Blabels%3A%5B%27Jan%27%2C%27Feb%27%2C%27Mar%27%2C%27Apr%27%2C%27May%27%2C%27Jun%27%5D%2Cdatasets%3A%5B%7Blabel%3A%27Sales%27%2Cdata%3A%5B65%2C59%2C80%2C81%2C56%2C95%5D%2CborderColor%3A%27rgb(75%2C192%2C192)%27%7D%5D%7D%7D
https://quickchart.io/chart?c=%7Btype%3A%27pie%27%2Cdata%3A%7Blabels%3A%5B%27CompanyA%27%2C%27CompanyB%27%2C%27CompanyC%27%2C%27Others%27%5D%2Cdatasets%3A%5B%7Bdata%3A%5B35%2C25%2C20%2C20%5D%2CbackgroundColor%3A%5B%27%23FF6384%27%2C%27%2336A2EB%27%2C%27%23FFCE56%27%2C%27%234BC0C0%27%5D%7D%5D%7D%7D
https://quickchart.io/chart?c=%7Btype%3A%27bar%27%2Cdata%3A%7Blabels%3A%5B%27North%27%2C%27South%27%2C%27East%27%2C%27West%27%5D%2Cdatasets%3A%5B%7Blabel%3A%27Revenue%27%2Cdata%3A%5B120%2C190%2C300%2C250%5D%7D%5D%7D%7D

# SCHEMA (use schema-qualified names ONLY; do NOT include datasource name)
- postgres_web_sales_datasource.websales_orders     (order_id, order_date, ship_date, ship_mode, ...)
- postgres_web_sales_datasource.websales_sales      (order_id, product_id, customer_id, sales, quantity, discount, profit)
- postgres_web_sales_datasource.websales_products   (product_id, category, sub_category, product_name)
- postgres_web_sales_datasource.websales_customers  (customer_id, customer_name, segment, country, city, state, postal_code, region)

# JOINS

- websales_sales.product_id  -> websales_products.product_id
- websales_sales.order_id    -> websales_orders.order_id
- websales_sales.customer_id -> websales_customers.customer_id

# GUIDELINES

When answering questions, follow these guidelines:
    For questions about database tables and their contents:
    - Use the sql_db_query to query the tables directly
    - You can join tables if needed to get comprehensive information
    - **Important Rule for SQL Queries:** If you formulate an SQL query as part of answering a user's question,
    you *must* then use the `sql_db_query` tool to execute that query and get its results.
    The SQL query string itself is NOT the final answer to the user unless the user has specifically asked for the query.
    Your final AI response should be based on the *results* obtained from executing the query.
    For factual questions, ALWAYS use the available tools to look up information rather than relying on your internal knowledge.
"""

Lets see what we defined in the above prompt:

Section	Purpose
ROLE AND TASK	Defines the AI's persona, core responsibilities, and provides URL-encoded QuickChart.io examples for the AI to learn the format
SCHEMA	Documents available tables and their columns
JOINS	Explains table relationships for multi-table queries
GUIDELINES	Highlights the importance of following the execute-then-answer pattern

None of these sections in the prompt are mandatory but they would help the AI to better understand your data and what you are trying to achieve.

Why QuickChart.io? Minds can generate plots without requiring the user to provide their own plotting API. These plots are currently available in the Minds UI. Because we are using Minds via the Python API, we will use an external service for plots. QuickChart.io generates chart images from URL-encoded JSON configurations, allowing charts to be embedded in Markdown without any client-side JavaScript.

Step 5: Create the Mind

With all components ready, we create the Mind. This registers the AI agent with MindsDB, connecting it to the specified datasource and prompt template:

mind = client.minds.create(
    name=MIND_NAME,
    datasources=DATA_SOURCES,
    parameters={
        'system_prompt': PROMPT_TEMPLATE
    },
    replace=True
)

Setting replace=True results in the new mind overwriting an old one with the same name.

Step 6: Create the OpenAI-Compatible Client

MindsDB exposes Minds through an OpenAI-compatible API. This means you can use the standard OpenAI Python client to interact with your Mind:

oa = OpenAI(api_key=MINDS_API_KEY, base_url=BASE_URL)

Step 7: Define Helper Functions

As you can see in a minute, we will interact with our Mind the same way as we usually interact with a chatbot. But first, we need several helper functions to handle API responses, extract the final answer from the AI's reasoning trace, and render Markdown output.

7.1 Non-Streaming Response Collection

For simple use cases where you don't need real-time output:

def collect_response(question):
    """
    Collect the final response from the Mind.
    """
    r = oa.chat.completions.create(
        model=MIND_NAME,
        messages=[{"role": "user", "content": question}],
        temperature=0.1
    )
    full_response = r.choices[0].message.content or ""

    return full_response

The low temperature=0.1 setting makes responses more deterministic — ideal for data analysis where consistency matters.

7.2 Streaming Response with Reasoning Trace

For a better user experience, streaming shows the AI's thinking process in real-time:

def stream_and_collect_response(question):
    """
    Stream the LLM response, print reasoning trace, but only return the final answer
    """
    print(f"\n=== QUESTION ===\n{question}\n")
    print("=== LLM REASONING TRACE START ===\n")

    completion = oa.chat.completions.create(
        model=MIND_NAME,
        messages=[{"role": "user", "content": question}],
        temperature=0.1,
        stream=True
    )

    full_response = ""

    try:
        for chunk in completion:
            if hasattr(chunk, 'choices') and chunk.choices:
                choice = chunk.choices[0]
                if hasattr(choice, 'delta') and choice.delta:
                    content = getattr(choice.delta, 'content', None)
                    if content is not None:
                        print(content, end='', flush=True)
                        full_response += content
    except Exception as e:
        print(f"\nStreaming error: {e}")
        # Fallback to non-streaming
        r = oa.chat.completions.create(
            model=MIND_NAME,
            messages=[{"role": "user", "content": question}],
            temperature=0.1
        )
        full_response = r.choices[0].message.content or ""
        print(full_response)

    print("\n")
    print("=== LLM REASONING TRACE END ===\n")

    # Extract only the final answer (everything after "I finished executing the SQL query")
    final_answer = extract_final_answer(full_response)

    return final_answer

This function provides transparency into the AI's decision-making process, showing the SQL it generates and executes before presenting the final answer.

7.3 Extract Final Answer

The AI's response includes both reasoning steps and the final answer. This function extracts just the user-facing content:

def extract_final_answer(full_response):
    """
    Extract the final answer from the full response using the end-of-reasoning marker
    """
    end_marker = "I finished executing the SQL query"

    # Find the position of the end-of-reasoning marker
    marker_pos = full_response.find(end_marker)

    if marker_pos != -1:
        # Extract everything after the marker
        final_answer = full_response[marker_pos + len(end_marker):].strip()
        return final_answer
    else:
        # Fallback: if marker not found, return the full response
        return full_response.strip()

The end_marker string acts as a delimiter between the AI's internal reasoning and the polished response meant for end users.

7.4 Render Markdown Output

Finally, we need to display the formatted response with charts rendered:

def render_markdown(response):
    """
    Render the markdown content generated by the Mind.
    """
    try:
        from IPython.display import Markdown, display
        display(Markdown(response))
    except ImportError:
        # Fallback for non-Jupyter environments: simply print the markdown string
        print(response)

In Jupyter environments, this renders charts inline. In other environments, it might print the raw Markdown (including chart URLs).

Step 8: Create the Main Query Interface

We wrap everything into a single, easy-to-use function:

def ask(question):
    """
    Ask a question to the mind, show reasoning trace, and return DataFrame
    """
    response = stream_and_collect_response(question)
    #response = collect_response(question)
    render_markdown(response)

This is the primary interface users will interact with — simply call ask() with a natural language question.

Example Queries and Results

Let's see the system in action with real business questions.

Query 1: Sales by Product Category

ask("What's the total sales revenue by product category?")

    === QUESTION ===
    What's the total sales revenue by product category?

    === LLM REASONING TRACE START ===

    I will now generate the SQL query to answer your question.Here is the generated SQL query along with its execution result:
    Query executed: SELECT wp.category, SUM(s.sales) as total_sales_revenue 
    FROM postgres_web_sales_datasource.websales_sales s 
    JOIN postgres_web_sales_datasource.websales_products wp ON s.product_id = wp.product_id 
    GROUP BY wp.category

    Results: 3 rows x 2 columns

    category                total_sales_revenue
    Hybrid Work Essentials 1525521.9           
      Smart Office Devices 4288440.0           
         Connected Devices 1288424.8           Here are the total sales revenues by product category:

    - **Smart Office Devices:** $4,288,440.00
    - **Hybrid Work Essentials:** $1,525,521.90
    - **Connected Devices:** $1,288,424.80

    Here is a bar chart showing the sales revenue by category:

    ![Sales Revenue by Category](https://quickchart.io/chart?c=%7Btype%3A%27bar%27%2Cdata%3A%7Blabels%3A%5B%27Smart%20Office%20Devices%27%2C%27Hybrid%20Work%20Essentials%27%2C%27Connected%20Devices%27%5D%2Cdatasets%3A%5B%7Blabel%3A%27Total%20Sales%20Revenue%27%2Cdata%3A%5B4288440.0%2C1525521.9%2C1288424.8%5D%7D%5D%7D%7D)

    === LLM REASONING TRACE END ===

Query 2: Revenue by Shipping Mode

ask("What is the total sales revenue for each shipping mode, and which delivery option generates the most revenue?")

=== QUESTION ===
What is the total sales revenue for each shipping mode, and which delivery option generates the most revenue?

=== LLM REASONING TRACE START ===

I will now generate the SQL query to answer your question.Here is the generated SQL query along with its execution result:
Query executed: SELECT wo.ship_mode, SUM(ws.sales) AS total_revenue
FROM postgres_web_sales_datasource.websales_orders wo
JOIN postgres_web_sales_datasource.websales_sales ws ON wo.order_id = ws.order_id
GROUP BY wo.ship_mode
ORDER BY total_revenue DESC

Results: 3 rows x 2 columns

ship_mode         total_revenue
 Premium Express 2407209.8     
    Eco Delivery 2397529.8     
Instant Delivery 2297646.0     Here is the total sales revenue generated by each shipping mode:

| Shipping Mode       | Total Revenue  |
|---------------------|----------------|
| Premium Express     | $2,407,209.80  |
| Eco Delivery        | $2,397,529.80  |
| Instant Delivery    | $2,297,646.00  |

The shipping mode that generates the most revenue is **Premium Express** with a total revenue of $2,407,209.80.

![Revenue by Shipping Mode](https://quickchart.io/chart?c=%7Btype%3A%27bar%27%2Cdata%3A%7Blabels%3A%5B%27Premium%20Express%27%2C%27Eco%20Delivery%27%2C%27Instant%20Delivery%27%5D%2Cdatasets%3A%5B%7Blabel%3A%27Total%20Revenue%27%2Cdata%3A%5B2407209.8%2C2397529.8%2C2297646.0%5D%2CbackgroundColor%3A%5B%27%234BC0C0%27%2C%27%23FFCE56%27%2C%27%23FF6384%27%5D%7D%5D%7D%7D)

=== LLM REASONING TRACE END ===

Query 3: Customer Segment Comparison

ask("How does total sales revenue compare across different customer segments (Startup vs Enterprise vs others)?")

=== QUESTION ===
How does total sales revenue compare across different customer segments (Startup vs Enterprise vs others)?

=== LLM REASONING TRACE START ===

I will now generate the SQL query to answer your question.Here is the generated SQL query along with its execution result:
Query executed: SELECT c.segment, SUM(s.sales) AS total_sales_revenue
FROM postgres_web_sales_datasource.websales_sales s
JOIN postgres_web_sales_datasource.websales_customers c ON s.customer_id = c.customer_id
GROUP BY c.segment

Results: 3 rows x 2 columns

segment        total_sales_revenue
   Enterprise 2137724.0           
Remote Worker 1285370.2           
      Startup 3679287.8           Here's how the total sales revenue compares across different customer segments:

- **Enterprise**: $2,137,724
- **Remote Worker**: $1,285,370.2
- **Startup**: $3,679,287.8

As we can see, the "Startup" segment generates the highest total sales revenue, followed by "Enterprise", and finally "Remote Worker".

Here's a bar chart illustrating the total sales revenue for each customer segment:

![Total Sales Revenue by Segment](https://quickchart.io/chart?c=%7Btype%3A%27bar%27%2Cdata%3A%7Blabels%3A%5B%27Enterprise%27%2C%27Remote%20Worker%27%2C%27Startup%27%5D%2Cdatasets%3A%5B%7Blabel%3A%27Total%20Sales%20Revenue%27%2Cdata%3A%5B2137724%2C1285370.2%2C3679287.8%5D%7D%5D%7D%7D)

=== LLM REASONING TRACE END ===

Summary

In this tutorial, you learned how to:

Set up MindsDB Minds — Create intelligent AI agents connected to your databases and allow users to query data without writing SQL
Design effective prompt templates — Guide the AI with schema information, examples, and behavioral rules
Generate dynamic visualizations — Leverage QuickChart.io for automatic chart generation
Handle streaming responses — Provide real-time feedback and transparency into AI reasoning

Future Steps

To extend this tutorial, consider:

Add more chart types: Expand the prompt template with examples for scatter plots, area charts, and multi-series visualizations
Expand the datasource: Connect additional tables or databases to answer more complex cross-domain questions
Build a web interface: Wrap the ask() function in a Flask or FastAPI application for broader access

Conclusion

By now, you’ve built a complete AI-powered analytics system using MindsDB Minds — one that can translate natural language into SQL, execute queries safely, and return clear insights with automatically generated charts. What once required dashboards, analysts, and BI tools can now be done by simply asking a question.

This approach doesn’t just make analytics easier — it makes insights accessible to anyone, regardless of technical skill. MindsDB Minds turns your database into a conversational interface, unlocking faster decisions, richer exploration, and a more intuitive way to understand your business data.

As you continue exploring, you can extend your Mind with additional datasets, new chart types, and even a web interface to share with your team. This is just the starting point. With MindsDB, you have everything you need to build interactive, intelligent analytics experiences that grow with your data.

If you're ready to take the next step, explore more examples, connect new sources, or begin building your own AI-powered analytics applications- contact our team for a demo.

MindsDB in 2025: From SQL to the Universal AI Data Hub

MindsDB Team — Tue, 30 Dec 2025 00:33:25 +0000

Written by Alejandro Cantu, Senior Product Manager & Martyna Slawinska, Technical Product Manager at MindsDB

In 2025, the challenge of connecting enterprise data to AI shifted from "if" to "how fast." For the MindsDB open-source community, this year was a journey of evolution—moving beyond predictive models to building the essential infrastructure for the Agentic Web.

We began the year with a focus on Agentic AI, believing it would fundamentally reshape how we build and interact with software. Over the last 12 months, we’ve worked alongside our contributors to make that vision accessible to everyone. By sticking to our core philosophy—Connect, Unify, Respond—we have grown into a platform where developers can confidently build self-reasoning agents on top of any data source.

Here is the scorecard for 2025 and a look at how we got here.

2025 By The Numbers

It was a record-breaking year for activity in our repository.

11 Major Monthly Releases (v25.1 to v25.11).
1,500+ Pull Requests merged to main.
37,400+ GitHub Stars (and counting).
800+ Active Contributors driving the ecosystem forward.
500k+ Docker pulls
850k+ pip installs (no mirrors)

1. CONNECT: The Universal Adapter for Agents

The Goal: Make every data source "AI-Ready" instantly.

The biggest shift in 2025 was moving beyond simple database connections to becoming the universal language for AI Agents.

One Interface to Rule Them All: By abstracting 200+ data sources behind a universal SQL-like interface, we removed the need for agents to learn hundreds of different APIs. Whether it's a Postgres database, a Slack channel, or a Salesforce CRM, your agent logic stays simple, portable, and decoupled from the underlying data complexity.
The Big Pivot (MCP Support): In Q1, we re-architected our API layer to support the Model Context Protocol (MCP). This effectively turned MindsDB into a "universal adapter," allowing agents (like Claude Desktop) to plug into any backend data source without custom code.
Google MCP Toolbox (Nov): We collaborated with Google to supercharge their MCP Toolbox, bringing unstructured data support to the broader developer ecosystem.
Deep Integrations: We didn't just add more handlers; we made them smarter. We rolled out verified, agent-ready integrations for major platforms like Snowflake, BigQuery, Salesforce, Oracle, Databricks, PostgreSQL, MySQL, SQL Server, and Gong, enhancing them with advanced metadata extraction so AI agents can autonomously understand and navigate your data schemas. (Plus, we have Jira and Elasticsearch coming soon!)

2. UNIFY: Democratizing RAG

The Goal: Treat unstructured data (PDFs, Docs) just like a SQL table.

In Spring, we solved the "Last Mile" problem of AI: retrieval. We believed you shouldn't need a PhD in vector databases to build a knowledge bot.

Knowledge Bases (April): We officially launched Knowledge Bases, allowing users to ingest documents and query them with SQL-like syntax. This was enabled by a massive Q1 refactor of our storage engine to handle embeddings natively.
Hybrid Search (August): We upgraded the engine to support Hybrid Search, blending keyword accuracy (BM25) with semantic understanding (Vector).
- Result: AI answers that are both factually accurate and contextually rich.
Virtual Metadata Tables (June): We introduced META_HANDLER_INFO and META_COLUMNS, allowing agents to introspect the database structure themselves—a critical step for self-healing agent workflows.

See how this allows you to search unstructured data with SQL precision here.

3. RESPOND: Talking to Your Data

The Goal: A conversational interface for your data.

We wanted to empower agent builders—from expert engineers to "vibe coders"—to interact with data intuitively. Getting answers from your data is easier than ever; you can now use natural language instead of writing complex SQL yourself.

Chat with Your Data (May): We released the Open Source Chat Interface. This major UX overhaul unified structured SQL querying and unstructured document chat into one seamless window, allowing developers to prototype agent interactions instantly.
- Technical Win: This required merging our websocket handling and state management into a unified "Agent" backend.
Agent Skills: Throughout Q3, we refined how Agents handle SQL generation. They can now autonomously decide when to query a database vs. when to search a Knowledge Base, effectively giving your applications a "Natural Language API" for any connected data source.

4. EXPERIENCE: A GUI Built for Builders

The Goal: A seamless, intuitive environment for AI development.

We didn't just upgrade the engine; we overhauled the dashboard. In 2025, the MindsDB GUI evolved into a full-fledged IDE for AI
Refined Workflow: We introduced a richer tab experience with drag-and-drop organization, session persistence, and per-tab storage, ensuring you never lose context when switching between tasks.

Visual Overhaul: From the highly requested Dark Mode to a revamped full-width sidebar, the interface is now cleaner and easier to navigate.
Integrated Resources: We embedded documentation directly into the GUI, so you can find answers without leaving your workspace.
Seamless Onboarding: A new onboarding process makes getting started with MindsDB faster and more intuitive for new users.
Model Management: Enhanced model management screens provide deeper visibility and control, making it easier to oversee your AI models as you scale. You can now easily configure and switch between top-tier providers including O*penAI, Anthropic, Google Gemini,NVIDIA NIM, Ollama, and AWS Bedrock*, giving you the flexibility to choose the right model for the job.
Authentication & Access: We added standalone local login support with a fresh brand identity and streamlined OAuth support for tools like Microsoft Teams and Google Calendar.

The Foundation: Stability, Performance & Security

While features grab headlines, reliability runs production. This year saw the most significant engineering overhauls in our history.

Runtime and Operational Stability (September)

In v25.9, we made significant changes to improve runtime stability and operational reliability.

Impact:

Eliminated orphaned background processes in containerized environments
Reduced memory usage across typical deployments
Simplified setup and operational troubleshooting

Performance: Faster Inference

Smarter SQL: We overhauled planning, pruning, and pushdown; optimized handlers; reduced cold-start penalties; and validated everything with large-scale benchmarks. The result: faster federated SQL, smarter agents, and a cleaner architecture ready for the next step.
Faster Query Execution: Previously, MindsDB retrieved handler information directly from the underlying database – a process that added overhead to every query. We’ve now re-engineered this flow so that handler data is read as lightweight, in-code metadata. This architectural shift eliminates unnecessary I/O and streamlines execution. See results below.
Improved Startup and Shutdown Times: Historically, each new process spent 5+ seconds of CPU time initializing the MindsDB SQL parser, because a set of complex parsing tables was being regenerated on every startup. We have mitigated this by adopting and integrating smart caching capabilities.
Enhanced Logging and Configurability: We improved the clarity, structure, and consistency of log output, making it easier for users to diagnose issues and monitor system behavior. In addition, logging levels are now fully configurable via the MindsDB configuration file.

Security: Enterprise-Grade Trust

Trust is paramount. Throughout 2025, we hardened the platform to meet the needs of our growing enterprise user base

SOC2 Certification: We officially achieved SOC2 compliance, validating our commitment to the highest standards of data security and operational governance. You can verify our security posture in real-time at our new Trust Center.
Proactive Patching: We stayed ahead of the curve, patching critical CVEs (e.g., CVE-2024-45853) and updating dependencies like Werkzeug and Ray before they became risks.
Secret Management: We overhauled how environment secrets and SSL verifications are handled, ensuring that your connection to enterprise data sources remains hermetically sealed.
Proactive Patching: We stayed ahead of the curve, patching critical CVEs (e.g., CVE-2024-45853) and updating dependencies like Werkzeug and Ray before they became risks.
Secret Management: We overhauled how environment secrets and SSL verifications are handled, ensuring that your connection to enterprise data sources remains hermetically sealed.

Ready for 2026

2025 was the year MindsDB matured from a tool into a platform. Whether you’re building customer support agents, financial analysis bots, or just querying your database in natural language, the foundation is stronger than ever.

Thank you to our community, our contributors, and our users for building with us. See you in 2026!

MySQL & MindsDB Unlocks Intelligent Content Discovery For Web CMS with Knowledge Bases and Cursor

MindsDB Team — Tue, 30 Dec 2025 00:08:01 +0000

Written by Chandre Van Der Westhuizen, Community & Marketing Co-ordinator at MindsDB

Content teams run on data — not just page views, but topic trends, engagement behavior, author performance, and semantic relationships across hundreds or thousands of articles. And most of this content lives in MySQL databases powering CMS platforms like WordPress, Ghost, Webflow, Strapi, or custom-built systems.

But while MySQL gives you structured data (titles, metadata, authors, timestamps), it wasn’t designed to answer questions like:

“Which articles discuss AI trends and have high engagement?”
“Which authors consistently publish top-performing content?”
“Show me content with a high bounce rate but low click-through rate.”
“How many blog posts relate semantically to ‘cloud security’?”

These questions require semantic understanding + structured analytics, something that traditionally requires multiple systems: ETL jobs, a separate vector database, a search index, and various analytics tools.

MindsDB removes all of that overhead by allowing MySQL data to be queried, enriched, and analyzed with AI directly — using Knowledge Bases, Hybrid Search, and the MCP Server with Cursor for natural-language access.

The Insights Content Marketing Teams Don't Have Access To

content-driven companies—from media publishers to SaaS documentation teams—run on data stored inside MySQL-based Web CMS systems. Titles, slugs, authors, tags, publish dates, engagement metrics—everything lives neatly in tables.

But the actual content—the article bodies, guides, announcements, long-form pages—lives as unstructured text. When you need to search across hundreds or thousands of pieces, traditional CMS search breaks down:

Keyword search only catches exact matches
Metadata queries can’t capture semantic meaning
Editors waste time digging for articles, reusing content manually, or re-creating pieces that already exist
AI-driven tools struggle because the CMS provides no semantic understanding

This is where MindsDB changes the game.

By connecting your MySQL CMS directly to MindsDB, you can build AI-native search, insights, and content automation—without ETL pipelines, custom vector stores, or rewriting your CMS stack.

MindsDB Bridges The Gap for Content Teams Working in MySQL

For teams managing large volumes of CMS data inside MySQL, MindsDB unlocks capabilities that traditional SQL alone cannot provide:

AI-powered insight directly on top of your existing MySQL tables — no migrations or ETL.
Hybrid Search that blends semantic meaning with precise SQL filters for richer content discovery.
Knowledge Bases that turn unstructured CMS text into searchable, analyzable AI-ready data.
Natural-language querying via MindsDB’s MCP Server + Cursor, enabling editors and analysts to “ask” questions instead of writing complex SQL.
Smarter decision-making around content performance, SEO, user behavior, and trends — all while keeping MySQL as your source of truth.

Now let’s explore how you can build an intelligent AI layer on top of your Web CMS data.

Elevate Your MySQL CMS- Unlock AI-Powered Content Intelligence with MindsDB

Now that we’ve explored why AI unlocks deeper insight from web-based content systems, let’s walk through how to actually build this inside MindsDB. In the next steps, you’ll learn how to take CMS data already stored in MySQL—titles, slugs, tags, authors, publish dates, full text—and convert it into powerful AI-ready Knowledge Bases.

To demonstrate this, we will make use of a sample Web CMS dataset hosted in MySQL.

*Pre-requisites: *

Access MindsDB’s GUI via Docker locally or MindsDB’s extension on Docker Desktop.
Configure your default models in the MindsDB GUI by navigating to Settings → Models.
Navigate to Manage Integrations in Settings and install the dependencies for MariaDB.

After the dependencies have been successfully installed, you can connect your MySQL data to MindsDB.

You can establish a connection with MySQL using the CREATE DATABASE syntax in MindsDB’s SQL Editor:

CREATE DATABASE mysql
WITH ENGINE = 'mysql', 
PARAMETERS = {
    "host": "samples.mindsdb.com",
    "port": 3306,
    "database": "public",
    "user": "user",
    "password": "MindsDBUser123!"
};

This connection will give you access to the Web CMS data and CMS Performance Metrics data.

How MindsDB Unifies MySQL Web CMS Data with Knowledge Bases

MindsDB Knowledge Bases turn your existing CMS data into an AI-ready search and reasoning layer. Instead of treating content as plain text inside tables, Knowledge Bases enrich it with metadata, embeddings, and hybrid search capabilities. This means your articles, pages, logs, and performance metrics become semantically searchable, instantly retrievable, and usable by AI agents without moving or duplicating data.

We will make use of:

The Web CMS data that stores information about blog content
The CMS Performance Metrics which stores information about the blogs’ performances.

To start, we will create a Knowledge Base using the CREATE KNOWLEDGE_BASE statement:

CREATE KNOWLEDGE_BASE web_cms_kb
USING
    storage = heroku.cms,
    metadata_columns = ['slug', 'author', 'created_at', 'updated_at', 'tags' ],
    content_columns = ['content'],
    id_column = 'title';

Here are the parameters provided:

web_cms_kb : The name of the knowledge base.
storage : The storage table where the embeddings of the knowledge base is stored. As you can see we are using the PGVector database we created a connection with and provide the name orders to the table that will be created for storage.
metadata_columns : Here columns are provided as meta data columns to perform metadata filtering.
content_columns : Here columns are provided for semantic search.
id_column: This uniquely identifies each source data row in the knowledge base.

You can insert the data using the INSERT INTO statement:

INSERT INTO web_cms_kb
SELECT slug, author, created_at, updated_at, tags, content, title
FROM mysql.real_cms_data;

Once inserted, you can select the data in the Knowledge Bases using the SELECT syntax:

SELECT * FROM web_cms_kb;

You can follow the same instructions to create a Knowledge Base for the Performance Metrics data:

-- Create Knowledge Base
CREATE KNOWLEDGE_BASE cms_performances_kb
USING
    storage = heroku.cms_performances_kb,
    metadata_columns = ['views', 'clicks', 'ctr', 'avg_time_seconds', 'bounce_rate', 'date'],
    content_columns = ['cms_id'],
    id_column = 'title';

--Insert data into Knowledge Base
INSERT INTO cms_performance_kb
SELECT title, `views`, clicks, ctr, avg_time_seconds, bounce_rate, `date`, cms_id
FROM mysql.cms_6month_performance_metrics;

-- Select Knowledge Base
SELECT * FROM cms_performance_kb;

Now that the Knowledge Bases are created, we can perform Hybrid Search to gain some insights.

Hybrid Search: The Missing Link Between Structured MySQL Data and AI Understanding

Hybrid Search in MindsDB brings together the best of semantic search and traditional filtering so teams can find the right content—not just content that contains matching keywords.

For MySQL-backed CMS systems, this means your content finally becomes discoverable in the way people actually search—using natural language, intent, and structured filters together.

CMS teams quickly understand how much of their content relates to a specific topic—guiding editorial planning, SEO focus, and audience targeting.

--Identify how many topics relate to a certain subject
SELECT * FROM web_cms_kb
WHERE content = 'food' AND
hybrid_search_alpha = 0.8;

It finds all CMS articles whose content semantically relates to the subject “food”, using hybrid search to surface both direct matches and deeper contextual connections. You can identify if this topic has been explored yet.

This query returns all blog posts where the associated author consistently generates high-view content.

--Identify authors whose blogs bring in high views
SELECT * FROM cms_performance_kb
JOIN web_cms_kb
ON cms_performance_kb.id = web_cms_kb.id
WHERE views >= 7000
AND cms_performance_kb.hybrid_search_alpha = 0.5
AND web_cms_kb.hybrid_search_alpha = 0.5;

CMS teams can quickly identify top-performing authors, understand what drives engagement, and prioritize creators or topics that meaningfully grow traffic.

This query finds all finance-tagged blog posts that not only attract a high number of clicks but also keep readers engaged for at least 3 minutes (200 seconds).

--Identify blogs with a specific tag that have high click rate with a average reading time of 3minutes
SELECT * FROM cms_performance_kb
JOIN web_cms_kb
ON cms_performance_kb.id = web_cms_kb.id
WHERE tags = 'finance'
AND clicks >=4000
AND avg_time_seconds >=200
AND cms_performance_kb.hybrid_search_alpha = 0.5
AND web_cms_kb.hybrid_search_alpha = 0.5;

This helps teams identify which finance-related content performs best so they can replicate or expand on successful topics.

This query identifies blog posts that lose readers quickly — high bounce rate, low click-through rate, and very short time-on-page.

--Identify blogs which has a high bounce rate and a low CTR
SELECT * FROM cms_performance_kb
JOIN web_cms_kb
ON cms_performance_kb.id = web_cms_kb.id
WHERE
bounce_rate >= 0.7
AND ctr <= 0.4
AND avg_time_seconds <= 30
AND cms_performance_kb.hybrid_search_alpha = 0.5
AND web_cms_kb.hybrid_search_alpha = 0.5;

It alerts content teams to pages that may need rewriting, updated SEO, better UX, or clearer topic alignment to improve performance.

Hybrid search gives CMS teams a powerful new lens for understanding their content — blending semantic meaning with structured performance data to answer questions that were previously tedious or impossible to query in SQL alone. Once this intelligence is available inside MindsDB’s Knowledge Bases, the next step is making it accessible anywhere your team works.

That’s where MindsDB’s MCP Server and Cursor's MCP Client comes in.

In the next section, we’ll show how to expose these same hybrid-search insights to natural-language queries inside Cursor — allowing developers, editors, and analysts to interact with your MySQL-backed CMS data as effortlessly as chatting with an AI agent.

From SQL to Natural Language: Using MindsDB’s MCP Server + Cursor with Your Web CMS Data in MySQL

With MindsDB’s MCP Server and Cursor’s MCP Client, your MySQL-based CMS data becomes instantly accessible through natural-language queries. Instead of writing SQL or building custom endpoints, editors, analysts, and engineers can simply ask questions—“Which articles trended last month?” or “Show posts tagged ‘AI’ with high engagement”—and get real, grounded results directly from your database. This bridges the gap between technical data access and everyday content decision-making, making AI-powered insights available to anyone on your team.

To begin, make sure you have used the below command to start the MCP server with Cursor:

docker run --name mindsdb_container -p 47334:47334 -p 47335:47335 mindsdb/mindsdb

Once this is up and running, you can access Cursor and follow the below instructions:

Navigate to Cursor’s Settings and open the MCP tab.
Select Tools and MCP
Select Add Custom MCP. It will automatically open a tab to the mcp.json file. Add the following details and save the changes:

{
    "mcpServers": {
      "mindsdb": {
        "url": "http://127.0.0.1:47334/mcp/sse"
      }
    }
}

4.Navigate back to the Cursor Settings tab and you will see MindsDB is listed under ‘Installed MCP Servers’.
5.Select ‘Toggle AI Pane’ and make sure the mode is set to Agent with the LLM model on Auto.

You can now start chatting with your data. Ask the Agent to access your data in the MySQL database:

1. Which authors produce the most engaging content over time?

This identifies high-performing creators, helps allocate editorial resources, and informs hiring, content planning, and promotion decisions. Cursor provides the top 5 authors with their metrics,as well as detailed overall insights.

2. Which content topics or tags drive the most traffic and engagement?

This flags content that ranks or gets traffic but fails to convert — the clearest signal for SEO optimization, rewriting, or improving titles, snippets, or structure.

3.Which articles have a high bounce rate and low engagement?

Cursor provides 8 articles with performance metrics and key patterns below.

This identifies pages harming user experience and SEO quality signals; these are high-priority candidates for revision, redesign, or content replacement.

4.Which articles are trending upward or downward in performance?

Here we can see early signals of growth or decline; essential for capitalizing on rising content and repairing slipping performers before they lose rankings.

6.Which tags or topics are underrepresented but performing strongly?

Cursor provides an overall comparison for the averages, recommendations and a summary:

This helps teams discover high-potential opportunities — underserved content themes that deliver strong engagement and should be expanded into content clusters.

7. Which titles have strong SEO indicators based on long-term CTR trends?

Here we get a list of articles with a summary below.

Titles with consistently strong CTR trends signal which content performs well in search over time, helping teams focus SEO efforts where it drives the most impact.

Example Use Cases Powered by MindsDB + MySQL + MCP

1. SEO Insights Assistant

Ask: “What topics performed best in October by traffic?”
How it works: MindsDB blends SQL metrics with hybrid retrieval to evaluate topic performance using both structured (views, CTR) and unstructured (content themes) data.

2. Editorial Planning Assistant

Ask: “Show me all drafts tagged ‘sustainability’ that are over 1,000 words.”
How it works: MySQL filters structured fields while Knowledge Bases surface semantic matches, helping editors plan upcoming content efficiently.

3. Content Quality & Redundancy Auditor

Ask: “Find duplicate or highly similar articles across the blog.”
How it works: Hybrid search compares article embeddings, making it easy to detect overlap, outdated pages, or pieces needing consolidation.

4. Author Performance Dashboard

Ask: “Which authors published fewer than 3 posts last month?”
How it works: MindsDB queries author output trends, enabling managers to spot gaps in coverage and support team-wide performance insights.

5. Content Gap & Opportunity Analyzer

Ask: “Which trending topics from external sources are missing on our blog?”
How it works: Combine CMS data with external knowledge sources via MCP, revealing blind spots and high-value content opportunities.

Why This Matters (and What You Gain)

Content teams rely on MySQL to store massive volumes of articles, metadata, and performance metrics — but turning that raw data into actionable insight usually requires dashboards, manual queries, or complex BI setups. MindsDB removes these barriers by letting teams search semantically, analyze trends instantly, and ask natural-language questions directly against their CMS data.

With Knowledge Bases, Hybrid Search, and MCP + Cursor:

You unlock deeper content intelligence that isn’t possible with SQL alone.
You discover patterns across topics, engagement, and SEO performance without manual digging.
You give editors, marketers, and SEO analysts AI-powered visibility into what drives results.
You connect structured (MySQL) and unstructured (content) data, enabling decisions grounded in both narrative and numbers.
You eliminate ETL and extra infrastructure, using the MySQL data you already have.
You future-proof your CMS workflow by enabling AI-native querying, automation, and discovery.

MindsDB turns your MySQL-based CMS into a real-time, AI-driven content intelligence engine — without changing where your data lives.

Conclusion

MySQL has always been the backbone of modern Web CMS platforms—reliable, structured, and central to how content teams store, organize, and publish information. But as content ecosystems grow more complex and audiences expect higher-quality, highly relevant material, traditional SQL alone isn’t enough to surface the insights teams need.

MindsDB transforms this entire landscape.

By pairing your existing MySQL data with Knowledge Bases, Hybrid Search, and MCP Server + Cursor, MindsDB turns ordinary CMS tables into an intelligent content engine—one capable of understanding semantic meaning, revealing performance patterns, and answering nuanced editorial questions in natural language.

Most importantly, you achieve all of this without moving your data, rewriting your CMS, or introducing new databases. MySQL stays your source of truth—MindsDB simply makes it smarter.

As you move forward, the combination of MySQL + MindsDB provides a scalable, future-ready foundation for content discovery, analytics, and decision-making. Whether you're running a global media operation, a SaaS documentation hub, or a content-driven startup, MindsDB helps your teams work faster, learn more, and create better content—all powered by the data you already own. If you would like to see MindsDB in action, contact our team.

Your CMS just became intelligent.

Blend Hybrid Retrieval with Structured Data using MindsDB Knowledge Bases

MindsDB Team — Mon, 29 Dec 2025 23:17:24 +0000

Written by Andriy Burkov, Ph.D. & Author, MindsDB Advisor

This tutorial is a follow-up to this tutorial, where we took the first steps in creating and using a MindsDB Knowledge Base feature. In this follow-up project, we will walk through creating a semantic search knowledge base using the famous Enron Emails Dataset. While in the previous tutorial, we simply used an existing dataset, in this one, we'll preprocess the original dataset by extracting structured attributes (also known as metadata) from it using Named Entity Recognition (NER). We will then create a knowledge base and perform both semantic and metadata-filtered searches.

Before we get our hands dirty, let's refresh some basics. Download the webinar code and materials here to follow along the tutorial.

1. Introduction to Knowledge Bases in MindsDB

Knowledge Bases (KB) in MindsDB provide advanced semantic search capabilities, allowing you to find information based on meaning rather than just keywords. They use embedding models to convert text into vector representations and store them in vector databases for efficient similarity searches.

In addition to searching for knowledge nuggets using semantic similarity (soft search criteria), MindsDB KBs allow the user to combine both soft search criteria with hard ones called "metadata," which can be seen as regular relational database table columns.

In this tutorial, we assume that the user has a free open-source MindsDB instance running in their local environment. Please follow these steps to set it up.

To demonstrate both soft and hard searches in a MindsDB KB, we'll use the Enron Corpus - one of the largest publicly available collections of corporate emails, containing over 500,000 emails from Enron executives during the years leading up to the company's collapse in 2001. This dataset is particularly interesting because it contains real business communications, including scandal-related content, making it perfect for demonstrating knowledge base search capabilities.

Named Entity Recognition is the technique we'll use to automatically extract those structured attributes—such as people, organizations, dates, and locations—from the raw email text. These extracted entities will become the metadata columns in our knowledge base, allowing us not only to search semantically by meaning, but also to filter results using precise, structured criteria like sender, company, or time period.

2. Settings Things Up

2.1 Dependencies Installation

First, let's install the dependencies and set up the NER. We will use SpaCy for this, since its pretrained models can automatically extract entities like people, organizations, dates, and locations from the raw email text. Those extracted entities will then be transformed into structured metadata columns, which we’ll store alongside the email content and later use to power rich, metadata-aware queries in our MindsDB knowledge base.

!pip install mindsdb mindsdb_sdk pandas requests datasets yaspin spacy

# Download spaCy English model for Named Entity Recognition
!python -m spacy download en_core_web_sm

print("✅ Dependencies installed successfully!")

2.2 Dataset Selection and Download

We'll will download the Enron email's dataset from Hugging Face, which is a large collection of real-world corporate messages from the Enron corpus, paired with their original subject lines and cleaned body text. Each entry includes the email’s metadata (such as sender, recipients, and timestamp) along with the full message content, organized into standard train/validation/test splits so the dataset be uses for tasks like summarization, classification, or downstream NLP experiments.

For our tutorial purposes, we will only use the train fraction of the dataset.

# Download Enron Emails Dataset from Hugging Face
from datasets import load_dataset
import pandas as pd
import re
from datetime import datetime
import json

# Load the Enron dataset (536k emails)
print("Downloading Enron emails dataset...")
dataset = load_dataset("snoop2head/enron_aeslc_emails", split="train")
df = pd.DataFrame(dataset)

print(f"Dataset shape: {df.shape}")
print("Dataset columns:", df.columns.tolist())

def parse_email_text(email_text):
    """Parse raw email text to extract subject, body, and metadata"""
    if pd.isna(email_text) or email_text == '':
        return {'subject': '', 'body': '', 'from': '', 'to': '', 'date': ''}

    email_text = str(email_text)

    # Initialize result dictionary
    parsed = {'subject': '', 'body': '', 'from': '', 'to': '', 'date': ''}

    # Extract subject
    subject_match = re.search(r'Subject:\s*(.*?)(?:\n|$)', email_text, re.IGNORECASE)
    if subject_match:
        parsed['subject'] = subject_match.group(1).strip()

    # Extract from
    from_match = re.search(r'From:\s*(.*?)(?:\n|$)', email_text, re.IGNORECASE)
    if from_match:
        parsed['from'] = from_match.group(1).strip()

    # Extract to
    to_match = re.search(r'To:\s*(.*?)(?:\n|$)', email_text, re.IGNORECASE)
    if to_match:
        parsed['to'] = to_match.group(1).strip()

    # Extract date
    date_match = re.search(r'Date:\s*(.*?)(?:\n|$)', email_text, re.IGNORECASE)
    if date_match:
        parsed['date'] = date_match.group(1).strip()

    # Extract body (everything after the headers)
    # Look for the end of headers (usually marked by double newline or start of actual content)
    header_end = re.search(r'\n\s*\n', email_text)
    if header_end:
        parsed['body'] = email_text[header_end.end():].strip()
    else:
        # Fallback: try to find content after common header patterns
        body_start = re.search(r'(?:Subject:.*?\n.*?\n|X-.*?\n)', email_text, re.DOTALL)
        if body_start:
            parsed['body'] = email_text[body_start.end():].strip()
        else:
            parsed['body'] = email_text

    # Clean up body text
    parsed['body'] = re.sub(r'\n+', ' ', parsed['body'])
    parsed['body'] = re.sub(r'\s+', ' ', parsed['body'])

    return parsed

# Parse first few emails to understand structure
print("Parsing email structure...")
sample_emails = df.head(10)

parsed_samples = []
for idx, row in sample_emails.iterrows():
    # The dataset might have different column names, let's check
    email_content = None
    for col in df.columns:
        if row[col] and len(str(row[col])) > 100:  # Find the column with email content
            email_content = row[col]
            break

    if email_content:
        parsed = parse_email_text(email_content)
        parsed['email_id'] = f"email_{idx:06d}"
        parsed_samples.append(parsed)

    Downloading Enron emails dataset...
    Dataset shape: (535703, 1)
    Dataset columns: ['text']
    Parsing email structure...

Now let's print some records to see what's inside:

df_parsed_sample = pd.DataFrame(parsed_samples)
print("\nSample of parsed emails:")
print("="*100)

for idx, row in df_parsed_sample.head(5).iterrows():
    print(f"\nEmail #{idx+1}")
    print(f"ID: {row['email_id']}")
    print(f"From: {row['from'][:80]}{'...' if len(row['from']) > 80 else ''}")
    print(f"To: {row['to'][:80]}{'...' if len(row['to']) > 80 else ''}")
    print(f"Date: {row['date']}")
    print(f"Subject: {row['subject']}")
    print(f"Body Preview: {row['body'][:200]}{'...' if len(row['body']) > 200 else ''}")
    print("-" * 80)

print(f"\nSuccessfully parsed email structure!")
print(f"Columns extracted: {df_parsed_sample.columns.tolist()}")

Sample of parsed emails:
====================================================================================================

Email #1
ID: email_000000
From: phillip.allen@enron.com
To: tim.belden@enron.com
Date: Mon, 14 May 2001 16:39:00 -0700 (PDT)
Subject: Body:
Body Preview: Here is our forecast
--------------------------------------------------------------------------------

Email #2
ID: email_000001
From: phillip.allen@enron.com
To: john.lavorato@enron.com
Date: Fri, 4 May 2001 13:51:00 -0700 (PDT)
Subject: Re:
Body Preview: Traveling to have a business meeting takes the fun out of the trip. Especially if you have to prepare a presentation. I would suggest holding the business plan meetings here then take a trip without a...
--------------------------------------------------------------------------------

Email #3
ID: email_000002
From: phillip.allen@enron.com
To: leah.arsdall@enron.com
Date: Wed, 18 Oct 2000 03:00:00 -0700 (PDT)
Subject: Re: test
Body Preview: test successful. way to go!!!
--------------------------------------------------------------------------------

Email #4
ID: email_000003
From: phillip.allen@enron.com
To: randall.gay@enron.com
Date: Mon, 23 Oct 2000 06:13:00 -0700 (PDT)
Subject: Body:
Body Preview: Randy, Can you send me a schedule of the salary and level of everyone in the scheduling group. Plus your thoughts on any changes that need to be made. (Patti S for example) Phillip
--------------------------------------------------------------------------------

Email #5
ID: email_000004
From: phillip.allen@enron.com
To: greg.piper@enron.com
Date: Thu, 31 Aug 2000 05:07:00 -0700 (PDT)
Subject: Re: Hello
Body Preview: Let's shoot for Tuesday at 11:45.
--------------------------------------------------------------------------------

Successfully parsed email structure!
Columns extracted: ['subject', 'body', 'from', 'to', 'date', 'email_id']

The data looks good, so now, let's load SpaCy and its pretrained NLP model for English:

import spacy
from spacy import displacy
# Load spaCy model for NER
print("Loading spaCy model for Named Entity Recognition...")
nlp = spacy.load("en_core_web_sm")
try:
    nlp = spacy.load("en_core_web_sm")
    print("spaCy model 'en_core_web_sm' loaded successfully!")
except OSError as e:
    print("Failed to load spaCy model:", e)

Loading spaCy model for Named Entity Recognition...
spaCy model 'en_core_web_sm' loaded successfully!

2.3 Preparing the Dataset for the Knowledge Base

So far, we have got a raw collection of email messages, but we need a dataset to created a knowledge base from. In this dataset, we want to have natural language texts for soft semantic search and named attributes for hard filtering of data rows.

The first step in preparing a dataset for a KB is cleaning it up and making sure we have a unique ID column. The second step is extracting the named entities from the cleaned records. We will perform both steps in the below cell:

MAX_EMAILS_TO_PROCESS = 500_000
MIN_BODY_SIZE_CHARS = 50

import json
from tqdm import tqdm

def clean_email_content(text):
    """Clean and prepare email text for processing"""
    if pd.isna(text) or text == '':
        return ""

    # Convert to string and clean
    text = str(text)
    # Remove excessive whitespace and newlines
    text = re.sub(r'\n+', ' ', text)
    text = re.sub(r'\s+', ' ', text)
    # Remove common email artifacts
    text = re.sub(r'-----Original Message-----.*$', '', text, flags=re.MULTILINE | re.DOTALL)
    text = re.sub(r'________________________________.*$', '', text, flags=re.MULTILINE | re.DOTALL)

    return text.strip()

def extract_entities_with_ner(text, nlp_model):
    """Extract named entities using spaCy NER"""
    if not text or len(text.strip()) == 0:
        return {}

    # Limit text length to avoid memory issues
    text = text[:5000] if len(text) > 5000 else text

    try:
        doc = nlp_model(text)
        entities = {
            'persons': [],
            'organizations': [],
            'locations': [],
            'money': [],
            'dates': [],
            'events': [],
            'products': []
        }

        for ent in doc.ents:
            entity_text = ent.text.strip()
            if len(entity_text) < 2:  # Skip very short entities
                continue

            if ent.label_ == "PERSON":
                entities['persons'].append(entity_text)
            elif ent.label_ == "ORG":
                entities['organizations'].append(entity_text)
            elif ent.label_ in ["GPE", "LOC"]:  # Geopolitical entities, locations
                entities['locations'].append(entity_text)
            elif ent.label_ == "MONEY":
                entities['money'].append(entity_text)
            elif ent.label_ == "DATE":
                entities['dates'].append(entity_text)
            elif ent.label_ == "EVENT":
                entities['events'].append(entity_text)
            elif ent.label_ == "PRODUCT":
                entities['products'].append(entity_text)

        # Remove duplicates and limit to top entities
        for key in entities:
            entities[key] = list(set(entities[key]))[:5]  # Max 5 entities per type

        return entities
    except Exception as e:
        print(f"Error processing text: {e}")
        return {}

# Use the real Enron dataset that was loaded earlier
print(f"📧 Working with real Enron dataset: {df.shape[0]} emails")
print(f"Dataset columns: {df.columns.tolist()}")

# Sample a reasonable subset for this tutorial (full dataset is very large)
print("Sampling real Enron emails for processing...")
df_sample = df.sample(n=MAX_EMAILS_TO_PROCESS, random_state=42).reset_index(drop=True)

# Process the real emails and extract entities
print("Processing real Enron emails and extracting entities...")
processed_emails = []

for idx, row in tqdm(df_sample.iterrows(), total=len(df_sample), desc="Processing emails"):
    # Get the raw email content from 'text' column
    email_content = row['text']

    if not email_content or len(str(email_content)) < 100:
        continue

    # Parse the email using the function from Cell 3
    parsed = parse_email_text(email_content)

    # Clean the content
    subject = clean_email_content(parsed['subject'])
    body = clean_email_content(parsed['body'])

    if len(body) < MIN_BODY_SIZE_CHARS:  # Skip very short emails
        continue

    # Extract entities from both subject and content
    full_text = f"{subject} {body}"
    entities = extract_entities_with_ner(full_text, nlp)

    # Create email ID
    email_id = f"email_{idx:06d}"

    processed_email = {
        'email_id': email_id,
        'from_address': parsed['from'][:100] if parsed['from'] else '',  # Limit length
        'to_address': parsed['to'][:100] if parsed['to'] else '',        # Limit length  
        'date_sent': parsed['date'][:50] if parsed['date'] else '',      # Limit length
        'subject': subject,
        'content': body,
        'persons': ', '.join(entities['persons']) if entities['persons'] else '',
        'organizations': ', '.join(entities['organizations']) if entities['organizations'] else '',
        'locations': ', '.join(entities['locations']) if entities['locations'] else '',
        'money_amounts': ', '.join(entities['money']) if entities['money'] else '',
        'dates_mentioned': ', '.join(entities['dates']) if entities['dates'] else '',
        'events': ', '.join(entities['events']) if entities['events'] else '',
        'products': ', '.join(entities['products']) if entities['products'] else '',
        'content_length': len(body),
        'entity_count': sum(len(v) for v in entities.values())
    }

    processed_emails.append(processed_email)

# Convert to DataFrame
df_processed = pd.DataFrame(processed_emails)

print(f"\n✅ Processed {len(df_processed)} real Enron emails with entities extracted")

📧 Working with real Enron dataset: 535703 emails
Dataset columns: ['text']
Sampling real Enron emails for processing...
Processing real Enron emails and extracting entities...


Processing emails: 100%|███████████████████████████████████████████████| 500000/500000 [2:55:44<00:00, 47.42it/s]



✅ Processed 453905 real Enron emails with entities extracted

We now have a Pandas dataframe containing, for each email, its text and the extracted attributes. Let's look at some of them and some stats:

# Show a sample of processed data
print(f"\n📊 Sample of Enron emails processed:")
print("="*120)

for idx, row in df_processed.head(5).iterrows():
    print(f"\n📧 Real Email #{idx+1}")
    print(f"🆔 ID: {row['email_id']}")
    print(f"👤 From: {row['from_address']}")
    print(f"👤 To: {row['to_address']}")
    print(f"📅 Date: {row['date_sent']}")
    print(f"📝 Subject: {row['subject']}")
    print(f"👥 Persons: {row['persons'] if row['persons'] else 'None detected'}")
    print(f"🏢 Organizations: {row['organizations'] if row['organizations'] else 'None detected'}")
    print(f"📍 Locations: {row['locations'] if row['locations'] else 'None detected'}")
    print(f"💰 Money: {row['money_amounts'] if row['money_amounts'] else 'None detected'}")
    print(f"💬 Content Preview: {row['content'][:200]}{'...' if len(row['content']) > 200 else ''}")
    print("-" * 100)

# Show statistics on real data
print(f"\n📈 Real Data Processing Statistics:")
print(f"• Total real emails processed: {len(df_processed)}")
print(f"• Average content length: {df_processed['content_length'].mean():.0f} characters")
print(f"• Average entities per email: {df_processed['entity_count'].mean():.1f}")
print(f"• Emails with persons mentioned: {len(df_processed[df_processed['persons'] != ''])}")
print(f"• Emails with organizations mentioned: {len(df_processed[df_processed['organizations'] != ''])}")
print(f"• Emails with money amounts: {len(df_processed[df_processed['money_amounts'] != ''])}")

# Show some interesting real examples
print(f"\n🔍 Most interesting real emails (by entity count):")
top_emails = df_processed.nlargest(3, 'entity_count')
for idx, row in top_emails.iterrows():
    print(f"\n📧 High-entity email from {row['from_address']}")
    print(f"📝 Subject: {row['subject']}")
    print(f"👥 Persons: {row['persons']}")
    print(f"🏢 Organizations: {row['organizations']}")
    print(f"💰 Money: {row['money_amounts']}")

📊 Sample of Enron emails processed:
========================================================================================================================

📧 Real Email #1
🆔 ID: email_000000
👤 From: daren.farmer@enron.com
👤 To: susan.trevino@enron.com
📅 Date: Fri, 10 Dec 1999 08:33:00 -0800 (PST)
📝 Subject: Re: Meter 5892 - UA4 1996 and 1997 Logistics Issues
👥 Persons: Daren J Farmer/HOU, Susan, Meter 5892 - UA4 1996, Mary M Smith/HOU, Susan D Trevino
🏢 Organizations: Volume Management
📍 Locations: UA4
💰 Money: None detected
💬 Content Preview: Susan, I need you to do the research on this meter. You will need to review the various scheduling systems to see how this was handled prior to 2/96. You can also check with Volume Management to see i...
----------------------------------------------------------------------------------------------------

📧 Real Email #2
🆔 ID: email_000001
👤 From: eric.bass@enron.com
👤 To: jason.bass2@compaq.com, phillip.love@enron.com, bryan.hull@enron.com,
📅 Date: Fri, 18 Aug 2000 05:03:00 -0700 (PDT)
📝 Subject: DRAFT
👥 Persons: Bcc
🏢 Organizations: None detected
📍 Locations: Rice Village
💰 Money: None detected
💬 Content Preview: Cc: timothy.blanchard@enron.com Bcc: timothy.blanchard@enron.com Remember, the draft is this Sunday at 11:45 am at BW-3 in Rice Village. Please try to be there on time so we can start promptly. -Eric
----------------------------------------------------------------------------------------------------

📧 Real Email #3
🆔 ID: email_000003
👤 From: larry.campbell@enron.com
👤 To: pdrumm@csc.com
📅 Date: Mon, 31 Jul 2000 09:53:00 -0700 (PDT)
📝 Subject: More July CED-PGE
👥 Persons: Susan Fick, Patty
🏢 Organizations: None detected
📍 Locations: None detected
💰 Money: None detected
💬 Content Preview: Patty Could you please forward this to Susan Fick. I don't have her e-mail. LC
----------------------------------------------------------------------------------------------------

📧 Real Email #4
🆔 ID: email_000004
👤 From: phillip.allen@enron.com
👤 To: christi.nicolay@enron.com, james.steffes@enron.com, jeff.dasovich@enron.com,
📅 Date: Wed, 13 Dec 2000 07:04:00 -0800 (PST)
📝 Subject: Body:
👥 Persons: None detected
🏢 Organizations: None detected
📍 Locations: None detected
💰 Money: None detected
💬 Content Preview: Attached are two files that illustrate the following: As prices rose, supply increased and demand decreased. Now prices are beginning to fall in response these market responses.
----------------------------------------------------------------------------------------------------

📧 Real Email #5
🆔 ID: email_000005
👤 From: kurt.lindahl@elpaso.com
👤 To: atsm@chewon.com, aarmstrong@sempratrading.com, neilaj@texaco.com,
📅 Date: Tue, 31 Jul 2001 08:28:00 -0700 (PDT)
📝 Subject: El Paso
👥 Persons: Origination El Paso, Tx 77252-2511, Kurt Lindahl Sr., Rob Bryngelson
🏢 Organizations: the ElPaso Corporation, El Paso, Global LNG Division, El Paso Merchant Energy, Business Development
📍 Locations: Houston
💰 Money: None detected
💬 Content Preview: Dear Friends and Colleagues, This note is to inform you that I have joined El Paso Merchant Energy in their Global LNG Division reporting to Rob Bryngelson, Managing Director, Business Development. Pl...
----------------------------------------------------------------------------------------------------

📈 Real Data Processing Statistics:
• Total real emails processed: 453905
• Average content length: 1474 characters
• Average entities per email: 8.1
• Emails with persons mentioned: 383312
• Emails with organizations mentioned: 363549
• Emails with money amounts: 63303

🔍 Most interesting real emails (by entity count):

📧 High-entity email from tradersummary@syncrasy.com
📝 Subject: Syncrasy Daily Trader Summary for Wed, Jan 16, 2002
👥 Persons: Data, NC ERCOT(SP, Max, Aquila, Andy Weingarten
🏢 Organizations: Trader Summary, ERCOT(SP, SPP(= SP, Average-Daily Maximum Temperature', MAPP(HP
💰 Money: 37 -1 MAIN(CTR, 50,000, 43 -1 MAIN(CTR, 36 -1 MAIN(CTR, 40 -1 WSCC(RK

📧 High-entity email from lucky@icelandair.is
📝 Subject: Iceland Food Festival
👥 Persons: Hotel Klopp, Rich, Mar 1 - National Beer Day, David Rosengarten, Subject
🏢 Organizations: Reykjav?k/K?pavogur, Party, Party Gourmet Dinner, BWI, SCENIC SIGHTSEEING Blue Lagoon
💰 Money: 65, 66, 69, 50, 55

📧 High-entity email from truorange@aol.com
📝 Subject: True Orange, November 27, Part 2
👥 Persons: Sooners, Jody Conradt, ESPN, Harris, Northwestern
🏢 Organizations: Oregon State, K-State, Texas A&M, SEC, Big East
💰 Money: $1.1 million, $1.9 million, $2.5 million, $1.2 million, 750,000

The data looks good, so let's now save it into a CSV file that we will then load to our knowledge base:

# Save processed real data
df_processed.to_csv('enron_emails_processed_real.csv', index=False)
print(f"\n✅ Real Enron emails saved to 'enron_emails_processed_real.csv'")

✅ Real Enron emails saved to 'enron_emails_processed_real.csv'

2.3 Connecting to the Vector Store

When the user creates a MindsDB Knowledge Base, MindsDB chunks all the text fragments into pieces (chunks) and uses an external text embedding model to convert each chunk into an embedding vector. Embedding vectors are numerical arrays that have the following property: if two texts are similar semantically, then their embedding vectors are close to each other in the vector space. This allows us to compare two texts semantically by applying a mathematical operation (like cosine similarity) to two vectors to see how close they are in the vector space.

These embedding vectors need to be stored somewhere. There are various vector databases, including several open-source ones. MindsDB supports ChromaDB by default. However, ChromaDB doesn't support the "LIKE" operation, which is a standard operation in relational database SELECT queries. We will use LIKE in our tutorial; therefore, we will use a different open-source vector store, PGVector, which is part of the Postgres ecosystem.

For this tutorial, we provisioned a PGVector instance on AWS. You can install it locally too. Here's how you can do it.

Let's create a vector database enron_kb_pgvector, which will store knowledge base's embedding vectors:

# Drop an existing pgvector database if it exists
try:
    print("🗑️  Dropping existing pgvector database...")
    server.query("DROP DATABASE IF EXISTS enron_kb_pgvector;").fetch()
    print("✅ Dropped existing database")
except Exception as e:
    print(f"⚠️  Drop error: {e}")

# Create fresh pgvector database connection
try:
    server.query("""
        CREATE DATABASE enron_kb_pgvector
        WITH ENGINE = 'pgvector',
        PARAMETERS = {
            "host": "c3hsmn51hjafhh.cluster-czrs8kj4isg7.us-east-1.rds.amazonaws.com",
            "port": 5432,
            "database": "df1f3i5s2jrksf",
            "user": "u36kd0g64092pk",
            "password": "pc08df7cb724a4ad6b1a8288c3666fa087f1a89c1ba5d1a555b40a8ba863672e4"
        };
    """).fetch()
    print("✅ Created pgvector database connection 'enron_kb_pgvector'")
except Exception as e:
    print(f"❌ Database connection error: {e}")
    raise

🗑️  Dropping existing pgvector database...
✅ Dropped existing database
✅ Created pgvector database connection 'enron_kb_pgvector'

2.4 Uploading the Dataset to MindsDB

Now let's connect to our local MindsDB instance and upload the dataset:

Remember, that in this tutorial, we assume that the user has a free open-source MindsDB instance running in their local environment. Please follow these steps to set it up.

import mindsdb_sdk
import re

# Connect to the MindsDB server
server = mindsdb_sdk.connect('http://127.0.0.1:47334')
print("Connected to MindsDB server")

# List available databases to confirm connection
databases = server.databases.list()
print("Available databases:")
for db in databases:
    print(f"- {db.name}")

# First drop any knowledge bases
try:
    print("🗑️  Dropping knowledge bases...")
    server.query("DROP KNOWLEDGE_BASE IF EXISTS enron_kb;").fetch()
    print("✅ Dropped knowledge base enron_kb")
except Exception as e:
    print(f"⚠️  KB drop error: {e}")

# Check if df_processed exists and has real data
try:
    print(f"\n📊 Checking real processed Enron data...")
    print(f"Shape: {df_processed.shape}")
    print(f"Columns: {df_processed.columns.tolist()}")

    if len(df_processed) == 0:
        print("❌ df_processed is empty. Please run the cell that creates it first.")
        raise ValueError("No processed data available")

    df_upload = df_processed.copy()
    print(f"✅ Using {len(df_upload)} real processed Enron emails")

except NameError:
    print("❌ Error: df_processed not found. Please run the cell that creates it first.")
    print("Cannot continue without real processed data.")
    raise

def clean_for_upload(text):
    """Clean text data for safe upload to MindsDB"""
    if pd.isna(text) or text == '':
        return ''

    text = str(text)
    # Remove problematic characters that might cause encoding issues
    text = re.sub(r'[^\w\s\-\.\@\,\;\:\!\?\(\)\[\]\/]', ' ', text)
    # Remove excessive whitespace
    text = re.sub(r'\s+', ' ', text)
    # Limit length to prevent upload issues
    if len(text) > 2000:
        text = text[:1997] + '...'

    return text.strip()

# Clean the real data for upload
print("Cleaning real Enron data for upload...")

# Clean text fields
text_columns = ['from_address', 'to_address', 'date_sent', 'subject', 'content', 
                'persons', 'organizations', 'locations', 'money_amounts', 
                'dates_mentioned', 'events', 'products']

for col in text_columns:
    if col in df_upload.columns:
        df_upload[col] = df_upload[col].apply(clean_for_upload)

# Ensure numeric columns are properly typed
for col in ['content_length', 'entity_count']:
    if col in df_upload.columns:
        df_upload[col] = pd.to_numeric(df_upload[col], errors='coerce').fillna(0).astype(int)

print(f"\n📋 Final real Enron dataset for upload:")
print(f"Shape: {df_upload.shape}")
print(f"Sample from addresses: {df_upload['from_address'].head(3).tolist()}")
print(f"Sample subjects: {df_upload['subject'].head(3).tolist()}")

# Upload to MindsDB
files_db = server.get_database("files")
table_name = "enron_emails"

# Delete existing table if it exists
try:
    files_db.tables.drop(table_name)
    print(f"Dropped existing table {table_name}")
except Exception:
    pass

# Upload real Enron data
try:
    print("Uploading real Enron emails to MindsDB...")
    files_db.create_table(table_name, df_upload)
    print(f"✅ Created table files.{table_name} with real Enron data")

    # Verify upload with real data
    sample_data = server.query(f"SELECT email_id, subject, persons, organizations FROM files.{table_name} LIMIT 5").fetch()
    print("\n✅ Sample real Enron data uploaded:")
    for idx, row in sample_data.iterrows():
        print(f"📧 {row['email_id']}: {row['subject'][:60]}...")
        print(f"   👥 Persons: {row['persons']}")
        print(f"   🏢 Orgs: {row['organizations']}")

    # Check total count
    count_result = server.query(f"SELECT COUNT(*) as total FROM files.{table_name}").fetch()
    total_emails = count_result.iloc[0]['total']
    print(f"\n📊 Total real Enron emails uploaded: {total_emails}")

except Exception as e:
    print(f"❌ Upload failed: {e}")

print("\n✅ Real Enron data upload process completed!")

Connected to MindsDB server
Available databases:
- files
- movies_kb_chromadb
🗑️  Dropping knowledge bases...
✅ Dropped knowledge base enron_kb

📊 Checking real processed Enron data...
Shape: (453905, 15)
Columns: ['email_id', 'from_address', 'to_address', 'date_sent', 'subject', 'content', 'persons', 'organizations', 'locations', 'money_amounts', 'dates_mentioned', 'events', 'products', 'content_length', 'entity_count']
✅ Using 453905 real processed Enron emails
Cleaning real Enron data for upload...

📋 Final real Enron dataset for upload:
Shape: (453905, 15)
Sample from addresses: ['daren.farmer@enron.com', 'eric.bass@enron.com', 'larry.campbell@enron.com']
Sample subjects: ['Re: Meter 5892 - UA4 1996 and 1997 Logistics Issues', 'DRAFT', 'More July CED-PGE']
Dropped existing table enron_emails
Uploading real Enron emails to MindsDB...
✅ Created table files.enron_emails with real Enron data

✅ Sample real Enron data uploaded:
📧 email_000000: Re: Meter 5892 - UA4 1996 and 1997 Logistics Issues...
   👥 Persons: Daren J Farmer/HOU, Susan, Meter 5892 - UA4 1996, Mary M Smith/HOU, Susan D Trevino
   🏢 Orgs: Volume Management
📧 email_000001: DRAFT...
   👥 Persons: Bcc
   🏢 Orgs: None
📧 email_000003: More July CED-PGE...
   👥 Persons: Susan Fick, Patty
   🏢 Orgs: None
📧 email_000004: Body:...
   👥 Persons: None
   🏢 Orgs: None
📧 email_000005: El Paso...
   👥 Persons: Origination El Paso, Tx 77252-2511, Kurt Lindahl Sr., Rob Bryngelson
   🏢 Orgs: the ElPaso Corporation, El Paso, Global LNG Division, El Paso Merchant Energy, Business Development

📊 Total real Enron emails uploaded: 453905

✅ Real Enron data upload process completed!

2.4 Creating a Knowledge Base

Now, let's create a knowledge base enron_kb using our emails data. We'll use OpenAI's embedding model to convert the text into vectors. Note the storage = enron_kb_pgvector.enron_vectors parameter which tells MindsDB to use our PGVector vector store. If we omit this parameter, teh default ChromaDB vector store will be used.

# Drop existing knowledge base if it exists
server.query("DROP KNOWLEDGE_BASE IF EXISTS enron_kb;").fetch()

# Create knowledge base with pgvector storage
try:
    kb_creation_query = server.query(f"""
    CREATE KNOWLEDGE_BASE enron_kb
    USING
        storage = enron_kb_pgvector.enron_vectors,
        embedding_model = {{
           "provider": "openai",
           "model_name": "text-embedding-3-large"
        }},
        metadata_columns = [
            'subject', 'persons', 'organizations', 'locations', 
            'money_amounts', 'dates_mentioned', 'events', 'products',
            'content_length', 'entity_count',
            'from_address', 'to_address', 'date_sent'
        ],
        content_columns = ['content'],
        id_column = 'email_id';
    """)
    kb_creation_query.fetch()
    print("✅ Created knowledge base 'enron_kb' with email address and date filtering support")
except Exception as e:
    print(f"❌ Knowledge base creation error: {e}")

✅ Created knowledge base 'enron_kb' with email address and date filtering support

Now let's insert our email data into the knowledge base:

# Insert the email data into the knowledge base (including the new metadata columns)
from yaspin import yaspin

try:
    with yaspin(text="Inserting emails into updated knowledge base..."):
        insert_query = server.query(f"""
            INSERT INTO enron_kb
            SELECT email_id,
                   subject,
                   persons,
                   organizations,
                   locations,
                   money_amounts,
                   dates_mentioned,
                   events,
                   products,
                   content_length,
                   entity_count,
                   from_address,
                   to_address,
                   date_sent,
                   content
            FROM   files.{table_name}
            USING
                batch_size = 200,
                threads = 10,
                error = 'skip',
                track_column = email_id;
            """).fetch() 
    print("✅ Emails inserted successfully into updated knowledge base!")
except Exception as e:
    print(f"❌ Insert error: {e}")

✅ Emails inserted successfully into updated knowledge base!

Let's see what the data in the KB looks like:

search_query = server.query("SELECT * FROM enron_kb;")
display(search_query.fetch())

search_query = server.query("SELECT count(*) FROM enron_kb;")
display(search_query.fetch())

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	id	chunk_id	chunk_content	metadata	relevance	distance
0	email_024653	email_024653:content:3of3:1993to2382	Palmer of Caminus Corp on European Markets! ht...	{'events': None, '_source': 'TextChunkingPrepr...	None	None
1	email_024585	email_024585:content:1of1:0to429	Most of you already know, but the move is taki...	{'events': None, '_source': 'TextChunkingPrepr...	None	None
2	email_024902	email_024902:content:1of2:0to997	To facilitate these changes, you received an O...	{'events': None, '_source': 'TextChunkingPrepr...	None	None
3	email_025043	email_025043:content:1of1:0to801	According to our system records, you have not ...	{'events': None, '_source': 'TextChunkingPrepr...	None	None
4	email_025044	email_025044:content:1of3:0to996	The attached preliminary comments were finaliz...	{'events': None, '_source': 'TextChunkingPrepr...	None	None
...	...	...	...	...	...	...
676450	email_024824	email_024824:content:1of1:0to585	Cc: m..love@enron.com, scott.palmer@enron.com ...	{'events': None, '_source': 'TextChunkingPrepr...	None	None
676451	email_024829	email_024829:content:1of3:0to998	20 [IMAGE] CO.O.L. Travel Specials [IMAGE] Wed...	{'events': 'Love Field', '_source': 'TextChunk...	None	None
676452	email_024829	email_024829:content:2of3:999to1998	on either Monday, May 21 or Tuesday, May 22, 2...	{'events': 'Love Field', '_source': 'TextChunk...	None	None
676453	email_024829	email_024829:content:3of3:1999to2391	IN 139 - Louisville, KY return to top Featured...	{'events': 'Love Field', '_source': 'TextChunk...	None	None
676454	email_024804	email_024804:content:1of1:0to354	FYI -- David Leboe in Investor Relations autho...	{'events': None, '_source': 'TextChunkingPrepr...	None	None

676455 rows × 6 columns

.dataframe tbody tr th:only-of-type {
    vertical-align: middle;
}

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

	count_0
0	676455

As we can see, the data looks very much like a regular relational table. However, the fact that it's a knowledge base instance rather than a regular database connection, allows us to use a special syntax mixing the semantic similarity with regulare SQL "WHERE" constructs.

You can also notice that the knowledge base contains chunks rather than the original texts of the email messages. Each chunk has its own embedding vectr. This allows finding more granular pieces of content similar to the user's question.

3. Performing Semantic Searches

Now that our knowledge base is ready (or being populated), let's do some Q&A. For convenience, we will setup a utility function answer_question_about_enron which will take as input question about the data and the attribute this data is expected to contain such as people names, organizations, locaitons, etc: the attributes thet the NER was supposed to have extracted. This utility function will combine the inputs into a SELECT query by using the MindsDB syntax. For example, if our question/request is "I need to see emails mentioning fraud." and we want to only to see emails from "John Smith", our SELECT query constructed by answer_question_about_enron would look like this:

SELECT 
    id, 
    chunk_content, 
    relevance,
    metadata
FROM enron_kb_full
WHERE content = 'I need to see emails mentioning fraud.' AND persons LIKE '%John Smith%'
ORDER BY relevance DESC
LIMIT 100;

The above query will return emails that mention any fraud even if the word "fraud" itself isn't used in the emails' texts. This is a soft search. Only those emails will be retuned whose "persons" attribute contains "John Smith". This is a hard search.

import openai
import time
import json
from IPython.display import display

# Set up OpenAI client (replace with your API key)
client = openai.OpenAI(api_key="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")

def answer_question_about_enron(question: str, persons=None, organizations=None, locations=None, money_amounts=None, subjects=None):
    """
    Answer questions about Enron using the knowledge base with metadata filtering

    Args:
        question (str): The question to ask
        persons (str or list): Person name(s) to filter by
        organizations (str or list): Organization name(s) to filter by  
        locations (str or list): Location name(s) to filter by
        money_amounts (str or list): Money amount(s) to filter by
        subjects (str or list): Subject keyword(s) to filter by

    Returns:
        str: Generated answer based on relevant emails
    """

    def create_like_conditions(column_name, values):
        """Helper function to create LIKE conditions for single values or lists"""
        if values is None:
            return []

        # Convert single value to list for uniform processing
        if isinstance(values, str):
            values = [values]
        elif not isinstance(values, (list, tuple)):
            values = [str(values)]

        conditions = []
        for value in values:
            conditions.append(f"{column_name} LIKE '%{value}%'")

        return conditions

    print(f"🤔 Question: {question}")

    # Build WHERE clause with optional filters
    where_conditions = [f"content = '{question}'"]

    # Handle persons filter
    if persons:
        person_conditions = create_like_conditions("persons", persons)
        if person_conditions:
            where_conditions.extend(person_conditions)
            if isinstance(persons, list):
                print(f"🔍 Filtering by persons: {', '.join(persons)}")
            else:
                print(f"🔍 Filtering by persons: {persons}")

    # Handle organizations filter
    if organizations:
        org_conditions = create_like_conditions("organizations", organizations)
        if org_conditions:
            where_conditions.extend(org_conditions)
            if isinstance(organizations, list):
                print(f"🔍 Filtering by organizations: {', '.join(organizations)}")
            else:
                print(f"🔍 Filtering by organizations: {organizations}")

    # Handle locations filter
    if locations:
        loc_conditions = create_like_conditions("locations", locations)
        if loc_conditions:
            where_conditions.extend(loc_conditions)
            if isinstance(locations, list):
                print(f"🔍 Filtering by locations: {', '.join(locations)}")
            else:
                print(f"🔍 Filtering by locations: {locations}")

    # Handle money_amounts filter
    if money_amounts:
        money_conditions = create_like_conditions("money_amounts", money_amounts)
        if money_conditions:
            where_conditions.extend(money_conditions)
            if isinstance(money_amounts, list):
                print(f"🔍 Filtering by money amounts: {', '.join(money_amounts)}")
            else:
                print(f"🔍 Filtering by money amounts: {money_amounts}")

    # Handle subjects filter
    if subjects:
        subject_conditions = create_like_conditions("subject", subjects)
        if subject_conditions:
            where_conditions.extend(subject_conditions)
            if isinstance(subjects, list):
                print(f"🔍 Filtering by subjects: {', '.join(subjects)}")
            else:
                print(f"🔍 Filtering by subjects: {subjects}")

    where_clause = " AND ".join(where_conditions)

    try:
        search_query = f"""
        SELECT 
            id, 
            chunk_content, 
            relevance,
            metadata
        FROM enron_kb
        WHERE {where_clause}
        ORDER BY relevance DESC
        LIMIT 100;
        """

        print(f"🔍 SQL Query: {search_query}")
        search_results = server.query(search_query).fetch()

        if len(search_results) == 0:
            return "❌ No relevant emails found matching your criteria."

        print(f"✅ Found {len(search_results)} results")

        # Show sample results
        for idx, row in search_results.head(3).iterrows():
            try:
                metadata = json.loads(str(row['metadata'])) if row['metadata'] and str(row['metadata']) != 'nan' else {}
                print(f"\n📧 Result #{idx+1}")
                print(f"🆔 Email ID: {row['id']}")
                print(f"📊 Relevance: {row['relevance']:.4f}")
                print(f"📝 Subject: {metadata.get('subject', 'No subject')}")
                print(f"👥 Persons: {metadata.get('persons', 'None')}")
                print(f"🏢 Organizations: {metadata.get('organizations', 'None')}")
                print(f"📍 Locations: {metadata.get('locations', 'None')}")
                content = str(row['chunk_content'])
                preview = content[:200] + "..." if len(content) > 200 else content
                print(f"💬 Content: {preview}")
                print("-" * 60)
            except Exception as e:
                print(f"Error processing result: {e}")

        # Prepare context for GPT
        context_parts = []
        for idx, row in search_results.iterrows():
            try:
                metadata = json.loads(str(row['metadata'])) if row['metadata'] and str(row['metadata']) != 'nan' else {}
                email_context = f"""
Email ID: {row['id']}
Subject: {metadata.get('subject', 'No subject')}
Persons mentioned: {metadata.get('persons', 'None')}
Organizations mentioned: {metadata.get('organizations', 'None')}
Content: {row['chunk_content'][:800]}...
"""
                context_parts.append(email_context)
            except:
                continue

        context = "\n---\n".join(context_parts[:8])

        # Create prompt for GPT
        prompt = f"""
        You are an expert analyst studying the Enron corporate emails dataset. Based ONLY on the following 
        email excerpts from the Enron corpus, answer the user's question.

        EMAIL EXCERPTS FROM ENRON CORPUS:
        {context}

        QUESTION: {question}

        Instructions:
        - Provide a factual answer based only on the email content provided above
        - If the emails mention specific people, organizations, or amounts, include those details
        - If the emails don't contain enough information to answer the question, state that clearly
        - Reference specific email IDs when making claims
        """

        print("🤖 Generating answer using GPT-4...")
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": "You are a helpful analyst answering questions about Enron emails. Use only the provided email content and be specific about sources."},
                {"role": "user", "content": prompt}
            ],
            temperature=0.1
        )

        answer = response.choices[0].message.content
        print(f"\n💡 ANSWER:\n{answer}")
        return answer

    except Exception as e:
        print(f"❌ Error during search: {e}")
        return f"Error during search: {e}"


# Process the three original questions with metadata filtering
print("=== ENRON EMAIL ANALYSIS WITH METADATA FILTERING ===\n")

print("📋 Question 1:")
answer1 = answer_question_about_enron(
    "What concerns did Sherron Watkins express to Ken Lay in her email about Enron''s accounting practices?",
    persons=["Watkins", "Lay"]
)

print("\n" + "="*100 + "\n")

print("📋 Question 2:")

answer = answer_question_about_enron(
    "How did David Delainey justify inflating Mariner''s valuation from $250M to $600M in his email to Ken Lay?",
    persons=["Delainey"]
)

print("📋 Question 3:")

answer = answer_question_about_enron(
    "How did Tim DeSpain coach Ken Lay on what to tell credit rating agencies about Enron''s financial condition?",
    organizations=["Moody"]
)

=== ENRON EMAIL ANALYSIS WITH METADATA FILTERING ===

📋 Question 1:

🤔 Question: What concerns did Sherron Watkins express to Ken Lay in her email about Enron''s accounting practices?

🔍 Filtering by persons: Watkins, Lay

🔍 SQL Query: 

        SELECT 

            id, 

            chunk_content, 

            relevance,

            metadata

        FROM enron_kb

        WHERE content = 'What concerns did Sherron Watkins express to Ken Lay in her email about Enron''s accounting practices?' AND persons LIKE '%Watkins%' AND persons LIKE '%Lay%'

        ORDER BY relevance DESC

        LIMIT 100;

✅ Found 16 results

📧 Result #1

🆔 Email ID: email_048101

📊 Relevance: 0.7053

📝 Subject: The key questions I asked Lay on Aug 22

👥 Persons: Sherron S. Watkins, Lay

🏢 Organizations: Enron Corp.

📍 Locations: None

  
  
  💬 Content: Sherron S. Watkins Vice President, Enron Corp. 713-345-8799 office 713-416-0620 cell


📧 Result #2

🆔 Email ID: email_335299

📊 Relevance: 0.6832

📝 Subject: TEAM 4 - HR ENERGY COMMERCE SUBPOENA (1) (01/14/02) AND (2) (12/10/01)

👥 Persons: Ken Lay, Sherron Watkins, JEDI

🏢 Organizations: BLUE DOG, LJM2, 09 09The, the RAP TEAM, Enron

📍 Locations: V E, electr

  
  
  💬 Content: Please search your files and collect all records and documents covered by o r relevant to the following requests: (1) 09All records relating to any investigations/review of the allegations raised by S...


📧 Result #3

🆔 Email ID: email_178480

📊 Relevance: 0.6727

📝 Subject: TEAM 4 - HR ENERGY COMMERCE SUBPOENA (1) (01/14/02) AND (2)

👥 Persons: Ken Lay, Sherron Watkins, JEDI, Bcc

🏢 Organizations: BLUE DOG, LJM2, k..heathman@enron.com, minutes , 09 09The

📍 Locations: V E, electr

  
  
  💬 Content: (12/10/01) Cc: k..heathman@enron.com, team.response@enron.com Bcc: k..heathman@enron.com, team.response@enron.com We remain in the process of gathering information sought by various governm ental agen...


🤖 Generating answer using GPT-4...

💡 ANSWER:

Sherron Watkins expressed concerns to Ken Lay about Enron's accounting practices in her email, stating that she was "incredibly nervous that we will implode in a wave of accounting scandals." This concern was highlighted in an email discussing the broader context of Enron's financial issues, where it was noted that Andersen, the government, and Enron itself had access to financial data indicating the company's potential collapse (Email ID: email_446563). Additionally, her concerns were significant enough to prompt investigations and reviews of the allegations she raised in her August memo to Ken Lay, as mentioned in emails discussing subpoenas and document requests (Email IDs: email_335299 and email_178480).

====================================================================================================

📋 Question 2:

🤔 Question: How did David Delainey justify inflating Mariner''s valuation from $250M to $600M in his email to Ken Lay?

🔍 Filtering by persons: Delainey

🔍 SQL Query: 

        SELECT 

            id, 

            chunk_content, 

            relevance,

            metadata

        FROM enron_kb

        WHERE content = 'How did David Delainey justify inflating Mariner''s valuation from $250M to $600M in his email to Ken Lay?' AND persons LIKE '%Delainey%'

        ORDER BY relevance DESC

        LIMIT 100;

✅ Found 100 results

📧 Result #1

🆔 Email ID: email_006062

📊 Relevance: 0.7015

📝 Subject: Mariner

👥 Persons: Delainey, Ken, Kase Lawal, Bcc

🏢 Organizations: un, IPO, Mariner, E P

📍 Locations: None

  
  
  💬 Content: Cc: jeff.donahue@enron.com, raymond.bowen@enron.com Bcc: jeff.donahue@enron.com, raymond.bowen@enron.com Ken, in response to your note, I am not aware of any official dialogue with Mr. Kase Lawal abou...


📧 Result #2

🆔 Email ID: email_400597

📊 Relevance: 0.7015

📝 Subject: Mariner

👥 Persons: Delainey, Ken, Kase Lawal, Bcc

🏢 Organizations: un, IPO, Mariner, E P

📍 Locations: None

  
  
  💬 Content: Cc: jeff.donahue@enron.com, raymond.bowen@enron.com Bcc: jeff.donahue@enron.com, raymond.bowen@enron.com Ken, in response to your note, I am not aware of any official dialogue with Mr. Kase Lawal abou...


📧 Result #3

🆔 Email ID: email_275215

📊 Relevance: 0.7015

📝 Subject: Mariner

👥 Persons: Delainey, Ken, Kase Lawal, Bcc

🏢 Organizations: un, IPO, Mariner, E P

📍 Locations: None

  
  
  💬 Content: Cc: jeff.donahue@enron.com, raymond.bowen@enron.com Bcc: jeff.donahue@enron.com, raymond.bowen@enron.com Ken, in response to your note, I am not aware of any official dialogue with Mr. Kase Lawal abou...


🤖 Generating answer using GPT-4...

💡 ANSWER:

David Delainey justified inflating Mariner's valuation from $250M to $600M based on several factors mentioned in the emails. According to the content of multiple emails (email IDs: email_006062, email_400597, email_275215, email_372317, email_280761, email_372732, email_332901), the justification included:



Successful Wells: Mariner had enjoyed a series of successful wells that were expected to be booked in reserve reports by the following March.

Increases in Gas and Oil Prices: There were significant increases in gas and oil prices, which contributed to the higher valuation.

Reserve Growth: The reserve growth was a key factor in the increased valuation.

Current Energy Prices: The current energy prices at the time supported the higher valuation.

Future Goals: The goal was to demonstrate three to four quarters of increasing operating cash flow and reserves growth before attempting further actions.


These factors collectively contributed to the stretch target valuation of $600M, which Delainey noted was not incredibly out of line given the circumstances.

📋 Question 3:

🤔 Question: How did Tim DeSpain coach Ken Lay on what to tell credit rating agencies about Enron''s financial condition?

🔍 Filtering by organizations: Moody

🔍 SQL Query: 

        SELECT 

            id, 

            chunk_content, 

            relevance,

            metadata

        FROM enron_kb

        WHERE content = 'How did Tim DeSpain coach Ken Lay on what to tell credit rating agencies about Enron''s financial condition?' AND organizations LIKE '%Moody%'

        ORDER BY relevance DESC

        LIMIT 100;

✅ Found 100 results

📧 Result #1

🆔 Email ID: email_028284

📊 Relevance: 0.6674

📝 Subject: Yesterday s Call: Feedback

👥 Persons: Good Luck, Jeff P.S., Cal Ed

🏢 Organizations: LJM, ENE, Moody s, Fastow, SEC

📍 Locations: Citi, Skilling

  
  
  💬 Content: Ken, Thanks for having the call yesterday. I am a believer in Enron and we are buying your debt. Here s short feedback on the call. I give the call a B-/C grade. If you want a good example of a compan...


📧 Result #2

🆔 Email ID: email_268144

📊 Relevance: 0.6636

📝 Subject: Moody s Annual Review Meeting

👥 Persons: Jeff McMahon, Stephen Moore - Relationship, Foley, Tim, Ben

🏢 Organizations: Sierra Pacific, EBS, International Asset Sales, Wholesale Services, Moody s

📍 Locations: California

  
  
  💬 Content: Director, and Stephen Moore - Relationship Manager (our analyst). Diaz and Moore are very familiar with the Enron credit profile. Foley is their boss. He apparently is the leader of their ratings comm...


📧 Result #3

🆔 Email ID: email_152456

📊 Relevance: 0.6528

📝 Subject: Moody s and Standard Poor s

👥 Persons: John Diaz, Ben, Bcc, Andy, Tim DeSpain

🏢 Organizations: Credit Ratings - emphasize, Moody s Call:, Standard Poor s, EBS, Dhabol

📍 Locations: None

  
  
  💬 Content: Cc: ben.glisan@enron.com, andrew.fastow@enron.com Bcc: ben.glisan@enron.com, andrew.fastow@enron.com Two conference calls have been tenatively scheduled to allow you to directly discuss Enron s commit...


🤖 Generating answer using GPT-4...

💡 ANSWER:

Tim DeSpain, along with Andy and Ben, coached Ken Lay on what to tell credit rating agencies about Enron's financial condition by emphasizing several key assurances. According to email ID: email_152456 and email ID: email_372468, they advised Ken Lay to stress the following points:


Commitment to Maintaining Credit Ratings: They emphasized that maintaining credit ratings was critical to Enron's fundamental businesses, particularly gas and power marketing. They noted that both counterparties and creditors placed significant importance on Enron's consistent rating profile.
Strength of Core Businesses: They highlighted that Enron's core businesses were strong, positioning Enron as the leading franchise in energy marketing. They anticipated continued strength in financial performance from the commodity groups.


These points were intended to assure the credit rating agencies of Enron's financial stability and commitment to its credit ratings.

Conclusion

In this tutorial, we've successfully built a sophisticated question-answering system over the Enron email corpus by combining MindsDB's Knowledge Base capabilities with Named Entity Recognition. This demonstrates how modern AI tools can transform unstructured text into a queryable, intelligent knowledge base.

Key Achievements

Automated metadata extraction: By leveraging SpaCy's NER models, we automatically extracted structured entities (people, organizations, locations) from raw email text, converting unstructured data into a hybrid storage system that supports both semantic and structured queries.
Hybrid search capabilities: The knowledge base enables both soft search criteria (semantic similarity through embeddings) and hard search criteria (metadata filtering), allowing for precise and flexible information retrieval. This combination significantly enhances search accuracy and reduces irrelevant results.
Simplified query interface: MindsDB abstracts away the complexity of vector databases, embedding models, and similarity calculations behind a familiar SQL interface. The addition of a simple content attribute in SQL SELECT statements makes semantic search accessible to anyone familiar with SQL.
Practical RAG implementation: By integrating the knowledge base with a chat LLM, we've created a Retrieval-Augmented Generation (RAG) system that can answer complex questions by first retrieving relevant context and then generating informed answers, significantly reducing hallucinations.

Real-World Applications

The techniques demonstrated in this tutorial have broad applications beyond the Enron dataset:

Corporate knowledge management: Search through internal documents, emails, and reports using both semantic queries and metadata filters
Legal discovery: Find relevant communications filtered by sender, recipient, date range, or mentioned entities
Customer support: Build intelligent support systems that can search through product documentation and past support tickets
Research analysis: Query academic papers, research notes, or experimental data with combined semantic and structured filtering

Next Steps

To extend this project, consider:

Expanding entity types: Extract additional metadata such as monetary amounts, dates, or custom domain-specific entities
Finetuning embeddings: Use domain-specific embedding models for improved semantic matching in specialized fields
Multi-modal knowledge bases: Incorporate documents, images, and other file types into your knowledge base
Advanced filtering: Implement complex boolean logic and date-range queries for more sophisticated searches
Production deployment: Scale the system to handle larger datasets and concurrent users

For more information on MindsDB Knowledge Bases and advanced features, visit the official documentation.

Watch the playback of the live webinar on youtube:

MariaDB & MindsDB Turns WooCommerce Data to Insights with Real-Time AI Analytics for eCommerce Teams

MindsDB Team — Mon, 29 Dec 2025 23:07:03 +0000

Written by Chandre Van Der Westhuizen, Community & Marketing Co-ordinator at MindsDB

MariaDB is a widely used open-source relational database that powers eCommerce operations with features like high availability, scalability, and flexible storage engine options. Platforms such as WooCommerce rely on it to manage product catalogs, customer information, and transaction workflows efficiently - and this is why so many eCommerce teams choose MariaDB.

eCommerce businesses run on data - customer activity, product trends, abandoned carts, shipping timelines, and returns. But too often, that data lives locked away in silos: MariaDB for transactions, spreadsheets for performance metrics, and dashboards for reporting.

By the time a marketing or operations team reacts, the opportunity has passed.

With MindsDB, eCommerce teams can connect directly to their MariaDB databases and use AI to analyze, predict, and act on data - all in real time, using SQL or natural language, and with zero ETL pipelines.

The Challenge: Too Many Tools, Too Much Latency

WooCommerce stores backed by MariaDB collect a wealth of information:

Orders, payments, and refunds
Customer profiles and buying behavior
Product inventory and pricing
Shipment tracking and delivery times

Traditionally, turning that raw data into insights required exporting CSVs, setting up ETL pipelines, or using third-party BI dashboards. These processes add friction, delay decisions, and make it nearly impossible to deliver real-time personalization or dynamic pricing.

MindsDB + MariaDB: AI-Powered Insights Without Moving Your Data

MindsDB brings the power of AI directly into the database layer - meaning your WooCommerce data becomes instantly searchable, explainable, and predictable without ever moving it.

With MindsDB’s integration with MariaDB, eCommerce teams can run real-time analytics, semantic search, and predictions directly on live orders, customers, products, and reviews - all without ETL or data duplication.

With MindsDB’s integration with MariaDB, you can:

Query live WooCommerce data (orders, customers, products, reviews) directly in the database using SQL or natural language - no ETL required
Perform hybrid search that blends structured and unstructured data (e.g., review-text + order history) to surface insights like shipping issues, churn risk, or product defects
Predict outcomes (reorders, stockouts, customer lifetime value) in real time, since everything operates inside MariaDB
Preserve data governance and security-data stays in MariaDB’s environment, enabling easier auditability and compliance
Empower cross-functional teams (CX, marketing, operations, finance) with on-demand analytics from the same system powering your eCommerce

MindsDB + MariaDB turns your WooCommerce store into a real-time AI analytics engine- without changing your infrastructure or data stack.

The MindsDB Solution: AI-Native, Zero-ETL Analytics Inside MariaDB

MindsDB’s Federate Query Engine allows you to connect to MariaDB using a single interface and SQL where you can unify your data with Knowledge Bases and query it using Hybrid Search by performing SQL operations or natural language via Agents.

To showcase how you can turn your raw data into valuable insights, we will be using a sample dataset for WooCommerce stored in MariaDB.

Pre-requisites:

Access MindsDB’s GUI via Docker locally or MindsDB’s extension on Docker Desktop.
Configure your default models in the MindsDB GUI by navigating to Settings → Models.
Navigate to Manage Integrations in Settings and install the dependencies for MariaDB.

Once you have installed the dependencies in the GUI, you can connect to MariaDB using the SQL Editor:

CREATE DATABASE mariadb  --- display name for database.
WITH ENGINE = 'mariadb',      --- name of the mindsdb handler
PARAMETERS = {
   "user": "demo_user",              --- Your database user.
   "password": "demo_password",          --- Your password.
   "host": "samples.mindsdb.com",              --- host, it can be an ip or an url.
   "port": "3307",           --- common port is 3306.
   "database": "test_data"           --- The name of your database *optional.
};

The data we will access is the Woocommerce Orders, Products, Reviews and Customers tables.

Unifying MariaDB’s WooCommerce Data using MindsDB’s Knowledge Bases

A MindsDB Knowledge Base is an AI-enhanced table that organizes information by meaning instead of keywords, using embeddings, rerankers, and vector storage to understand context. This allows it to perform semantic reasoning across data points, delivering deeper insights and highly accurate, context-aware answers.

Knowledge Bases will be created for the Woocommerce Orders, Products, Reviews and Customers tables.

The first table we will use is the Products table. To create a Knowledge Base, the CREATE KNOWLEDGE_BASE statement will be used:

CREATE KNOWLEDGE_BASE products_kb
USING
storage = heroku.product_kb,
metadata_columns = ['price', 'stock', 'rating', 'category'],
content_columns = ['name'],
id_column = 'product_id';

Here are the parameters provided:

products_kb: The name of the knowledge base.
storage : The storage table where the embeddings of the knowledge base is stored. As you can see we are using the PGVector database we created a connection with and provide the name orders to the table that will be created for storage.
metadata_columns : Here columns are provided as meta data columns to perform metadata filtering.
content_columns : Here columns are provided for semantic search.
id_column: This uniquely identifies each source data row in the knowledge base

The data from MariaDB can be inserted into this Knowledge Base using the INSERT INTO statement:

INSERT INTO products_kb
SELECT product_id, category, price, stock, rating, name
FROM mariadb.woocommerce_products;

You can select the Knowledge Base to query the data that has been inserted using the SELECT statement:

SELECT * FROM products_kb;

Using the same steps as above, knowledge bases have been created for the remaining tables:

Customers_kb : Created with the WooCommerce Customers table in MariaDB

Reviews_kb : Created with the WooCommerce Reviews table in MariaDB

Orders_kb : Created with the WooCommerce Orders table in MariaDB

Performing Keyword and Semantic Search using MindsDB’s Hybrid Search

Knowledge bases offer both semantic search and keyword-based search, each suited for different types of queries. Hybrid search combines them, ensuring users get results that match meaning as well as exact terms, covering scenarios where embeddings miss specific keywords or identifiers.

eCommerce teams can understand which products in key categories (like Electronics and Fitness) are generating positive customer sentiment.

-- Correlate positive reviews with products
SELECT  *
FROM reviews_kb
JOIN products_kb
ON reviews_kb.id = products_kb.id
WHERE reviews_kb.content = 'would buy again'
AND reviews_kb.rating >= 2.0
AND products_kb.category IN ('Electronics', 'Fitness')
AND reviews_kb.hybrid_search_alpha = 0.5
AND products_kb.hybrid_search_alpha = 0.5;

By correlating favorable reviews with specific products using semantic intent (“would buy again”), teams can identify high-performing items, refine marketing strategies, improve product recommendations, and prioritize inventory for products that foster strong customer loyalty.

Let's identify products that consistently generate negative sentiment-specifically complaints about quality and value.

--Correlate products with negative reviews
SELECT  *
FROM reviews_kb
JOIN products_kb
ON reviews_kb.id = products_kb.id
WHERE reviews_kb.content LIKE 'Not worth the price.'
AND reviews_kb.content = 'Quality could be better.'
AND reviews_kb.hybrid_search_alpha = 0.5
AND products_kb.hybrid_search_alpha = 0.5;

By correlating these reviews directly with the affected products, eCommerce teams can detect potential quality issues, supplier problems, misleading product descriptions, or customer-experience gaps early, allowing them to take corrective action before negative feedback impacts sales, returns, or brand reputation.

eCommerce teams can quickly identify which customers have experienced refunded orders- a strong signal of friction, dissatisfaction, or potential operational issues.

--Identify customers with refunded orders
SELECT *
FROM orders_kb
JOIN customers_kb
ON orders_kb.id = customers_kb.id
WHERE orders_kb.content = 'Refunded'
AND customers_kb.hybrid_search_alpha = 0.5
AND orders_kb.hybrid_search_alpha = 0.5 ;

By linking refunds directly to customer profiles, teams can investigate root causes, prevent churn among high-value customers, and improve support, logistics, or product quality before these problems escalate.

Teams can highlight high-priority operational risks by surfacing pending orders belonging to Platinum (top-tier) customers.

--Track orders that are pending for Platinum customers
SELECT *
FROM orders_kb
JOIN customers_kb
ON orders_kb.id = customers_kb.id
WHERE  customers_kb.content = 'Platinum'
AND orders_kb.content = 'Pending'
AND customers_kb.relevance >= 0.9
AND customers_kb.hybrid_search_alpha = 0.5
AND orders_kb.hybrid_search_alpha = 0.5 ;

These customers contribute disproportionately to revenue and loyalty, so delays or issues in their orders can directly impact retention and brand trust. By proactively identifying pending orders for Platinum customers, support and operations teams can intervene faster, reduce dissatisfaction, and ensure premium customers receive the service level they expect.

Together, these AI-powered Hybrid Search queries demonstrate how MindsDB and MariaDB transform raw WooCommerce data into actionable intelligence- helping teams anticipate issues, understand customers more deeply, and make smarter, faster decisions. By combining semantic and keyword search with structured analytics, businesses gain a real-time, 360° view of product performance, customer sentiment, and operational health, ultimately enabling a more resilient, data-driven eCommerce strategy.

Create a MindsDB Agent That Understands Your MariaDB’s WooCommerce Data

MindsDB’s Agents make it possible to interact conversationally with your data- both structured and unstructured- through MindsDB. Here we will create an AI Agent with the Knowledge Bases we have previously created with the MariaDB data.

Use the CREATE AGENT statement to create the AI Agent:

CREATE AGENT mariadb_ecommerce_agent
USING
data = {
        "knowledge_bases": ["orders_kb", "customers_kb", "reviews_kb", "products_kb"]
   },
   prompt_template='You are an AI assistant working with WooCommerce eCommerce data stored across several Knowledge Bases:

orders_kb - contains order-level information such as customer_id, order_date, total_amount, payment_method, and order_status.

customers_kb - contains customer profiles including first_name, last_name, email, country, loyalty_tier, signup_date, and total_spent.

reviews_kb - contains product reviews including review_text, rating, review_date, and the customer who wrote the review.

products_kb - contains product catalog information such as product name, category, price, stock, and rating.

Use these Knowledge Bases to answer questions about WooCommerce performance, customer behavior, product insights, order patterns, and review sentiment.
Always provide grounded, data-backed answers using the available knowledge.';

Here is the breakdown of the parameters provided to the agent:

mariadb_ecommerce_agent : The name provided to the agent
data : This parameter stores data connected to the agent, including knowledge bases and data sources connected to MindsDB.
knowledge_bases : stores the list of knowledge bases to be used by the agent.
prompt_template : This parameter stores instructions for the agent. It is recommended to provide data description of the data sources listed in the knowledge_bases parameter to help the agent locate relevant data for answering questions.

MindsDB offers a Chat Interface in the GUI that allows you to chat with your AI Agent in natural language. Lets ask a few questions to gain insights.

Question 1: Which products have low stock levels?
By identifying products that are running low, eCommerce teams can prevent stockouts, avoid lost revenue, maintain accurate delivery estimates, and plan timely replenishment—ensuring a smoother shopping experience and better inventory management overall.

You can expand on the table and scroll to see the full list.

Question2: Which categories have the highest average rating?
Knowing which product categories have the highest average rating helps eCommerce teams understand where customers are most satisfied. It guides decisions around merchandising, marketing focus, supplier relationships, and future product investments—allowing the business to double down on categories that consistently deliver strong customer experiences.

Question 3: Is there any meaningful correlation between review sentiment and repeat purchasing?
This helps reveal whether customer satisfaction directly influences loyalty and repeat purchases. Understanding this correlation allows eCommerce teams to prioritize experience improvements that have the greatest impact on long-term revenue and customer retention.

Question 4: Which customers have made the most purchases this year?
This identifies your highest-engagement customers—the ones driving the most transactions and revenue. Knowing who they are allows teams to target rewards, personalized marketing, and retention strategies toward their most valuable buyers.

Question 5: Give me a summary of customer sentiment for electronics products.
This provides a clear snapshot of how customers feel about a key category, helping teams quickly identify strengths, weaknesses, and emerging issues. Understanding sentiment for electronics products guides product improvements, marketing decisions, and support priorities.

Question 6: What is the sum of the total amount per order status
This shows how revenue is distributed across different order statuses, such as Completed, Pending, or Refunded. This helps teams understand fulfillment efficiency, identify bottlenecks, and quantify the financial impact of delays or cancellations.

These questions empower eCommerce teams to turn raw operational data into clear, actionable insights. By understanding customer behavior, product performance, sentiment trends, and revenue patterns, businesses can make smarter decisions that improve customer experience, streamline operations, and drive sustainable growth- and with MindsDB and MariaDB working together, these insights become real-time, AI-driven, and accessible directly from the data source.

Real-World Use Cases for Teams Using MariaDB with MindsDB:

1. Customer Retention & Re-Order Predictions: **Identify who will buy again in the next 30 days, and trigger automated win-back campaigns.
**2. Smarter Product Recommendations: **Blend product metadata, purchase history, and review sentiment to power AI personalization.
**3. Real-Time Inventory Forecasting: Predict stockouts or slow-moving items and optimize replenishment.
4. Operational Intelligence: Understand why refunds spike, what customers complain about, and where shipment delays occur.
5. Executive Dashboarding Without Limits: Query everything directly from SQL or natural language - without waiting for ETL jobs.

Why This Matters for Teams Using MariaDB

With MindsDB and MariaDB, you get:

A single unified view of your entire operation: Orders, reviews, returns, and inventory - all searchable and analyzable.
**Instant insights without exporting data: **Skip ETL. Skip spreadsheets. See what’s happening now.
AI-native intelligence across every workflow: Forecasting, summarization, classification, hybrid search - all inside SQL.
Built-in trust and traceability: Every answer is grounded in your actual database rows.

This is the future of your business - AI that understands because it sits directly on top of your real data.

Conclusion

MariaDB and MindsDB together unlock a new era of real-time, AI-powered intelligence- moving teams beyond traditional dashboards, slow ETL pipelines, and disconnected tools. By unifying structured operational data with unstructured review text and layering semantic understanding on top, MindsDB transforms MariaDB into a live decision engine that answers complex questions, reveals hidden patterns, and predicts what happens next.

Whether you're optimizing inventory, improving product quality, reducing refunds, understanding customer sentiment, or driving retention, MindsDB makes these insights available instantly and directly where your data already lives. With hybrid search, knowledge bases, and intelligent agents, teams can finally interact with their MariaDB data the way they think- conversationally, contextually, and without friction.

The result is faster decisions, more resilient operations, and a smarter, data-driven business powered by AI that sits right inside your database. If you are using MariaDB and would like to supercharge your data with AI, contact our team to get started.

MindsDB Supercharges Google's MCP Toolbox with Unstructured Data Support

MindsDB Team — Mon, 29 Dec 2025 20:46:39 +0000

Written by Erik Bovee, Head of Business Development at MindsDB

We’re happy to announce that we’ve integrated MindsDB with Google's open-source project, MCP (Model Context Protocol) Toolbox. This will make your AI applications very, very smart. This enhancement expands the Toolbox's reach, especially for organizations grappling with lots of siloed data.

The MindsDB integration at a glance

At its core, MindsDB is a federated query engine designed specifically for AI applications, that acts as a universal translator, enabling you to query hundreds of data sources (structured, semi-structured, unstructured) using familiar SQL. We’ve contributed this powerful capability as a new connector into MCP Toolbox, allowing developers and AI agents to seamlessly interact with a broader spectrum of enterprise data through MindsDB.

Now, with MindsDB, MCP Toolbox can connect to hundreds of datasources, including popular business applications like Salesforce, Jira, and GitHub, and even unstructured data sources like Gmail and Slack. This means you can break down data silos and connect all your data from a single API to your AI applications. Popular use cases include AI-powered search, analytics, and the ability to provide real-time data to agentic applications.

Key features: Bridging structured and unstructured worlds

MindsDB brings several essential features to the MCP Toolbox:

Datasource expansion: The most immediate and impactful benefit is the sheer volume of new data sources accessible. Imagine querying Salesforce opportunities alongside GitHub activity, or analyzing email patterns with Slack conversations—all through a unified interface. This greatly expands the Toolbox's utility for enterprise users.
SQL interface for any data: This is where MindsDB shines for developers. It allows you to write standard SQL queries that automatically translate to various API protocols, including REST APIs, GraphQL, and native protocols. This reduces the complexity and learning curve associated with accessing diverse data.
Cross-datasource AI analytics: MindsDB SQL capability makes it possible to perform joins and analytics across different data sources. For instance, you can correlate sales data from Salesforce with development activity from GitHub, providing a holistic view of your business operations that was previously unattainable. To facilitate this, MindsDB treats each data source, unstructured or structured, as a virtual table, facilitating sophisticated SQL operations across all sources.
Access to unstructured data: MindsDB provides over 200 data connectors with knowledge bases bringing the ability of indexing text data that then can be queried using SQL.
Knowledge Bases for unstructured data: MindsDB allows you to create Knowledge Bases which are essentially autonomous Retrieval-Augmented Generation (RAG) systems. You can ingest unstructured data like emails (Gmail/Outlook), messages (Slack, Microsoft Teams, Discord), and files (S3, filesystems) into these knowledge bases. Once ingested, this unstructured data becomes queryable by an AI application or model. This querying across data sources is facilitated by an auto-generated data catalog that contains metadata and a relational model across all data sources. MindsDB also supports hybrid search, combining vector similarity with keyword search to surface the most relevant results for AI search and analytics use cases.

Technical implementation updates

This integration brings a suite of powerful updates and benefits for developers:

New MindsDB Source implementation: The Toolbox now includes a new MindsDB source implementation, leveraging the MySQL wire protocol for robust connectivity.
Comprehensive test coverage: Extensive unit and integration tests ensure reliability and backward compatibility of the new MindsDB tools with existing SQL features.
Dedicated MindsDB tools: New mindsdb-execute-sql for direct SQL execution and mindsdb-sql for parameterized queries offer enhanced flexibility.

Quickstart Guide

To get started with MindsDB and the MCP Toolbox, follow these general steps:

1. Set up the MCP Toolbox: Ensure you have the MCP Toolbox service running. You can find detailed instructions in the official documentation.
2. Install MindsDB: The fastest way to get MindsDB up and running is via Docker:

docker run --name mindsdb_container \
-e MINDSDB_APIS=http,mysql \
-p 47334:47334 -p 47335:47335 \
mindsdb/mindsdb

For more installation options (e.g., PyPI), refer to the MindsDB documentation.

3. Connect your data sources in MindsDB: Within MindsDB, you'll need to create "databases" that connect to your external data sources (e.g., Salesforce, GitHub, Gmail). This typically involves CREATE DATABASE statements. For example:

-- connect to salesforce
CREATE DATABASE salesforce_datasource
WITH
    ENGINE = 'salesforce',
    PARAMETERS = {
        "username": "your-username@email.com",
        "password": "your-password",
        "client_id": "your-client-id",
        "client_secret": "your-client-secret"
    };

-- connect to postgres
CREATE DATABASE postgresql_datasource 
WITH
ENGINE = 'postgres', 
PARAMETERS = {
    "host": "postgres.sample.com",
    "port": 5432,
    "database": "postgres",
    "user": "postgres",
    "schema": "data",
    "password": "password"
};

Refer to the MindsDB documentation for specific connector details.

4. Use MindsDB Tools in MCP Toolbox: The MindsDB integration within MCP Toolbox allows you to execute SQL queries across your connected MindsDB data sources. You can use tools like mindsdb-execute-sql for direct querying.

-- run federated queries
SELECT *
FROM salesforce_datasource.account a
JOIN postgresql_datasource.customer_int c
ON a.`Id` = c.`AccountId`;

5. Unify data into the Knowledge Base: You can then load data into a MindsDB ‘Knowledge Base’, which is particularly useful for things like semantic search over large, unstructured data sets.

CREATE KNOWLEDGE_BASE my_kb
USING
    embedding_model = {
        "provider": "openai",
        "model_name" : "text-embedding-3-large",
        "api_key": "sk-..."
    },
    reranking_model = {
        "provider": "openai",
        "model_name": "gpt-4o",
        "api_key": "sk-..."
    },
    storage = my_vector_store.storage_table,
    metadata_columns = ['AccountId', 'Created_At', ...],
    content_columns = ['Description', 'Notes', ...],
    id_column = 'Id';


INSERT INTO my_kb
    SELECT *
    FROM salesforce_datasource.account a
    JOIN postgresql_datasource.customer_int c
    ON a.`Id` = c.`AccountId`;

Learn more about knowledge bases here.

Resources

**MCP Toolbox GitHub repo: **https://github.com/googleapis/genai-toolbox
MindsDB in MCP Toolbox: https://github.com/googleapis/genai-toolbox/blob/main/docs/en/resources/tools/mindsdb/_index.md
MindsDB Documentation: https://docs.mindsdb.com/
MindsDB GitHub:https://github.com/mindsdb/mindsdb

Streamline Financial Analysis with MindsDB’s Knowledge Bases and Hybrid Search

MindsDB Team — Mon, 29 Dec 2025 20:29:37 +0000

Written by Chandre Van Der Westhuizen, Community & Marketing Co-ordinator at MindsDB

Finance teams are expected to deliver real-time, audit-ready insights — but their data still lives across disconnected systems like Salesforce, ERP, and spreadsheets. This creates delays, compliance risks, and manual effort in reporting and reconciliation. MindsDB solves this by unifying all financial data in place — no ETL, no copying — through AI-native Knowledge Bases, Hybrid Search, and Agents. The result: a single, explainable layer for querying invoices, orders, opportunities, and financial reports in real time. Finance leaders can now ask natural questions, trace every answer to source data, and ensure decisions are both accurate and compliant — all from one secure, intelligent platform.

Finance teams at publicly traded B2B companies juggle dozens of systems—from financial reports and CRM dashboards to invoices, shipments, and contracts.
Each tells part of the story, but the real insight lies across them:

How does revenue growth relate to delayed shipments or unpaid invoices?
Which large enterprise accounts contribute most to quarterly performance?
What risks could impact the next earnings call?

Answering these questions required ETL pipelines, manual dashboards, and delayed reporting. With MindsDB, you can now build AI-native, zero-ETL analytics that query this data directly—powered by Knowledge Bases, Hybrid Search, and Agents.

The Challenge

In many organizations, financial data lives across disconnected systems:

Sales and deal data in Salesforce Opportunities
Contract and delivery data in Orders and Shipments
Billing and payments in Invoices
Consolidated revenue in Financial Statements

This fragmentation creates revenue recognition risk, audit gaps, and delayed reporting, especially when:

Orders are fulfilled before contracts (POs) are signed.
Invoices are partially paid or overdue.
Revenue appears in financial reports without matching delivery evidence.

Traditional ETL workflows are slow and error-prone, making real-time reporting and compliance impossible.

The MindsDB Advantage: Real-Time, Explainable AI for Finance Teams

MindsDB transforms how finance teams access, analyze, and trust their data — eliminating the need for manual aggregation, spreadsheets, and disconnected systems.

Unified, Zero-ETL Access to All Financial Data:

MindsDB connects directly to your existing systems — Salesforce, ERP, CRM, spreadsheets, and financial databases — without the need for ETL. This means your Opportunities, Orders, Invoices, Shipments, and Financial Statements can be analyzed together, privately and securely, without moving data out of its source. All unified by MindsDB’s Knowledge Bases.

Ask Complex Questions in SQL or Plain English — Privately and Securely:

Ask complex financial questions in SQL and natural language and get accurate, evidence-backed answers in seconds. Instead of manually combining spreadsheets or exporting CSVs, MindsDB’s Hybrid Search and AI Agents query and reason across your data — ensuring every response is grounded in real numbers and systems you already trust.

The Result: Faster, Smarter, and More Trustworthy Financial Operations

With MindsDB, finance teams gain a single, AI-native layer that unifies data access, reasoning, and compliance:

Financial reporting and audit checks across systems
Eliminate ETL and manual spreadsheet aggregation
Achieve real-time, explainable insights grounded in verifiable data
Empower teams to focus on decision-making, not data wrangling

In short, for finance teams operating within public B2B companies, MindsDB offers a data-native, AI-first approach: bringing intelligence to your data rather than moving your data to intelligence. The result is faster, reliable, conversational analytics that tie together finance, CRM, logistics and operations into one cohesive view.

Use Case: Real-Time Revenue Validation Across Salesforce and Financial Systems

MindsDB enables finance teams to query, analyze, and validate financial reporting accuracy by connecting operational Salesforce data (CRM, Orders, Invoices, Shipments) directly with official financial statements — without manual reconciliation or ETL.

For this use case, we will explore gaining insights into Salesforce CRM Data and Financial Reports for the fiscal year 2025 from Q1-Q3. We will connect this data to MindsDB, unify it using MindsDB Knowledge Bases, query it using Hybrid Search and Agents.

Pre-requisites:

Access MindsDB’s GUI via Docker locally or MindsDB’s extension on Docker Desktop.
Configure your default models in the MindsDB GUI by navigating to Settings → Models.

MindsDB’s Federated query engine allows you to connect directly to Salesforce using SQL. The CREATE DATABASE statement will be used:

CREATE DATABASE salesforce_datasource
WITH
   ENGINE = 'salesforce',
   PARAMETERS = {
       "username": "chandre-bsbv@force.com",
       "password": "xxxx",
       "client_id": "3MVG9SiMaxxxxxx",
       "client_secret": "047CE0DB7AB8834FBxxxxxx"
   };

A database connection to PGVector will be created to use as storage for the Knowledge Base embeddings:

CREATE DATABASE heroku
WITH
    ENGINE = 'pgvector',
    PARAMETERS = {
    "host": "127.0.0.1",
    "port": 5432,
    "database": "postgres",
    "user": "user",
    "password": "password",
    "distance": "cosine"
    };

For the sake of this tutorial, we have uploaded the Financial Report 2025 dataset as a file. You can check out how to Upload Files in our GUI here.

Unifying Your Data By Building Knowledge Bases

You can unify your data using MindsDB Knowledge Bases. A Knowledge Base in MindsDB is an AI-powered table that understands data by meaning, not just keywords — combining embeddings, reranking models, and vector stores for context-aware retrieval.

It enables semantic reasoning across multiple data sources, providing deeper, more accurate insights for intelligent data access. Here we will create Knowledge Bases for our Shipments, Invoices, Orders and Opportunities Salesforce CRM tables, as well as the Financial Report 2025 Spreadsheet.

Start with creating the knowledge base using the CREATE KNOWLEDGE_BASE statement:

CREATE KNOWLEDGE_BASE orders_kb
USING
storage = heroku.orders,
metadata_columns = ['account number', 'activated by', 'activated byt', 'activated date', 'company authorized by', 'company authorized date', 'contract end date', 'contract number', 'contract name', 'contract end date', 'billingaddress', 'created by', 'currency', 'customer authorized by', 'customer authorized date', 'last modified by', 'opportunity', 'order amount', 'order end date', 'order name', 'order number', 'order record type','order reference number', 'order start date', 'order type', 'owner', 'PO date', 'PO number', 'quote', 'reduction order', 'shipping city', 'shipping country', 'shipping street', 'ship to contact' ],
content_columns = ['description', 'status'],
id_column = 'account name';

Here are the parameters provided:

orders_kb: The name of the knowledge base.
storage : The storage table where the embeddings of the knowledge base is stored. As you can see we are using the pgvector database we created a connection with and provide the name orders to the table that will be created for storage.
metadata_columns : Here columns are provided as meta data columns to perform metadata filtering.
content_columns : Here columns are provided for semantic search.
id_column: This uniquely identifies each source data row in the knowledge base.

We can Insert the data using the INSERT INTO statement:

INSERT INTO orders_kb
SELECT `account number` as 'account number', `activated by` as 'activated by', 
  `activated date` as 'activated date', `company authorized by` as 'company authorized by', 
  `company authorized date` as 'company authorized date', `contract end date` as 'contract end date', 
  `contract number` as 'contract number', `contract name` as 'contract name', `contract end date` as 'contract end date', 
  `created by` as 'created by', currency, `customer authorized by` as 'customer authorized by', `customer authorized date` as 'customer authorized date', 
  `last modified by` as 'last modified by', opportunity, `order amount` as 'order amount', `order end date` as 'order end date', `order name` as 'rder name', 
  `order number` as 'order number', `order record type` as 'order record type',`order reference number` as 'order reference number`', `order start date` as 'order start date', 
  `order type` as 'order type', owner, `PO date` as 'PO date', `PO number` as 'PO number', quote, `reduction order` as 'reduction order', `shipping city` as 'shipping city', `shipping country` as 'shipping country', 
  `shipping street` as 'shipping street', `ship to contact` as 'ship to contact', description, status, `account name` as 'account name'
FROM salesforce_datasource.orders;

We can query the Knowledge Base using the SELECT statement:

SELECT * FROM orders_kb;

The same can be done for the Financial Report:

-- Create Knowledge Base
CREATE KNOWLEDGE_BASE financial_report2025
USING
storage = heroku.financial_report,
metadata_columns = ['net_revenue_millions', 'operating_income_millions', 'net_income_millions', 'operating_margin_percent', 'net_margin_percent', 
                    'earnings_per_share', 'expenses_millions', 'r_and_d_expenses_millions', 'sales_and_marketing_expenses_millions', 'general_and_admin_expenses_millions', 
                    'cash_flow_operations_millions', 'total_assets_millions', 'total_liabilities_millions', 'shareholders_equity_millions'],
content_columns = ['month_name'],
id_column = 'quarter';


--Insert Into Knowledge Base
INSERT INTO financial_report2025
SELECT net_revenue_millions, operating_income_millions, net_income_millions, operating_margin_percent,
net_margin_percent, earnings_per_share, expenses_millions, r_and_d_expenses_millions, sales_and_marketing_expenses_millions,
general_and_admin_expenses_millions, cash_flow_operations_millions, total_assets_millions, total_liabilities_millions, 
  shareholders_equity_millions, month_name, quarter
FROM files.financial_report_2025;


--Select Knowledge Base
SELECT * FROM financial_report2025;

The following Knowledge Bases were also created using the above steps:

Shipments_kb : contains shipment data hosted in Salesforce.

invoices_kb : contains invoice details hosted in Salesforce.

opportunities_kb: contains opportunities data hosted in Salesforce.

Hybrid Search for Finance: Bridging Operational Data and Financial Reporting with AI

MindsDB’s Hybrid Search lets you combine semantic and structured filters for deeper insights.

Lets identify overdue invoices tied to key accounts that might affect cash flow or require write-offs.

-- Detect Invoices with Long Outstanding Balances
SELECT *
FROM invoices_kb
WHERE
 hybrid_search = TRUE
 AND hybrid_search_alpha >= 0.5
 AND content = 'payment delays or pending collection'
 AND `days outstanding` > 45;

This helps finance teams flag overdue receivables, especially important for SOX Section 404 compliance and aging report validation.

Lets try to correlate Orders and Invoices for Revenue Reconciliation to Ensure every billed order is represented as a recorded invoice, and verify consistency between order_amount and impact_amount.

-- Correlate Orders and Invoices for Revenue Reconciliation
SELECT *
FROM orders_kb
JOIN invoices_kb
ON orders_kb.id = invoices_kb.id
WHERE
 orders_kb.hybrid_search = TRUE
 AND invoices_kb.hybrid_search = TRUE
 AND orders_kb.hybrid_search_alpha >= 0.5
 AND invoices_kb.hybrid_search_alpha >= 0.5
 AND orders_kb.content = 'order fulfillment completed'
 AND invoices_kb.content = 'invoice posted and awaiting payment';

We can see customers like Acme Corp 12 and Finserve partners have activated fulfilled orders.

And that their orders are Partially paid or the Invoice is locked due to the payment method being mismatched and it requires action. This supports audit traceability from “Order → Invoice → Payment,” ensuring recognized revenue matches delivery obligations

You can validate if forecasted opportunities have corresponding orders within the same quarter — key for forecast reliability.

-- Cross-Join Opportunities and Orders for Forecast Accuracy
SELECT *
FROM opportunities_kb
JOIN orders_kb
ON opportunities_kb.id = orders_kb.id
WHERE
 opportunities_kb.hybrid_search = TRUE
 AND orders_kb.hybrid_search = TRUE
 AND opportunities_kb.hybrid_search_alpha >= 0.5
 AND orders_kb.hybrid_search_alpha >= 0.5
 AND opportunities_kb.content = 'enterprise expansion opportunity'
 AND orders_kb.content = 'active or pending fulfillment'
 AND opportunities_kb.forecast_category IN ('Commit', 'Best Case');

This helps financial controllers confirm that pipeline forecasts are grounded in real order creation, improving rolling forecast accuracy and budget variance analysis.

We can identify customers like Smart Corp and Acme Corp have an expansion to Enterprise tier Opportunity.

And they have active orders that are partially fulfilled.

We can create a full end-to-end trace from CRM opportunity → booked order → issued invoice and identify accounts where the revenue chain is incomplete.

--Compliance Audit Trail Across All Three — Opportunities, Orders & Invoices
SELECT *
FROM opportunities_kb
JOIN orders_kb
 ON opportunities_kb.id = orders_kb.id
JOIN invoices_kb
 ON opportunities_kb.id = invoices_kb.id
WHERE
 opportunities_kb.hybrid_search = TRUE
 AND orders_kb.hybrid_search = TRUE
 AND invoices_kb.hybrid_search = TRUE
 AND opportunities_kb.hybrid_search_alpha >= 0.5
 AND orders_kb.hybrid_search_alpha >= 0.5
 AND invoices_kb.hybrid_search_alpha >= 0.5
 AND opportunities_kb.content = 'PO awaiting signature'
 AND orders_kb.content = 'activated fulfilled, partially fulfilled'
 AND invoices_kb.content = 'partially paid';

There has been orders which have been partially fulfilled. A control gap exists when fulfillment starts before contract or payment completion, creating risks of premature revenue and misstated receivables.

The order is also partially invoiced.Overstated pipeline and delayed payments distort forecasts and increase collection risk, requiring potential reserve adjustments.

Obtain Insights From CRM Data and Financial Statements In Natural Language Using MindsDB Agents

By creating a MindsDB Agent connected to multiple Knowledge Bases, finance teams gain a single, intelligent interface to ask natural language questions about the entire revenue cycle.

The agent can be created by using the CREATE AGENT statement:

CREATE AGENT financial_reporting_agent
USING
data = {
"knowledge_bases": ['shipments_kb', 'orders_kb', 'invoices_kb', 'opportunities_kb', 'financial_report2025']
},
prompt_template =
"
You are a Financial Data Analysis Agent connected to Salesforce and finance datasets stored in MindsDB Knowledge Bases.


Your purpose is to help finance, compliance, and audit teams analyze and interpret data across:
- `shipments_kb` → shipment and logistics data
- `orders_kb` → Salesforce order and fulfillment data
- `invoices_kb` → Salesforce invoices and payment data
- `opportunities_kb` → Salesforce CRM pipeline and deal data
- `financial_report2025` → official financial statements for fiscal year 2025 (Q1 till Q3)


### Your Reasoning Context


1. **Date and Quarter Understanding**
  - All datasets use date formats in `YYYY-MM-DD` (e.g., 2025-09-29).
  - You can parse and interpret these dates to determine the correct month and quarter.
  - Map months to quarters as follows:
    - Q1 = January (01), February (02), March (03)
    - Q2 = April (04), May (05), June (06)
    - Q3 = July (07), August (08), September (09)
    - Q4 = October (10), November (11), December (12)
  - Fiscal Year 2025 = January 2025 to December 2025.
  - Understand temporal phrases such as:
    - “This quarter” → the current quarter in 2025 based on the date field.
    - “Last quarter” → the quarter immediately before the current quarter.
    - “Year-to-date (YTD)” → all months from January up to the current month in 2025.
    - “Previous month” → the month before the most recent date in the dataset.


2. **Revenue recognition context**
  - Revenue can only be recognized once a contract is enforceable (PO signed) and performance obligations are fulfilled (order delivered or service rendered).
  - Closed-won opportunities indicate potential revenue, but only fulfilled and invoiced orders should be recognized in financial statements.
  - Partially fulfilled or partially paid orders indicate deferred or unrecognized revenue.


3. **Compliance and audit understanding**
  - Identify mismatches between CRM, billing, and financial statements.
  - Highlight potential risks like premature revenue, unsigned contracts, or overdue invoices.


4. **Analytical objectives**
  - Reconcile Salesforce opportunities, orders, and invoices with financial statements.
  - Summarize revenue performance, outstanding balances, and fulfillment progress per quarter.
  - Detect control exceptions, data gaps, and compliance risks.
  - Respond to natural language queries about contracts, invoices, orders, payments, shipments, or quarterly financial performance.
     - When users ask about a specific quarter or month, filter data by the date fields.
    - Example: For Q3 2025, include all records from 2025-07-01 to 2025-09-30.
  - When comparing across time, use these date ranges to calculate quarterly or monthly aggregates.
  - If users ask “Compare Q2 and Q3 2025 revenue,” calculate totals based on these date filters.


### Output Style
  - Clearly reference time periods (month, quarter, or fiscal year) when summarizing data.
  - Provide both narrative and tabular summaries if multiple time periods are compared.
  - Always ground insights in date-based logic derived from the dataset fields.


Always ground responses in data retrieved from the provided Knowledge Bases and interpret time periods accurately based on quarterly definitions.
";

Here is the breakdown of the parameters provided to the agent:

financial_reporting_agent : The name provided to the agent
data : This parameter stores data connected to the agent, including knowledge bases and data sources connected to MindsDB.
Knowledge_bases : stores the list of knowledge bases to be used by the agent.
Prompt_template : This parameter stores instructions for the agent. It is recommended to provide data description of the data sources listed in the knowledge_bases parameter to help the agent locate relevant data for answering questions.

MindsDB’s GUI offers a Chat Interface that allows you to chat with your agent and query your data using natural language. To start, make sure the correct agent is selected in the Agent’s tab.

Let's ask the agent multi-layered, compliance-aware finance questions in plain English that blend operational and financial data.

1.Revenue Recognition & Contract Compliance

Question 1: What revenue in Q3 2025 could be overstated due to uncollected or unsigned contracts?

2.Forecasting, Cash Flow & Risk Analysis

Question 1: Forecast expected cash inflow from all partially paid invoices.

Question 2: Which accounts pose the highest risk to quarterly revenue due to incomplete payments or unsigned POs?

3.Insight / Executive Summary Prompts

Question 1: Summarize the overall health of our revenue recognition process for FY2025.

The agent further provides key observations:

Question 2: Provide a summary of Q3 2025 revenue risks and outstanding invoices.

Here is the Summary Table:

4.Financial Statement Reconciliation

Question 1: Reconciled the total ‘Closed Won’ opportunity amounts with the net revenue reported in Q2 2025

Question 2: Which business units have the most incomplete revenue cycles?

Question 3: How do the balances of unpaid invoices compare with the receivables in the financial report for Q3 2025?

Further analysis:

Question 3: List discrepancies between revenue recognized in the financial report and billed amounts in invoices.

Not only did the agent provide an analysis of discrepancies, it also identified a possible data gap or delay in invoicing.

The Outcome: Audit-Ready Intelligence in Real Time

With MindsDB, finance teams can:

Perform financial reporting, reconciliation, and variance analysis
Gain end-to-end visibility across CRM → Orders → Invoices → Reports
Detect compliance gaps before audits
Deliver trustworthy, explainable insights without ETL

MindsDB turns enterprise finance data into a live, conversational, and compliant analytics layer — where every answer is backed by data, traceable, and ready for the boardroom.

Why It Matters

Finance leaders are under growing pressure to deliver accurate, real-time insights while staying compliant with ASC 606, IFRS 15, and SOX requirements. Yet, fragmented systems, manual reconciliations, and spreadsheet-driven workflows slow everything down — increasing audit risk and delaying decisions.

MindsDB changes that.

By unifying financial data from Salesforce, ERP, and reporting systems into a zero-ETL, AI-native layer, MindsDB allows teams to:

Validate revenue recognition before audits catch discrepancies.
Trace every financial insight to source data — improving audit readiness.
Detect control gaps early, such as uncollected invoices or unsigned contracts.
Deliver faster board reporting with transparent, explainable analytics.

In short, this matters because finance no longer has to choose between speed, accuracy, and compliance. MindsDB gives teams all three — in real time, with full transparency.

Conclusion

Financial data moves faster than traditional systems can handle, MindsDB gives finance teams the power to stay ahead — unifying fragmented data into a single, intelligent layer for real-time analysis. By connecting Salesforce, ERP, and reporting systems without ETL, MindsDB turns complexity into clarity, helping teams detect risks, validate revenue, and build confidence in every number they report.

MindsDB offers Minds Enterprise solution, making Enterprise data Intelligent and Responsive with AI. Powered by the Cognitive Engine, Knowledge Base, and Federated Query Engine, it transforms raw data into actionable insights. Build AI Search, Analytics, and Agents seamlessly—all from one solution. Contact our team to see Minds in action.

Whether it’s preparing for the next audit, presenting to the board, or ensuring compliance, MindsDB equips finance teams with real-time, explainable, and trustworthy insights — all backed by their own data.

Build the future of financial intelligence — with MindsDB.

Fast‑Track Knowledge Bases: How to Build Semantic AI Search by Andriy Burkov

MindsDB Team — Mon, 10 Nov 2025 11:33:43 +0000

Written by Andriy Burkov, Ph.D. & Author, MindsDB Advisor

Building a Knowledge Base with MindsDB

Traditional keyword-based search falls short when users don’t know the exact terms in your data or when they ask questions in natural language. Imagine trying to find “movies about an orphaned boy wizard” when the database only contains the word “magic” – a standard SQL query would miss the connection.

This is where knowledge bases with semantic search shine. By understanding the meaning behind queries rather than just matching keywords, they enable:

Natural language queries: Users can ask questions the way they naturally think (“Show me heartwarming family movies with elements of comedy”) instead of constructing complex keyword searches
Contextual understanding: Finding related content even when exact terms don’t match – searching for “artificial intelligence gone wrong” can surface movies about “rogue robots” or “sentient computers”
Metadata-aware filtering: Combine semantic understanding with structured filters (genre, ratings, dates) for precise, relevant results

This tutorial walks through creating a semantic search knowledge base using MindsDB, an open-source platform that brings machine learning capabilities directly to your data layer. MindsDB simplifies the integration of AI models with databases, making it easy to add semantic search, predictions, and other AI features without complex infrastructure.

We’ll use the IMDB Movies Dataset to learn how to upload data to MindsDB, create a knowledge base with embedding models, and perform both semantic and metadata-filtered searches. By the end, you’ll have a working system that can answer questions like “What movie has a boy defending his home on Christmas?”.

To follow along with the tutorial, download the Jupyter Notebook with the code and materials here for you to reuse.

1. Introduction to Knowledge Bases in MindsDB

Knowledge bases in MindsDB provide advanced semantic search capabilities, allowing you to find information based on meaning rather than just keywords. They use embedding models to convert text into vector representations and store them in vector databases for efficient similarity searches.

Let’s begin by setting up our environment and understanding the components of a MindsDB knowledge base.

!pip install mindsdb mindsdb_sdk pandas requests datasets yaspin

Once it is installed you will see this output:

Requirement already satisfied: llvmlite<0.45,>=0.44.0dev0 in /Users/burkov/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from numba->hierarchicalforecast~=0.4.0->mindsdb) (0.44.0)

Requirement already satisfied: et-xmlfile in /Users/burkov/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from openpyxl->mindsdb) (2.0.0)

Requirement already satisfied: psycopg-binary==3.2.9 in /Users/burkov/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from psycopg[binary]->mindsdb) (3.2.9)

Requirement already satisfied: mpmath<1.4,>=1.1.0 in /Users/burkov/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from sympy->onnxruntime>=1.14.1->chromadb~=0.6.3->mindsdb) (1.3.0)



[notice] A new release of pip is available: 25.1.1 -> 25.2

[notice] To update, run: pip install --upgrade pip

2. Dataset Selection and Download

We’ll use the IMDB Movies Dataset from Hugging Face (a popular platform for sharing ML datasets and models), which contains movie information from IMDB (the world’s most comprehensive movie and TV database) including descriptions, genres, ratings, and other metadata - perfect for demonstrating both semantic search and metadata filtering.

# Download IMDB Movies Dataset from Hugging Face
from datasets import load_dataset
import pandas as pd

# Load the dataset
print("Downloading IMDB Movies dataset...")
dataset = load_dataset("jquigl/imdb-genres")
df = pd.DataFrame(dataset["train"])

# Preview the dataset
print(f"Dataset shape: {df.shape}")
df.head()

Upon execution you will receive:

Downloading IMDB Movies dataset...
Dataset shape: (238256, 5)

As you can see, the dataset contains 238,256 movies with descriptive text spanning multiple decades and genres, though some entries have missing ratings (NaN values) that will need to be addressed during data preparation.

Let’s prepare our dataset for MindsDB by cleaning it up and making sure we have a unique ID column:

# Clean up the data and ensure we have a unique ID
# The 'movie title - year' column can serve as a unique identifier
df = df.rename(columns={
    'movie title - year': 'movie_id',
    'expanded-genres': 'expanded_genres',
    'description': 'content'
})

# Clean movie IDs to remove problematic characters
import re
def clean_movie_id(movie_id):
    if pd.isna(movie_id) or movie_id == '':
        return "unknown_movie"

    cleaned = str(movie_id)
    cleaned = re.sub(r"['\"\!\?\(\)\[\]\/\\*]", "", cleaned)
    cleaned = cleaned.replace("&", "and").replace(":", "_")
    cleaned = re.sub(r'\s+', ' ', cleaned).strip()

    return cleaned if cleaned else "unknown_movie"

# Apply the cleaning function to movie_id column
df['movie_id'] = df['movie_id'].apply(clean_movie_id)

# Remove duplicates based on cleaned movie_id, keeping the first occurrence
print(f"Original dataset size: {len(df)}")
df = df.drop_duplicates(subset=['movie_id'], keep='first')
print(f"After removing duplicates: {len(df)}")

# Make sure there are no NaN values
df = df.fillna({
    'movie_id': 'unknown_movie', 
    'genre': 'unknown', 
    'expanded_genres': '',
    'rating': 0.0, 
    'content': ''
})

# Save the prepared dataset
df.to_csv('imdb_movies_prepared.csv', index=False)
print("Dataset prepared and saved to 'imdb_movies_prepared.csv'")
df.head()

You should receive this output once done:

Original dataset size: 238256
After removing duplicates: 161765
Dataset prepared and saved to 'imdb_movies_prepared.csv'

What we’ve accomplished:

We’ve cleaned and prepared our dataset for MindsDB by standardizing column names, sanitizing movie IDs to remove problematic characters (quotes, special symbols, etc.), and handling missing values. Most importantly, we’ve removed duplicate entries - reducing the dataset from 238,256 to 161,765 unique movies. This deduplication is crucial because knowledge bases require unique identifiers for each entry. The cleaned data is now saved as imdb_movies_prepared.csv with properly formatted movie IDs, filled NaN values (rating defaults to 0.0), and consistent column names ready for upload to MindsDB.

With our dataset cleaned and prepared, we’re ready to connect to MindsDB and upload the data.

3. Uploading the Dataset to MindsDB

Now let’s connect to our local MindsDB instance running in a Docker container and upload the dataset. If you don’t have MindsDB Docker container installed, you should follow this simple official installation tutorial. We’ll first establish a connection to the MindsDB server running on localhost, verify the connection by listing available databases, then upload our prepared CSV file to MindsDB’s built-in files database where it will be accessible for creating the knowledge base.

import mindsdb_sdk

# Connect to the MindsDB server
# For local Docker installation, use the default URL
server = mindsdb_sdk.connect('http://127.0.0.1:47334')
print("Connected to MindsDB server")

# List available databases to confirm connection
databases = server.databases.list()
print("Available databases:")
for db in databases:
    print(f"- {db.name}")

In your console you will see:

Connected to MindsDB server
Available databases:
- files
- movies_kb_chromadb

What we’ve accomplished:

We’ve successfully connected to our local MindsDB instance and listed the available databases. Notice the files database - this is a special built-in database in MindsDB specifically designed for uploading and storing datasets (CSV, JSON, Excel files, etc.). We need to use this files database because it acts as a staging area for our data before we can reference it in knowledge base operations.

Now let’s upload our prepared CSV to the files database:

import os, pandas as pd, mindsdb_sdk

# connect
server = mindsdb_sdk.connect("http://127.0.0.1:47334")

# load (or generate) the DataFrame
csv_path  = os.path.abspath("imdb_movies_prepared.csv")
df_movies = pd.read_csv(csv_path)

# upload to the built‑in  `files`  database
files_db   = server.get_database("files")            # <- must be this name
table_name = "movies"

# delete the whole file‑table if it's there
try:
    files_db.tables.drop(table_name)
    print(f"dropped {table_name}")
except Exception:
    pass

files_db.create_table(table_name, df_movies)
print(f"Created table files.{table_name}")

print(
    server.query(
        f"SELECT movie_id, genre, rating FROM files.{table_name} LIMIT 5"
    ).fetch()
)

print(
    server.query(
        f"SELECT count(movie_id) FROM files.{table_name} where rating >= 7.5"
    ).fetch()
)

You will receive this output:

dropped movies
Created table files.movies
                  movie_id      genre  rating
0      Flaming Ears - 1992    Fantasy     6.0
1    Jeg elsker dig - 1957    Romance     5.8
2        Povjerenje - 2021   Thriller     0.0
3  Gulliver Returns - 2021    Fantasy     4.4
4   Prithvi Vallabh - 1924  Biography     0.0
   count_0
0    10152

What we’ve accomplished:

We’ve successfully uploaded our prepared dataset to MindsDB’s files database as a table named movies. The code first drops any existing movies table (ensuring a clean slate for re-runs), then creates a new table from our DataFrame. The sample query confirms our data is accessible - we can see the first 5 movies with their genres and ratings. The count query reveals we have 10,152 movies with ratings of 7.5 or higher.

4. Creating a Knowledge Base

Now, let’s create a knowledge base using our IMDB movies data. To enable semantic search, we need to convert our movie descriptions from plain text into numerical vector representations (embeddings) that capture their semantic meaning. This is where embedding models come in - they transform text into high-dimensional vectors where semantically similar content is positioned closer together in vector space. For example, “a boy wizard learning magic” and “young sorcerer at school” would produce similar vectors even though they share no common words.

We’ll use OpenAI’s text-embedding-3-large model for this task. OpenAI’s embedding models are industry-leading in quality, producing vectors that excel at capturing nuanced semantic relationships. They’re also widely supported, well-documented, and integrate seamlessly with MindsDB. While alternatives like open-source models exist, OpenAI offers an excellent balance of performance, reliability, and ease of use for production applications.

The below code assumes the OpenAI API key was set as a envronment variable in tee MindsDB UI settings. Go to the setting to set it up http://localhost:47334/

Alterntively, you can set it up manaually when starting the container: $ docker run –name mindsdb_container -e OPENAI_API_KEY=‘your_key_here’ -p 47334:47334 -p 47335:47335

If you don’t have an OpenAI API key, you should create one by following these steps.

# -- drop the KB if it exists ----------------------------------------------
server.query("DROP KNOWLEDGE_BASE IF EXISTS movies_kb;").fetch()

# Knowledge Base creation using mindsdb_sdk
try:
    # This assumes the OpenAI key was set as a envronment variable in tee MindsDB UI settings
    # Go to the setting to set it up http://localhost:47334/
    # Alterntively, you can set it up manaually when starting the container:
    # $ docker run --name mindsdb_container \
    # -e OPENAI_API_KEY='your_key_here' -p 47334:47334 -p 47335:47335
    kb_creation_query = server.query(f"""
    CREATE KNOWLEDGE_BASE movies_kb
    USING
        embedding_model = {{
           "provider": "openai",
           "model_name": "text-embedding-3-large"
        }},
        metadata_columns = ['genre', 'expanded_genres', 'rating'],
        content_columns = ['content'],
        id_column = 'movie_id';
    """)
    kb_creation_query.fetch()
    print("Created knowledge base 'movies_kb'")
except Exception as e:
    print(f"Knowledge base creation error or already exists: {e}")

You should receive the output:

Created knowledge base 'movies_kb'

Now let’s insert our movie data into this knowledge base:

from yaspin import yaspin
try:
    with yaspin(text="Inserting data into knowledge base..."):
        insert_query = server.query("""
            INSERT INTO movies_kb
            SELECT movie_id,
                   genre,
                   expanded_genres,
                   rating,
                   content
            FROM   files.movies
            WHERE rating >= 7.5
            USING
                track_column = movie_id
            """).fetch() 
    print("✅ Data inserted successfully!")
except Exception as e:
    print(f"❌ Insert error: {e}")

You should receive the output:

✅ Data inserted successfully!

What’s happening here:

This is where the magic happens - we’re inserting data into our knowledge base, and MindsDB is automatically generating embeddings for each movie’s content using the OpenAI model we specified earlier. We’re filtering for movies with ratings of 7.5 or higher to focus on high-quality films. The track_column = movie_id parameter tells MindsDB to use the movie_id as the unique identifier for tracking and updating entries.

This operation may take a few minutes since it’s making API calls to OpenAI to generate embeddings for thousands of movie descriptions.

We verify the upload by counting the entries in our knowledge base:

row_count_df = server.query("""
    SELECT COUNT(*) AS cnt
    FROM   (SELECT id FROM movies_kb) AS t;
""").fetch()

row_count = int(row_count_df.at[0, 'cnt'])
print(f"✅ movies_kb now contains {row_count:,} rows.")

Your output should be:

✅ movies_kb now contains 10,152 rows.

The 10,152 rows confirm that all highly-rated movies (rating ≥ 7.5) have been successfully embedded and stored. Our knowledge base is now ready for semantic search queries!

Let’s see some data in the knowledge base:

search_query = server.query("SELECT * FROM movies_kb where content='Christmas' order by relevance desc")
display(search_query.fetch())

Let’s see what’s inside a metadata column’s cell:

# Query to get full metadata content
metadata_query = server.query("""
    SELECT id, metadata 
    FROM movies_kb 
    WHERE content='Christmas' 
    ORDER BY relevance DESC 
    LIMIT 5
""").fetch()

# Display full metadata without truncation
import pandas as pd
pd.set_option('display.max_colwidth', None)
display(metadata_query)

# Or print metadata for each row to see the complete JSON structure
print("\nDetailed metadata for top results:")
for idx, row in metadata_query.iterrows():
    print(f"\n{row['id']}:")
    print(row['metadata'])

You will receive the output:

Detailed metadata for top results:

Pixi Post and the Gift Bringers - 2016:
{"_chunk_index": 0, "_content_column": "content", "_end_char": 51, "_original_doc_id": "Pixi Post and the Gift Bringers - 2016", "_original_row_index": "3912", "_source": "TextChunkingPreprocessor", "_start_char": 0, "_updated_at": "2025-10-14 18:17:11", "expanded_genres": "Animation, Adventure, Fantasy", "genre": "Fantasy", "rating": 7.9}

Jingle Vingle the Movie - 2022:
{"_chunk_index": 0, "_content_column": "content", "_end_char": 161, "_original_doc_id": "Jingle Vingle the Movie - 2022", "_original_row_index": "9700", "_source": "TextChunkingPreprocessor", "_start_char": 0, "_updated_at": "2025-10-14 18:17:11", "expanded_genres": "Family", "genre": "Family", "rating": 9.7}

For Unto Us - 2021:
{"_chunk_index": 0, "_content_column": "content", "_end_char": 96, "_original_doc_id": "For Unto Us - 2021", "_original_row_index": "4370", "_source": "TextChunkingPreprocessor", "_start_char": 0, "_updated_at": "2025-10-14 18:17:11", "expanded_genres": "Family", "genre": "Family", "rating": 9.1}

Joyeux Noel - 2005:
{"_chunk_index": 0, "_content_column": "content", "_end_char": 174, "_original_doc_id": "Joyeux Noel - 2005", "_original_row_index": "5021", "_source": "TextChunkingPreprocessor", "_start_char": 0, "_updated_at": "2025-10-14 18:17:11", "expanded_genres": "Drama, History, Music", "genre": "Romance", "rating": 7.7}

Christmas Snow Angels - 2011:
{"_chunk_index": 0, "_content_column": "content", "_end_char": 160, "_original_doc_id": "C

Understanding the metadata structure:

Pixi Post and the Gift Bringers - 2016{"_chunk_index": 0, "_content_column": "content", "_end_char": 51, "_original_doc_id": "Pixi Post and the Gift Bringers - 2016", "_original_row_index": "3912", "_source": "TextChunkingPreprocessor", "_start_char": 0, "_updated_at": "2025-10-14 18:17:11", "expanded_genres": "Animation, Adventure, Fantasy", "genre": "Fantasy", "rating": 7.9}

A chunk is a segment of text created when MindsDB breaks down longer content into smaller, searchable pieces to fit within the embedding model’s input limits. This chunking ensures searches can pinpoint specific passages within larger documents rather than only matching entire documents.

The metadata field contains two types of information:

1.System-generated fields (prefixed with underscores) that MindsDB automatically adds:

_chunk_index: The sequential position of this chunk (0 means it’s the first/only chunk)
_content_column: Which source column contained the text (“content” in our case)
_start_char and _end_char: Character positions showing where this chunk begins and ends in the original text (0 to 51 means a 51-character description)
_original_doc_id: The complete document identifier with content column appended
_original_row_index: The row number from the original dataset (row 3912)
_source: The preprocessor used for chunking (“TextChunkingPreprocessor”)
_updated_at: Timestamp of when this entry was inserted or updated

2.User-defined metadata columns that we specified during knowledge base creation:

genre: “Fantasy” - the primary genre we defined as metadata
expanded_genres: “Animation, Adventure, Fantasy” - the full genre list
rating: 7.9 - the movie’s IMDB rating

This combined metadata enables powerful hybrid search - you can perform semantic searches on content while filtering by structured metadata fields like genre or rating thresholds. For example, you could search for “Christmas adventure stories” and filter only for Animation genre with ratings above 7.5.

5. Performing Semantic Searches with RAG

Now that our knowledge base is populated and indexed, let’s implement a complete Retrieval-Augmented Generation (RAG) workflow. RAG combines semantic search with large language models to answer questions based on your specific data - in our case, movie descriptions.

What is RAG?

RAG is a technique that enhances LLM responses by grounding them in retrieved, relevant documents from your knowledge base. Instead of relying solely on the model’s training data, RAG retrieves the most relevant chunks from your knowledge base and uses them as context for generating answers. This ensures responses are factually accurate and based on your actual data.

The RAG workflow:

import openai
from IPython.display import display

client = openai.OpenAI(api_key=openai_api_key)

def answer_question_with_llm(question: str):
    # 1. Use the existing search_kb function to get the most relevant chunks.
    print(f"Searching knowledge base for: '{question}'\n")

    relevant_chunks_df = search_kb(question, limit=100)

    print("Found the following relevant chunks:")
    display(relevant_chunks_df[['id', 'chunk_content', 'relevance']])

    # 2. Concatenate the 'chunk_content' to form a single context string.
    context = "\n---\n".join(relevant_chunks_df['chunk_content'])

    # 3. Create the prompt for the gpt-4o model.
    prompt = f"""
    You are a movie expert assistant. Based *only* on the following movie summaries (context),
    answer the user's question. If the context doesn't contain the answer,
    state that you cannot answer based on the provided information.

    CONTEXT:
    {context}

    QUESTION:
    {question}
    """

    # 4. Call the OpenAI API to get the answer.
    print("\nSending request to GPT-4o to generate a definitive answer...")
    try:
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": "You are a helpful assistant that answers questions about movies using only the provided context."},
                {"role": "user", "content": prompt}
            ],
            temperature=0.0  # We want a factual answer based on the text
        )
        answer = response.choices[0].message.content
        return answer
    except Exception as e:
        return f"An error occurred while calling the OpenAI API: {e}"

user_question = "Who a boy must defend his home against on Christmas eve?"
final_answer = answer_question_with_llm(user_question)

print("\n--- Generated Answer ---")
print(final_answer)

You should receive the following output:

Searching knowledge base for: 'Who a boy must defend his home against on Christmas eve?'


    SELECT *
    FROM movies_kb
    WHERE content = 'Who a boy must defend his home against on Christmas eve?'
     ORDER BY relevance DESC LIMIT 100;
Found the following relevant chunks:

100 rows × 3 columns

Sending request to GPT-4o to generate a definitive answer...

--- Generated Answer ---
The boy must defend his home against a pair of burglars on Christmas Eve.

What happened here:

This function implements a complete RAG pipeline in four steps:

Semantic Search: The knowledge base is queried with the natural language question. MindsDB’s semantic search retrieves the top 100 most relevant movie chunks, ranked by relevance score. Notice how the search found “Home Alone” (relevance: 0.687) even though our question didn’t mention the movie title - semantic search understood the meaning of “boy defending home on Christmas.”
Context Assembly: All retrieved chunk contents are concatenated into a single context string, separated by dividers. This context now contains relevant information from multiple movies.
Prompt Engineering: We construct a carefully crafted prompt that instructs GPT-4o to act as a movie expert and answer only based on the provided context. This grounding reduces the chance that the model will hallucinate or use information outside our knowledge base.
LLM Generation: The OpenAI API processes the prompt with temperature=0.0 (deterministic, factual responses) and generates an answer by synthesizing information from the retrieved chunks.

The power of RAG:

The final answer - “The boy must defend his home against a pair of burglars on Christmas Eve” - demonstrates RAG’s strength. The LLM successfully:

Identified the most relevant movie (Home Alone) from the semantic search results
Extracted the key information about burglars from the movie description
Synthesized a clear, concise answer grounded in our actual data

This RAG approach ensures answers are (almost) always based on your knowledge base rather than the model’s general training data, making it a great solution for domain-specific applications like customer support, internal documentation systems, or specialized research assistants.

Let’s wrap up this tutorial with one more query:

user_question = "What Anakin was lured into by Chancellor Palpatine?"
final_answer = answer_question_with_llm(user_question)

print("\n--- Generated Answer ---")
print(final_answer)

The output:

Searching knowledge base for: 'What Anakin was lured into by Chancellor Palpatine?'

    SELECT *
    FROM movies_kb
    WHERE content = 'What Anakin was lured into by Chancellor Palpatine?'
     ORDER BY relevance DESC LIMIT 100;
Found the following relevant chunks:

100 rows × 3 columns


Sending request to GPT-4o to generate a definitive answer...

--- Generated Answer ---
Anakin was lured by Chancellor Palpatine into a sinister plot to rule the galaxy.

Conclusion

Congratulations! You’ve successfully built a semantic search knowledge base with MindsDB that can understand and answer natural language questions about movies. Let’s recap what we’ve accomplished:

What you’ve built:

A knowledge base containing 10,152 high-quality movies with semantic embeddings
A complete RAG (Retrieval-Augmented Generation) pipeline that combines semantic search with LLM-powered question answering
A system that understands meaning, not just keywords - finding “Home Alone” when asked about “a boy defending his home on Christmas” and “Star Wars Episode III” when queried about “Anakin and Chancellor Palpatine”

Key takeaways:

Semantic search transcends keywords: MindsDB’s knowledge bases use embeddings to understand the meaning behind queries, enabling more intuitive and natural search experiences
Hybrid search combines the best of both worlds: By integrating semantic understanding with metadata filtering (genre, ratings, etc.), you can create powerful, precise queries
RAG grounds AI in your data: Instead of relying on potentially outdated or hallucinated information, RAG ensures answers are most of the time (LLMs always hallucinate, but grounding them in your factual data reduces hallucinations to a great extent) based on your actual knowledge base

Where to go from here:

This tutorial demonstrated the fundamentals, but knowledge bases can power much more:

Build chatbots that answer questions about your company’s documentation, policies, or products
Create recommendation systems that understand user preferences semantically
Develop research assistants for academic papers, legal documents, or technical manuals
Scale to production by connecting MindsDB to your existing databases, APIs, or data warehouses

The beauty of MindsDB is that you control it entirely through SQL - a familiar interface that makes advanced AI capabilities accessible without complex infrastructure or learning new APIs or domain-specific programming languages. Whether you’re working with customer support tickets, research papers, code repositories, or movie databases, the same principles apply.

Now it’s your turn to build something amazing with your own data!

You can also check out the full webinar, Fast‑Track Knowledge Bases: How to Build Semantic AI Search.

The Future of Data-Native & Grounded AI with MindsDB: Moving Data to Models No Longer Makes Sense

MindsDB Team — Mon, 20 Oct 2025 09:49:55 +0000

Written by Chandre Van Der Westhuizen, Community & Marketing Co-ordinator at MindsDB

What does complex ETL and slow decision making have in common? Without proper AI implementation, it results in delayed access to critical insights, leading to missed opportunities, reduced agility, and lost revenue.

The default strategy for AI and analytics has been to move data to the model—centralizing it in warehouses, cleaning it through complex ETL pipelines, and then running models in isolation. But in an era where data is distributed, compliance is strict, and real-time insights are a necessity, this old approach is breaking down.

Enter data-native AI: a new paradigm where models go to the data instead. Instead of duplicating, transforming, and moving data across systems, data-native AI enables models to interact with information directly where it lives.

AI has also become vital for decision-making, a fundamental question keeps coming up: Can we trust the answers our AI gives us?

The answer hinges on a critical principle: grounding.

Grounded AI refers to models and agents that base their outputs on real, verifiable, and up-to-date data. Without this foundation, even the most advanced language models can hallucinate, mislead, or provide contextually irrelevant responses. And in high-stakes environments like finance, retail, or enterprise operations, that’s simply unacceptable.

The Case Against Moving Data

Moving data might seem like a necessary step in modern data workflows—but it’s often more harmful than helpful. Before you spin up another ETL pipeline or copy data into yet another system, consider the real costs. Here’s why relocating data can do more damage than good:

Latency and Freshness: By the time data is moved, cleaned, and transformed, it’s often stale. Real-time decision-making suffers as a result.
Cost and Complexity: ETL pipelines, data lakes, and duplicated storage come with steep infrastructure and maintenance costs.
Security and Compliance: Copying data increases the risk of exposure and makes it harder to enforce data governance policies. With regulations like GDPR and HIPAA, moving sensitive data unnecessarily is a liability.
Siloed Context: Context is often lost when data is stripped from its original environment and schema. This weakens model accuracy and relevance.

The Trust Gap in Traditional AI

Large language models are powerful, but they weren’t designed to know your business. They don’t understand the nuances of your financial reports, CRM notes, or policy documentation—unless you give them that context.

Most traditional AI workflows rely on static snapshots of data, embeddings, or manually curated inputs. This introduces a lag between what your AI "knows" and what’s actually happening. The result?

Outdated answers
Missing critical changes
Reduced trust in AI recommendations

What Is Data-Native AI?

Data-native AI flips the model by allowing AI to interact with data directly at its source—whether that's a SQL database, an API, a document repository, or an enterprise SaaS system. Instead of forcing data into a model's format, the model adapts to the native environment of the data.

Key features include:

Federated query engines that enable real-time data access
Agent-based architectures that query across systems
Security boundaries respected by default
Reduced need for data replication or transformation

Why Real-Time Access Matters

Real-time data access ensures that AI decisions are:

Contextual: Based on the latest state of your business
Verifiable: Easy to trace back to the exact source
Accurate: Reflecting current customer behavior, market changes, or system states

In regulated industries, it also means being able to show auditors where the AI got its answers and why they were reasonable at that time.

How MindsDB Delivers Grounded AI and the Data-Native AI Advantage

MindsDB is built from the ground up to support a data-native approach and enables grounded AI by allowing agents and LLMs to query your live structured and unstructured data sources—from SQL and NoSQL databases to PDFs, APIs, and SaaS tools. It doesn’t stop there, you can unify your data with the power of MindsDB’s Knowledge Bases.

With MindsDB, you can:

Seamlessly query multiple live data sources in real time
Ensure LLM outputs are grounded in current, explainable information
Eliminate the overhead of ETL processes and avoid redundant data storage
Uphold security and compliance by keeping data within trusted environments
Enable federated, real-time access to both structured and unstructured data
Leverage configurable agent tools to define accessible data and operational constraints
Deliver transparent, auditable outputs with citations, source documents, and metadata

Instead of relying on a static knowledge base, MindsDB-powered agents interact directly with the source of truth.

Use Case: Solving The Problem for E-commerce Stores with Siloed data

Lets take a use case for an E-commerce store that has siloed data. The goal would be to unify data from their online store with customer, sales, orders and products data to gain insights to make business decisions.

Problem: Moving, transforming, and duplicating data for AI creates high infrastructure and maintenance costs. Traditional pipelines rely on periodic data syncs, leading to stale insights.

Solution: MindsDB enables in-place querying, eliminating the need for complex ETL processes and redundant storage. Knowledge Bases provide real-time access to structured and unstructured data—ensuring decisions are based on the latest available information.

Pre-requisites:

Access MindsDB’s GUI via Docker locally or MindsDB’s extension on Docker Desktop.
Configure your default models in the MindsDB GUI by navigating to Settings → Models.
Add your data to MindsDB by creating a database connection or uploading your files.

First, a connection to the store's Postgres database, where all the data is stored in separate tables, will be made to MindsDB using the Postgres Integration and CREATE DATABASE statement. This will give you real-time access to your data in MindsDB:

CREATE DATABASE postgresql_conn
WITH ENGINE = 'postgres',
PARAMETERS = {
   "user": "demo_user",
   "password": "demo_password",
   "host": "samples.mindsdb.com",
   "port": "5432",
   "database": "demo",
   "schema": "sample_data"
};

Now a connection will be created between PGVector and MindsDB which will be used as a storage for our Knowledge Bases.:

CREATE DATABASE pvec
WITH
    ENGINE = 'pgvector',
    PARAMETERS = {
    "host": "127.0.0.1",
    "port": 5432,
    "database": "postgres",
    "user": "user",
    "password": "password",
    "distance": "cosine"
    };

Lets create a Knowledge Base using the CREATE KNOWLEDGE_BASE statement:

CREATE KNOWLEDGE_BASE sales_kb
USING
storage = heroku.sales_storage,
  metadata_columns = ['product_id', 'order_id', 'customer_id', 'order_date', 'ship_date'],
  content_columns = ['category', 'sub_category', 'product_name', 'ship_mode', 'customer_name', 'segment', 'country', 'city', 'region', 'sales', 'quantity', 'discount', 'profit']
;

Here is a breakdown of the parameters:

sales_kb: The name of the knowledgebase.
storage : The storage table where the embeddings of the knowledge base is stored.
metadata_columns : Here columns are provided as meta data columns to perform metadata filtering.
content_columns : Here columns are provided for semantic search The id_column has not been provided, therefore it will be generated from the hash of the content columns.

Now that the Knowledge Base is created, we can insert data into it. The goal is to unify multiple tables in one Knowledge Base. To do so, we will join the tables in the INSERT INTO statement:

 INSERT INTO sales_kb
SELECT a.product_id, a.order_id, a.customer_id, order_date, ship_date, category, sub_category, product_name, ship_mode, customer_name, segment, country, city, region, sales, quantity, discount, profit
FROM postgresql_conn.websales_sales AS a
LEFT JOIN postgresql_conn.websales_customers as b ON a.customer_id = b.customer_id
INNER JOIN postgresql_conn.websales_orders as c ON a.order_id = c.order_id
RIGHT JOIN postgresql_conn.websales_products as d ON a.product_id = d.product_id;

You can create a MindsDB Agent with the Knowledge Base and query it to gain insights using the CREATE AGENT statement:

CREATE AGENT sales_agent
USING
   data = {
        "knowledge_bases": ["mindsdb.sales_kb"]
   },
   prompt_template='
       mindsdb.sales_kb stores data about sales, products sold via e-commerce, customer information and orders placed in the online store';

Here is a breakdown of the parameters:

sales_agent: The name provided to the agent
data: This parameter holds the data linked to the agent, including knowledge bases and data sources integrated with MindsDB.
- knowledge_bases: Stores the list of knowledge bases
prompt_template: This parameter stores instructions for the agent and description of data. It is recommended to provide data description of the data sources listed in the knowledge_bases and tables parameters to help the agent locate relevant data for answering questions.

The agent is ready to be queried and you can gain insights on shipping, customer, product and sales and profit analysis.

Lets start with asking the agent how many customers belong to a specific segment.

SELECT answer
FROM sales_agent
WHERE question = 'How many customers belong to a specific segment?';

This shows the size and value of each customer group, helping businesses tailor strategies, allocate resources, and measure growth.

You can ask which orders drive the most profit, enabling sales teams and executives to focus on replicating high-value transactions and refining pricing or strategy for maximum impact.

SELECT answer
FROM sales_agent
WHERE question = 'Can you show me the top 5 most profitable transactions';

You can query the agent to reveal which products deliver the greatest profitability, guiding smarter decisions on pricing, promotions, and resource prioritization.

SELECT answer
FROM sales_agent
WHERE question = 'Which products have the highest profit margins? ';

The agent can also help track logistics efficiency, uncover customer preferences, and manage shipping costs for better supply chain decisions.

SELECT answer
FROM sales_agent
WHERE question = 'How many orders were shipped using a specific shipping mode';

Lets try to highlight overall performance patterns, helping businesses understand growth, seasonality, and shifts in customer demand.

SELECT answer
FROM sales_agent
WHERE question = 'What are the sales trends over the past year?';

Lets try to reveal the trade-off between higher sales volume and reduced profit margins, guiding smarter discounting and pricing strategies.

SELECT answer
FROM sales_agent
WHERE question = 'How does the discount offered affect the sales and profit?';

Real-World Impact

In industries like finance, energy, retail, and enterprise software vendors, MindsDB unlocks:

Faster insights: Real-time portfolio analysis or fraud detection
Smarter automation: AI that reacts to operational data without lag
More trust: Decisions backed by up-to-date, in-context data

Further use cases include:

Customer support: Agents reference current policy documents and CRM history to give personalized, accurate answers
Financial modeling: AI uses real-time transaction data and risk metrics to suggest actions
Compliance: Agents flag violations based on up-to-date regulatory documents and behavior logs

Final Thoughts

It used to make sense to move data to models—back when computing was scarce and data was centralized. But today, data is everywhere and speed matters. The future belongs to AI systems that go to the data, enabling real-time access without the delays of duplication or transfer. To stay competitive, AI must adapt to the data—not the other way around.

Data-native and grounded AI is faster, more secure, and better suited for the dynamic, distributed environments of modern enterprises.

With MindsDB, that future is already here. Contact our team to see our solution in action.