Teaching an AI agent to forget a deleted Redshift row

Here is a small horror story.

An AI agent reads your warehouse. Someone deletes a customer. A week later the agent is still talking about that customer. It sounds completely sure. It is completely wrong.

Nothing crashed. The agent just kept a memory of a row that no longer exists.

I wanted to fix that for Amazon Redshift. So for Mergetober I built a connector for Cognee. The code lives in PR #403 for issue #4773.

I had never used Redshift before this. I wanted to see how a warehouse really works under the hood. Building a connector seemed like a good excuse.

What it does

Cognee turns documents into a knowledge graph that an agent can search. The connector feeds it from Redshift. You pick the tables or SQL queries. Then:

• Every row becomes a short document.
• Every table gets a summary page built from its comments.
• Later runs read only what changed.
• Deleted rows are forgotten.

The summary page is my favourite part. Say a column is called priority. On its own that tells the model nothing. Add the comment "p1 = outage, p4 = question" and it knows what a p1 ticket is.

source = redshift_source(
    [RedshiftTable("public.support_tickets", primary_key="ticket_id", cursor_column="updated_at")],
    connection=DataApiConnection(database="dev", workgroup_name="cognee-test"),
)
await cognee.remember(source, dataset_name="redshift",
                      primary_key="id", write_disposition="merge")

Getting in without a tunnel

Redshift usually hides in a private network. The classic trick is an SSH tunnel through a bastion host. It works. It is also a chore.

So I used the Data API instead. It is a plain HTTPS endpoint. You send a statement. You wait until it is done. You read the result pages. AWS runs it inside your network for you.

No tunnel. No driver. No open port. It works with IAM or a Secrets Manager secret. A direct driver connection is still there as a backup.

Catching deletes

Redshift has no change feed. Nobody tells you a row is gone.

So every run also asks for the primary keys. That is one column and a columnar store reads it fast. If a key was there last time and is missing now then the row is gone. The connector removes it from the graph.

Two safety rules keep this honest:

• A failed query never deletes anything. The run stops and the next one starts from the same spot.
• A row that arrives late with an old timestamp is still found. The key list catches it.

The real test

First I tested against fakes. A SQLite stand-in. A stubbed AWS API. A real Postgres. 59 tests passed. But a fake only proves what you thought to fake.

So I created a real Redshift Serverless workgroup in eu-north-1 and filled a small ticket table.

The first sync took in 6 rows. I asked what the table holds. The answer even explained what p1 means. That came straight from the column comment in Redshift.

Then I changed ticket 3 and deleted ticket 2 right inside Redshift. The second sync reported rows: 3, deleted: 3. The deleted ticket's customer vanished from both the documents and the graph. The other two stayed.

That is the behaviour I wanted.

The bug only a real cluster could find

My first run failed with Redshift endpoint doesn't exist in this region.

My docs told people to set AWS_REGION. But boto3 does not read that variable. It reads AWS_DEFAULT_REGION or your profile. My profile said us-east-1 and my cluster lived in eu-north-1. No fake would ever have shown me this.

The connector now honours AWS_REGION and a test guards it.

What I did not test

The direct connection through an SSH or SSM tunnel only has unit tests. The Data API is the default so that is the path I ran live. I would rather say that than hide it.

Try it

The example and a seed file live in packages/connector/redshift/. Redshift Serverless has a free trial for new users so a short test costs very little. Delete the workgroup when you finish. It keeps billing until you do.

If you point it at your own warehouse I would love to hear what breaks.