Teaching an AI assistant to forget what you unpublish in Webflow

Picture this.

Your marketing team unpublishes a case study on Friday. The client asked for it. On Monday your AI assistant is still recommending that case study to people. It sounds completely sure. It is completely wrong.

Nothing crashed. The assistant just kept a memory of a page that is no longer there.

I wanted to fix that for Webflow. So for Mergetober I built a connector for Cognee. The code lives in PR #413 for issue #4791.

What it does

Cognee turns documents into a knowledge graph that an agent can search. The connector feeds it from a Webflow site. You pick the CMS collections. Then:

• Every item becomes a short document.
• Every collection gets a summary page that lists its fields and what they mean.
• Later runs send only what changed.
• Items that are deleted, unpublished or archived are forgotten.
• Static pages are optional.

The summary page is the part I like most. A field called Service tells a model very little. A summary that says "Service is one of Development, Branding, Web design" tells it a lot. Option fields store ids inside Webflow. The connector looks the names up so the document says Service: Branding and not a random id.

source = webflow_source(
    collections=["works"],  # slugs, ids or names. Leave it out for every collection.
    include_pages=True,
)
await cognee.remember(source, dataset_name="webflow",
                      primary_key="id", write_disposition="merge")

The token comes from WEBFLOW_API_TOKEN. A site token works. So does an OAuth access token.

Finding what changed

Webflow's list endpoints return whole items. They do not tell you what was deleted. If an item is gone, it is just missing from the list.

So every run lists the selected collections, 100 items per request. The connector keeps each item's lastUpdated from the last run. An item is sent on only if it is new or that stamp changed. Everything else is skipped, so the LLM never reads it twice.

One small trick helps here. If lastUpdated moves but the text does not, the connector sends that item again. But the document it builds is identical, so Cognee's content hash skips it. No LLM call.

Catching deletes

If an item was there last time and is missing now, the connector removes it from the graph. The same goes for a deleted collection. Unpublished and archived items count as gone too.

I added three safety rules because a wrong delete is worse than a missed one:

• A failed request never deletes anything. The run stops and the next one starts from the same spot.
• Offset pagination can shift while you read it. If someone deletes an item in the middle of a run, you can skip a neighbour and think it vanished. So a suspected delete is confirmed by reading the collection a second time.
• A page the token cannot read is skipped and kept. Unreadable is not the same as deleted.

The real test

First I tested against a fake. It is a small in-memory copy of the Webflow API with real pagination and a 429 rate limit. 93 tests passed, including one that runs the whole Cognee pipeline with a mocked LLM. But a fake only proves what you thought to fake.

So I generated a site token for a real Webflow template site and ran the example.

The first sync took in 7 items and 6 pages. The CMS template page is skipped on purpose. I asked what the site is about and the answer described it correctly. Running it again with nothing changed sent 0 items and made no LLM calls. Turning pages off forgot all 6 of them. The graph went from 121 nodes to 66.

Then I made real changes in Webflow's CMS. I edited one item's summary to "Rebranded for Zephyr Motors". I deleted one item. I added a new one called Test Item. I published and ran one more sync.

Sync stats: {'items': 3, 'unchanged': 10, 'deleted': 1, 'pages': 0, ...}

Three items were sent: the edit, the new item and one item that had been a draft until I published the site. Ten were unchanged: four items and six pages. One was forgotten. The graph now knew about Zephyr Motors and Test Item. The deleted item showed up in zero nodes.

The bug only a real site could find

My first live call failed with 409 Conflict: The site is not published.

Webflow has no live items until a site is published once. My fake never did that, because I never thought to ask. My template site had never been published.

The tempting fix is to treat that answer as "no items". That would have forgotten everything in the graph. Instead the connector now stops and tells you to publish the site, or to pass include_drafts=True to read the staged items. A test covers it.

What I did not test

Two things only have fake-API tests. One is unpublishing an item without deleting it. The other is a real 429 from Webflow, because you have to go over your plan's per-minute limit to trigger one. I would rather say that than hide it.

Reference fields and files are not turned into text because they only hold ids. Secondary-language versions of an item are not synced separately either. The README lists these.

Try it

The example is in packages/connector/webflow/examples/example.py. A Webflow site and a site token are enough. Set WEBFLOW_INCLUDE_DRAFTS=1 if you have not published it yet.

If you point it at your own site I would love to hear what breaks.