Teaching an AI assistant to forget what you unpublish in Webflow

It is Friday afternoon. A client emails your marketing team: "Please take our case study down."

Someone unpublishes it in Webflow. Two clicks. Done.

On Monday a visitor asks your AI assistant for a good example of your work. The assistant recommends that exact case study. It describes it warmly. It is completely sure.

The page is gone. The memory of it is not.

I wanted to fix that. So for Mergetober I built a Webflow connector for Cognee. The code lives in PR #413 for issue #4791.

What it does

Cognee turns documents into a knowledge graph that an agent can search. The connector feeds it from a Webflow site. You pick the CMS collections. Then:

• Every item becomes a short document.
• Every collection gets a summary page that lists its fields and what they mean.
• Later runs send only what changed.
• Items that are deleted, unpublished or archived are forgotten.
• Static pages are optional.

The summary page is my favourite part. A field called Service tells a model very little. A summary that says "Service is one of Development, Branding, Web design" tells it a lot.

There is a small catch. Option fields store ids inside Webflow. So the connector looks the names up. The document says Service: Branding, not a1b2c3.

source = webflow_source(
    collections=["works"],  # slugs, ids or names. Leave it out for every collection.
    include_pages=True,
)
await cognee.remember(source, dataset_name="webflow",
                      primary_key="id", write_disposition="merge")

The token comes from WEBFLOW_API_TOKEN. A site token works. So does an OAuth access token.

Three gates before the LLM

LLM calls cost money and time. So the connector is stubborn about when it lets one happen.

Webflow's list endpoints return whole items. They never say what was deleted. If an item is gone, it is simply missing from the list. So every run lists the selected collections, 100 items per request, and puts each item through three gates.

Gate two is my favourite trick. Sometimes lastUpdated moves but the text does not. Somebody saved without changing anything. The connector sends the item on. But the document it builds is identical, so Cognee's content hash stops it. No LLM call.

Catching deletes

If an item was there last time and is missing now, the connector removes it from the graph. A deleted collection works the same way. Unpublished and archived items count as gone too.

A wrong delete is worse than a missed one, so I added three safety rules:

• A failed request never deletes anything. The run stops and the next one starts from the same spot.
• Offset pagination can shift while you read it. If someone deletes an item mid-run, you can skip its neighbour and think that one vanished too. So a suspected delete is confirmed by reading the collection a second time.
• A page the token cannot read is skipped and kept. Unreadable is not the same as deleted.

The real test

I tested against a fake first. It is a small in-memory copy of the Webflow API, with real pagination and a 429 rate limit. 93 tests passed, including one that runs the whole Cognee pipeline with a mocked LLM.

Then I stopped trusting my fake. I generated a site token for a real Webflow template site and ran the example. Here is how it went.

I asked what the site is about. The answer described it correctly. Turning pages off forgot all 6 of them, and the graph shrank from 121 nodes to 66.

Then I made real changes in Webflow's CMS. I edited one item's summary to "Rebranded for Zephyr Motors". I deleted one item. I added a new one called Test Item. I published and ran one more sync.

Sync stats: {'items': 3, 'unchanged': 10, 'deleted': 1, 'pages': 0, ...}

Three items were sent: the edit, the new item, and one that had been a draft until I published. Ten were unchanged. One was forgotten. The graph now knew about Zephyr Motors and Test Item. The deleted item showed up in zero nodes.

The bug only a real site could find

My first live call failed with 409 Conflict: The site is not published.

Webflow has no live items until a site is published once. My fake never did that, because I never thought to ask. My template site had never been published.

The tempting fix is to treat that answer as "no items". That would have made the connector forget everything it knew. Instead it now stops and tells you to publish the site, or to pass include_drafts=True to read staged items. A test covers it.

What I did not test

Two things only have fake-API tests. One is unpublishing an item without deleting it. The other is a real 429 from Webflow. You have to blow past your plan's per-minute limit to get one.

Reference fields and files are not turned into text, because they only hold ids. Secondary-language versions of an item are not synced separately either. The README lists these.

Try it

The example is in packages/connector/webflow/examples/example.py. A Webflow site and a site token are enough. Set WEBFLOW_INCLUDE_DRAFTS=1 if you have not published yet.

Point it at your own site. Then unpublish something and see if your assistant lets go.