Mergetober: keeping an agent in sync with a Sanity CMS, with cognee

A headless CMS like Sanity is where a company's published knowledge lives: articles, authors, product pages, docs. An agent that answers questions about that content needs to stay in step with it: when an article is edited, unpublished or deleted, its answers should change too.

For Mergetober I built a Sanity connector for cognee.

• Pull request: topoteretes/cognee-community#460
• Issue: topoteretes/cognee#4788

The demo: a rename that doesn't touch the article

I wrote an author, Priya Raman, and two articles that reference her: "Who owns billing" and "Q4 migration plan". Then I synced them into cognee (OpenAI as the LLM) and asked:

Who wrote the article 'Who owns billing', and who should billing incidents be escalated to?

Priya Raman wrote it. Billing incidents should be escalated to the Payments on-call engineer within 30 minutes.

Then I renamed the author to Priya Iyer (through the API: my test project has no Studio deployed, and the connector doesn’t need one). Only the author document changed. Here's what Sanity says about the billing article itself, before and after the rename, and what the next sync did:

The article's revision id (SfcI2UfJhIU74ahVojMxr6) and its _updatedAt are exactly the same before and after. To any sync that re-reads changed documents, by _rev or by _updatedAt, nothing happened to it, and its document in memory would keep saying "Priya Raman" for good. This connector tracks the revisions of the documents each article references too, so the author's new revision re-rendered both articles that point to her (changed 1, reference_updates 2), and the same question now gets:

Priya Iyer wrote it. Billing incidents should be escalated to the Payments on-call engineer within 30 minutes.

Setting it up

A read-only Viewer token is enough. You choose which document types to sync, because a dataset also holds system and plugin documents you don't want in memory:

import cognee
from cognee_community_connector_sanity import sanity_source

source = sanity_source(
    project_id="abc123xy",
    dataset="production",
    document_types=["article", "author"],
    token=SANITY_TOKEN,
)
await cognee.remember(
    source,
    dataset_name="sanity_content",
    primary_key="id",
    write_disposition="merge",
    max_rows_per_table=0,
)

Each published document becomes one cognee document: its fields as Field: value lines, Portable Text as paragraphs, references as the referenced document's name or title, and images as their alt text and caption. The connector also reads the project's deployed schema and ingests it as one document that describes the selected types and their fields, so the agent knows what an "article" is. An optional groq_filter narrows the selection further, for example to one language or one section of the site.

No timestamp cursor

My plan on the issue was a _updatedAt cursor. I dropped it. To forget deleted documents, the connector needs the full list of ids anyway, and with each document's _rev (its revision id) in that list, the list already says what changed. That avoids the problem of several writes landing in the same second, and there's no timestamp to store. It costs one request per 1,000 documents.

How references are followed

Each document's references are rendered by name: an article shows Author: Priya Iyer, not an id. That's what makes the rename above a problem in the first place. So the connector keeps, for each synced document, the revisions of the documents it references. When one of those changes or is deleted, every document pointing to it is re-rendered. In the first live check, renaming an author re-rendered all four articles that referenced her (reference_updates: 4).

An empty listing never forgets on its own

This is the safety rule I care most about in this connector. In the live run I found that a private dataset read without a working token doesn't return an error. It returns no documents. So does a mistyped document type. A connector that trusts that listing would forget everything.

If a run lists nothing but earlier runs synced documents, the connector raises and forgets nothing. allow_empty_listing=True is there for a real "everything was deleted". Both cases ran live, and the table was unchanged both times:

8a. document_types mistyped as 'artcle'
   -> stopped: guard held, rows in table: 5 before, 5 after
8b. no token, dataset set to private
   -> stopped: guard held, rows in table: 5 before, 5 after

The schema document follows the same rule: only a token can read it, even in a public dataset, so "no schema" may just mean the token stopped working. Turning include_schema off is what forgets it.

The rest of the live run

(Step 1 was the first sync, step 3 the author rename covered above, and step 8 the empty-listing guard.)

2. edit + new article in ONE transaction (same _updatedAt): both appear
4. unpublished draft of article 1: nothing changes
5. article 3 deleted                 deleted: 1
6. no changes: nothing fetched
7. deployed schema document: found

What I learned

A document can change without changing. The text an agent reads is built from the article and everything it points to, so "has this document changed?" is the wrong question. The right one is "has anything this document shows changed?" I now ask that about every connector: what does the rendered text depend on, and does the sync notice when any of it moves?

One more thing I checked: cognee.forget(dataset=...) alone doesn't remove synced documents for good, because dlt keeps its staging copy and the next sync brings them back. The README says so: to forget a document, delete or unpublish it in Sanity, or drop its type.

The setup steps are in the README in the pull request, and examples/example.py syncs a dataset and asks it a question.