Mergetober: when updated_at isn't enough, building a Crisp connector for cognee

Crisp chats are short and there are lots of them: a visitor asks something, an operator answers, maybe someone leaves an internal note. One message on its own ("thanks!") means nothing, so the issue asked for one node per conversation, with incremental sync on the conversation's updated_at.

Before proposing anything for Mergetober, I tested that idea on a free Crisp workspace. It turned out updated_at misses most kinds of change. Here's the clearest one.

• Pull request: topoteretes/cognee-community#589
• Issue: topoteretes/cognee#4737

The demo: an edit that Crisp's timestamp doesn't see

I created a conversation where a customer, Priya, reports that order #4471 arrived with a cracked screen, and the operator replies that a replacement "ships on Monday". Priya says thanks. I synced the workspace into cognee (OpenAI as the LLM) and asked:

When will Priya's replacement order ship, and what was wrong with her order?

Priya's replacement order will ship on Monday. Her original order arrived with a cracked screen.

Then I edited the operator's old reply, "Monday" to "Thursday", the way an operator fixes a mistake after the fact. No new message, nothing else touched.

And here's the important part: the conversation's updated_at, the field the issue suggested for incremental sync, before and after the edit, and what cognee answered on the next sync:

updated_at didn't move: 21:25:00 before the edit, 21:25:00 after it. A connector that only follows it would never re-read this conversation, and the agent would keep telling people "Monday" for good. This one re-reads recently active conversations every run, so the next sync picked the edit up (scanned 1):

Priya's replacement order will ship on Thursday. Her original order (#4471) arrived with a cracked screen.

One more thing I noticed: in an earlier run I made the same edit in the Crisp app instead of through the API, and that time updated_at did move. I can't tell whether the app edit or simply opening the conversation as an operator moved it, which is exactly why the connector doesn't depend on it.

What moves updated_at, and what doesn't

Change	updated_at moved?	Visible in the conversation list?
New message	yes	yes
Edit a message, old or newest (through the API)	no	no
Edit an internal note	no	no
Remove a message	no, even the newest one	no: the list still shows the removed reply as last_message
Visitor nickname or segments	no	yes, in meta
Resolve	no	yes, in state
Remove the conversation	– (GET and HEAD answer 404)	it drops out of the list

A connector that only follows updated_at would only ever see new messages. Edits and removals would stay in memory for good.

How it syncs instead

• Every run reads the full conversation list (50 a page, with include_empty=1, because empty conversations are hidden otherwise). A history is read again when its updated_at, state or visitor details changed.
• Edits and removals leave no trace outside the history, so each run also re-reads conversations active in the last few days, plus a rotating batch of older ones. Unchanged conversations are skipped by fingerprint. An edit in a recent conversation lands on the next sync, and one in an older conversation within one rotation.
• A stored conversation missing from the list is checked with HEAD and forgotten on a 404. A made-up id answers 404 too, so this only applies to conversations synced before, and only after the whole list was read.
• Requests have a budget. A website token gets 10,000 requests a day. Each run stops cleanly at its own budget, and Crisp reports no request counters, only x-rate-limit-watermarks: g=low;u=low;r=low, so the client pauses when one reads high.

Setting it up

A website token (identifier and key) from Settings → Workspace Settings → Advanced configuration, and the website id. The Free plan is enough: the connector only reads conversations, and reading works on Free. (Only creating internal notes through the API needs a paid plan, which matters for testing, not for syncing.)

import cognee
from cognee_community_connector_crisp import crisp_source

await cognee.remember(
    crisp_source(website_id=website_id, identifier=identifier, key=key),
    dataset_name="crisp",
    primary_key="id",
    write_disposition="merge",
    max_rows_per_table=0,
)

Each conversation renders as Visitor (name), Operator (name) and Internal note (name) lines in time order, with edited messages in their current form. State changes and other event messages are left out. visitor_fields chooses which visitor details go in the header. The default is the nickname and segments; email, phone and IP only reach a document if you list them.

Live run

On my Crisp workspace, through a real dlt merge:

Step	Result
First sync	both test conversations stored (with two older ones: scanned: 4)
An old message edited, nothing else	caught by the recent window: scanned: 1, the edit in the document
The visitor renamed	seen in the list: scanned: 1, the new name in the document
A message removed from A, conversation B removed	scanned: 1, deleted: 1
Nothing changed	scanned: 0, deleted: 0

One honest gap: my workspace's trial ended just before the final run, and the Free plan refuses to create internal notes through the API (HTTP 402, subscription_upgrade_required). I had tested notes live while writing the proposal, and the tests cover note labelling and include_notes=False. The live table above just doesn't include one.

Checking my own work

Before opening the pull request, I went through my proposal line by line against the code. Two things I had promised weren't there, and the check turned up one bug:

• Backing off when a rate-limit watermark reads high. The client handled 420 and 429, but ignored the watermark header I had written about myself. It now pauses before the next request.
• Visitor fields as config. The nickname and segments were fixed. visitor_fields now chooses them, with email, phone and IP off unless you list them.
• A conversation removed in the middle of a sync answered 404 when its history was read, and that failed the whole run. Now it's skipped, and the next sync forgets it.

Each fix has a test that fails on the old code. Two smaller lessons came from preparing the live checks:

• The first draft of my live-check script edited an old message and renamed the visitor in the same step. The rename alone forces a re-read, so that step couldn't have shown the edit was caught. I split it before running it: a check has to change one thing at a time.
• My demo script printed the measured updated_at times, but next to them a fixed sentence saying they hadn't moved. In the run where I edited in the app, they had. The script now compares them and says which happened.

What I learned

Measure, don't assume, even your own findings. My proposal said editing a message doesn't move updated_at. That held when I edited through the API, but not when I made the same edit in the Crisp app. So the connector doesn't trust updated_at for edits at all: it re-reads what changed recently and lets the fingerprint decide what's really new.

The setup steps are in the README in the pull request, and examples/example.py remembers a workspace and asks it a question.