Mergetober: asking our team chat why we rolled back a deploy, with cognee and Mattermost

Teams that self-host Mattermost usually do it because their conversations can't live in a SaaS tool. Those conversations are where the real decisions are made: why a deploy was rolled back, what caused last Monday's outage, what the fix should be. A few weeks later, the only way to find that out is to scroll through channel history.

For Mergetober I built a Mattermost connector for cognee, so an agent can answer those questions from the team's own threads.

• Pull request: topoteretes/cognee-community#479
• Issue: topoteretes/cognee#5470

The use case

Say an incidents channel holds two threads. One is about rolling back the payments deploy, the other about a VPN outage. I want to ask "What did the team decide, and why?" and get an answer built from those threads. When a thread is deleted in Mattermost, I want the answer to stop using it.

Setting it up

Mattermost turns personal access tokens off by default, so a system admin enables them once under System Console → Integrations → Integration Management. Then:

1. Create a token under Profile → Security → Personal Access Tokens. A dedicated account that is only a member of the channels you want to sync keeps the token's reach small.
2. Find each channel's id. Click the channel name to open its Info panel; the id is the ID: line. The connector takes ids, not names, because channels get renamed.

The connector is one call into cognee:

import cognee
from cognee_community_connector_mattermost import mattermost_source

source = mattermost_source(
    "https://chat.example.com",
    token=token,
    channels=[channel_id],
)
await cognee.remember(
    source,
    dataset_name="mattermost_chat",
    primary_key="id",
    write_disposition="merge",
    max_rows_per_table=0,
)

answers = await cognee.recall(
    "What did the team decide, and why?",
    datasets=["mattermost_chat"],
)

write_disposition="merge" matters: after the first run, the connector only returns the threads that changed, so the default replace would wipe the rest.

What cognee answered

I ran it against a local Mattermost server (the official mattermost-preview Docker image) with OpenAI as the LLM. The first sync stored both threads, and cognee answered:

The team rolled back the payments deploy at 14:20 because the new fraud rules bypassed the cache, sending every payment to the database and making card authorisations three times slower; the fix is to put the rules behind the cache.

That answer comes from four messages by different people in one thread. That is why the connector stores one document per thread rather than one per message: a reply like "agreed, ship it" means nothing on its own.

Deleting a thread

Then I deleted the rollback thread in Mattermost and ran the same script again. The sync reported deleted: 1, and the same question now got a different answer:

They renewed the VPN gateway's expired certificate at 09:40 because it had caused the office VPN outage.

The rollback thread is gone from cognee's memory, not just hidden. For chat data this matters a lot: people delete messages for a reason, and an agent that keeps quoting them is a problem.

How it keeps up with the channel

Mattermost's since parameter returns every post changed after a timestamp, including deleted ones, which come back with delete_at set. That makes it a real change feed:

• A new reply or an edit bumps the thread's root post, so the connector re-reads just that thread.
• A deleted root becomes a tombstone, and cognee removes the thread's document from the graph.
• When nothing changed, the run reads nothing (scanned: 0).

A few things I only found by testing against a real server:

• Every edit leaves a hidden copy of the old version, which comes back as an already-deleted post with its own id. Treating it as a deletion would forget things on every edit.
• since is exclusive, and two posts can share a millisecond. The connector starts its window 1 ms early and skips posts it has already seen.
• since returns at most 1,000 posts and can't be paged. After a busy weekend, a connector that just moved its cursor forward would silently lose the rest. When since comes back full, this one re-reads the whole channel instead. I tested that live: 1,050 new posts plus a deleted thread in one interval gave scanned: 1050, deleted: 1, with all 1,051 live threads stored.

Safety choices

• A 403 or 404 never deletes anything. A channel the token lost access to keeps its memory, and the sync fails loudly after three runs in a row.
• An archived channel stays in memory and stops syncing, unless you pass forget_archived=True.
• The token is only sent to the server you configure, over https, and redirects are not followed.
• Documents name people by display name, never by email address.

What I learned

Ask every API what it does when there's too much. since looked like the perfect change feed until I asked what happens after a busy weekend: it stops at 1,000 posts, can't be paged, and says nothing about the rest. A connector that trusted it would have lost data silently, and only on the busiest days, the ones people ask about later. The fix was cheap once I knew, and the only way to know was to post 1,050 messages and look.

What's next

Very long threads are re-processed in full when they grow. Splitting them into windows would make that cheaper, and it's the first thing I'd add.

If your team runs Mattermost, the setup steps are in the README (on the PR branch until it's merged), and examples/example.py is the script I used for this post.