DEV Community

Manthan Rajpurohit
Manthan Rajpurohit

Posted on

I Built a GitHub Connector for Cognee So My AI Agent Remembers All My Repos

TL;DR: For Mergetober I built a data-source connector that syncs GitHub repos, issues, PRs, commits, and releases into Cognee's AI memory layer. 6 tests green, 46 real rows pulled from my live GitHub account. Here's the full story — including the bug that produced zero error messages.

The Problem: My AI Has Amnesia

I have repos everywhere — my portfolio, experiments, half-finished ideas, tutorial clones I swear I'll finish. When I want to know "where did I implement OAuth?" or "what did I fix last month?", I dig through GitHub manually like it's 2015.

Meanwhile my AI assistant is smart but has total amnesia. Every conversation starts from zero. It can't know what I built, what broke, or what I learned — unless I paste it all in by hand, every single time.

That's the gap Cognee fills: it's an open-source memory layer for AI agents. You feed it your data, it builds a knowledge graph, and your agents can actually remember things. But Cognee only shipped with five data-source connectors — Notion, Slack, Gmail, Google Drive, Confluence. GitHub, the place where developers actually live, was missing.

So for Mergetober, I built it.

What I Built

A GitHub data-source connector for Cognee — proposed in issue #5363, implemented in PR #228:

  • 🔑 Auth via personal access token (passed as an argument or via GITHUB_TOKEN)
  • 📦 Ingests repositories, issues, pull requests, commits, and releases
  • 🆔 Stable IDs like issue:owner/repo#42 — unchanged items are never re-processed
  • 🔄 Full-snapshot sync — deleted or privatized items drop out and Cognee's orphan cleanup forgets them, so the agent's memory stays honest
  • ⏱️ Pagination via GitHub's Link headers, with retries on 429/5xx honoring Retry-After and X-RateLimit-Reset

How It Works

The connector mirrors the shape of Cognee's existing Notion connector. It's a dlt source containing one resource:

@dlt.resource(name="github_documents", primary_key="id", write_disposition="replace")
def github_documents():
    ...
Enter fullscreen mode Exit fullscreen mode

Every item GitHub returns gets flattened into a document row of exactly {id, url, title, content}, with the content rendered as markdown — an issue becomes its title plus body plus all its comments; a commit becomes its message plus the list of changed files (never full diffs — otherwise the graph just mirrors the repo).

One line opts the whole thing into Cognee's document pipeline:

setattr(source, DOCUMENT_SOURCE_ATTR, "github")
Enter fullscreen mode Exit fullscreen mode

That marker is what routes rows through Cognee's entity-extraction (cognify) instead of the relational path — the right treatment for prose-like content.

Using it is three lines:

from cognee_community_connector_github import github_source
import cognee

source = github_source(token="...", repos=["myuser/myrepo"])
await cognee.remember(source, dataset_name="github")
Enter fullscreen mode Exit fullscreen mode

Scope it with repos=[...] or orgs=[...]; omit both and it ingests every repo the token can see.

The Testing Story: 6 Green, 1 War

I wrote two layers of tests, all runnable with zero live API calls:

  • 5 unit tests — fake dicts in, asserting every row builder returns the exact {id, url, title, content} shape with stable IDs.
  • 1 pipeline test — a fake HTTP client injected into a real dlt pipeline writing to temp SQLite, proving re-syncs pick up edits and deleted items vanish from the snapshot (forget-on-delete).

That pipeline test gave me the nastiest bug of the project — and it produced zero error messages. It kept failing with no such table: github_documents. After a long investigation I learned the real cause: dlt only creates a table when at least one row flows through it, and my test was silently yielding zero rows. Two culprits:

  • Leftover debug print statements that crashed — one referenced an undefined variable, another called len() on a method instead of a string.
  • A fake API response shaped as a list where the real GitHub endpoint returns a dict.

Both exceptions were swallowed by a per-repo try/except I'd written "for resilience."

Lesson learned: resilience that hides bugs isn't resilience. Make failure loud during development; be resilient in production.

Second lesson: stable IDs are the entire game for incremental sync. No cursor files, no timestamp bookkeeping — if the ID and content hash haven't changed, skip it. Simple beats clever.

Proof: 46 Real Rows

Unit tests are necessary but not sufficient — the Mergetober bar is "works against a real account, not just on paper." So I generated a real personal access token, pointed the connector at my live GitHub account, and pulled 46 real rows: my portfolio repo plus 45 real commits with their real messages. No mocks, no fixtures.

That's the moment it stopped being a coding exercise and became a real integration.

What's Next

Ideas I'm considering if the maintainers want them:

  • Incremental since-based fetching to cut API calls on huge repos
  • Syncing GitHub Discussions and Release assets
  • An orgs-wide dashboard example showing team activity memory

Try It Yourself

  • 💡 Proposal: issue #5363
  • 🔧 Implementation: PR #228
  • 📦 Code: packages/connector/github/ in cognee-community

If you want your agent to remember your GitHub too, that's where to start. And if you're doing Mergetober — pick the integration nobody else is building, verify everything against the real API, and make your failures loud. Good luck. 🚀

Top comments (0)