Asking my CI "why did it fail?" — a Jenkins connector for cognee

I'm a developer from Vietnam, and this is my contribution to WeMakeDevs Mergetober: a new data-source connector for cognee.

The problem

When a Jenkins build fails, the reason is usually a few lines at the bottom of a long console log.
A week later nobody remembers which job broke, or why.

I wanted to ask my CI a plain question — "which build failed last night, and why?" — and get an
answer with the exact error line.

What I built

A Jenkins data-source connector for cognee, an open-source
AI memory engine. It syncs Jenkins jobs and builds into cognee so you can search them in plain language.

• PR: https://github.com/topoteretes/cognee-community/pull/296
• Issue: https://github.com/topoteretes/cognee/issues/4714

It stores one record per job and one per build: result, time, duration, what started it, and for
failed builds the end of the console log.

How it works

• Small API calls. Every request to Jenkins asks only for the fields it needs (tree=), so a big Jenkins server never sends its whole object graph. Logs are streamed and only the last 64 KB is kept.
• Incremental sync. For each job it remembers the highest build number it has ingested. The next run only fetches newer builds. A build that is still running is skipped until it finishes.
• Forget on delete. If you delete a job, or Jenkins discards old builds, the next sync removes them from cognee's memory too.
• Careful by default. If a sync suddenly sees zero jobs, it deletes nothing (that is more likely a wrong folder or a permission problem than a real wipe). Values that look like secrets in logs (DEPLOY_TOKEN=…, password: …) are masked, and parameter default values are never read.

Trying it on a real Jenkins

I ran Jenkins locally in Docker with three small jobs: one that passes, one in a folder, and a
"nightly deploy" that fails and prints a fake token on purpose.

Then I synced it into cognee and asked what failed. The answer named the job and the exact error,
and the stored log shows the token masked:

Next I deleted nightly-deploy in Jenkins and synced again. The connector sent three deletions (the
job and its two builds), and cognee no longer knew about the failure:

What surprised me

• The unit tests passed, but the first run against a real Jenkins showed that DEPLOY_TOKEN=supersecret123 was not masked: the check only matched the word token on its own, not inside a name like DEPLOY_TOKEN or AWS_SECRET_ACCESS_KEY. I fixed it and added a test. Testing against the real thing was worth it.
• Asking cognee the exact same question twice can return a cached answer, so to check that the deleted job was really forgotten I asked a differently worded question.

How I worked

I used Claude Code as a coding assistant to write and test the connector, and I reviewed the code
before opening the PR. The tests and the local Jenkins check above are part of the PR description.

Try it

docker run --rm -p 8080:8080 jenkins/jenkins:lts-jdk21

Then follow the README in packages/connector/jenkins/ of the PR.

Feedback on the PR is very welcome.

Comments

[Nic Flemmer]

Asking CI why a build failed, instead of scrolling the log yourself, is a good use for a connector. What was the hardest part to get right: pulling the logs out of Jenkins, or deciding what the model should see of them?

[ai_adam]

Thấy approach "hỏi CI tại sao fail" rất thực tế — thay vì đọc log ngàn dòng, dev chỉ cần hỏi câu hỏi tự nhiên. Jenkins là cái khó nhất để parse vì log format không chuẩn, plugin tùy biến nhiều, stage/step lồng nhau phức tạp.

Một vài điểm mình quan tâm nếu dùng thực tế:

1. Context window: Log Jenkins build lớn dễ vượt token limit. Có strategy nào chunk/summarize log trước khi feed vào LLM không? (Ví dụ: chỉ lấy tail 200 lines + các stage failed + error pattern known)

2. False positive: LLM hay hallucinate nguyên nhân root cause khi log thiếu context (network flaky, resource limit, dependency conflict). Có mechanism validate lại kết quả không — ví dụ cross-check với known failure patterns hoặc run reproducer?

3. Incremental learning: Team thường có tribal knowledge về failure types riêng (custom plugin, internal library). Có cho phép fine-tune / few-shot với failure history của team không?

4. Latency vs cost: Query mỗi lần fail tốn token + thời gian. Có cache embedding cho common failure patterns không?

Mình đang maintain một internal tool tương tự cho GitLab CI, pain point lớn nhất là log noise (ansi color, progress bar, parallel job interleaving). Pre-processing log trước khi RAG quan trọng không kém phần retrieval — found it via LabAgent, site: labagent .tech