(a Mergetober story about cognee, file sync, and the most polite 401 error ive ever met.)
the issue that warned me
for Mergetober i picked issue #4765 on cognee: add an Egnyte data-source connector.
the issue was very honest with me. it said the difficulty was Advanced. it said Egnyte has "a trial that requires sales contact." it said, in plain english, "claim only if you already have access."
i did not have access. i claimed it anyway. this is what we in the industry call character development.
what is cognee, and what is Egnyte?
Cognee turns your data into memory for AI agents. You feed it documents, and it builds a knowledge graph you can search and ask questions about.
Egnyte is where a lot of companies keep their files. think shared folders full of contracts, invoices, and a file called final_v3_ACTUAL_final.docx.
A connector glues the two together. Point it at an Egnyte folder and your files become cognee memory. Change a file and the memory updates. delete a file and cognee forgets it, which is more than i can say for my brain and that one embarrassing thing from 2019.
act 1: the code (the easy part, apparently)
the connector is a dlt resource you hand straight to cognee:
await cognee.remember(
egnyte_source(domain="acme", folder_paths=["/Shared/Projects"]),
dataset_name="egnyte_projects",
primary_key="id",
write_disposition="merge",
)
a few design decisions im proud of:
files are keyed by Egnyte's group_id, not their path. the group_id survives new versions, renames and moves. so when someone renames notes.txt to notes_FINAL.txt, cognee updates the same memory instead of panicking and making a twin.
incremental sync with the Events API. the first run walks every folder. after that, the connector only asks Egnyte "what changed since last time?" and re-checks just those paths.
forget-on-delete. deleted files, and files moved out of the synced folder, get a _deleted marker, and cognee removes them from its data and its graph.
the folder trap. when you delete a whole folder, Egnyte sends exactly one event: "folder gone." it does not list the 400 files that were inside it. so the connector keeps a map of every file it knows, and uses it to work out what just vanished.
a typo never wipes your data. if you misspell a folder name, the connector raises an error. it doesnt decide the folder is "empty" and helpfully forget everything you ever synced from it.
i wrote 61 tests against a fake Egnyte that behaves like the real one. everything passed. i felt unstoppable.
then i tried to get an actual API key.
act 2: the emails (the hard part, definitely)
heres the saga, reconstructed from my inbox.
email 1-2: i sign up for an Egnyte trial. trial domains cant create API keys. cool, cool, cool.
email 3-4: i find Egnyte's developer sandbox form and fill it out with my gmail address. Egnyte replies, very kindly, that the admin needs a business email. my gmail and i take a moment to reflect on our life choices.
email 5-6: i resubmit with a business email, borrowed with permission from a grown-up who has a real job. i wait. i refresh my inbox like it owes me money.
email 7: developer account approved! i find the "Create API Key" button. i get a client ID and a client secret. i put them in my .env file, run the login, and Egnyte says:
{"message": "No valid app info found for api key"}
email 8: i explain this to support. i get a helpful reply about the app certification process, which takes up to three weeks and is for publishing your app to the world. i was not trying to publish my app to the world. i was trying to read one text file.
email 9: i write back to clarify. then, just to be sure i wasnt losing my mind, i tried logging in with a client ID i made up by mashing the keyboard. Egnyte gave the exact same error. my real key and asdkjhqwe were, as far as Egnyte was concerned, the same thing. i put this in the email. it felt good.
email 10: "thanks for clarifying. you should be approved now."
email 11: it still didnt work. i sent the client ID and the exact error again.
email 12: "try now."
i tried. Egnyte showed me a login page instead of an error. i have never been so happy to see a login form.
final score: about a week of code, about two weeks of email, one day before the deadline. to be clear, Egnyte's support team was patient and kind the whole way through. enterprise APIs are just built for enterprises, and i was one guy with a laptop and a dream.
act 3: the live test
with a working key, i did the thing my fake Egnyte had been pretending to be all along. i uploaded a .txt, a .md, a .pdf, a .docx, and a subfolder, synced them into cognee, and then abused them:
what i did in Egnyte what cognee did
uploaded a new version same memory, new content
renamed a file same memory, new path
renamed a whole folder files inside kept their memory
moved a file out of the folder forgot it (data and graph)
deleted a file forgot it
deleted a whole subfolder forgot everything inside it
did nothing also did nothing, as is tradition
9 out of 9 passed on the first live run. after weeks of emails, the code just worked, which was almost suspicious.
what i learned
read the warning labels. "claim only if you already have access" is not a suggestion. (id still do it again.)
fakes are only as good as your assumptions. i wrote my fake Egnyte from the docs, then compared it to real API responses. luckily the field names and event shapes matched. if they hadnt, 61 green tests would have been 61 lies.
when an error is confusing, test it with garbage. sending a made-up key turned "maybe im doing something wrong" into "this key is not switched on," which is a much easier email to write.
deletion is the hard part of sync. adding files is easy. knowing that a file is gone, especially when a whole folder disappears in one event, is where the real work is.
links
the PR: https://github.com/topoteretes/cognee-community/pull/577
the issue: topoteretes/cognee#4765
cognee: github.com/topoteretes/cognee
thanks to the cognee maintainers for the hackathon, to WeMakeDevs for Mergetober, and to Egnyte's developer support for answering every single one of my emails. sorry about the inbox.
Top comments (0)