DEV Community

Akshat Gupta
Akshat Gupta

Posted on

Teaching an AI assistant to forget a deleted DocuSign agreement

#mergetober #opensource #ai #python

Picture this.

Someone moves an old vendor agreement to the Deleted folder in DocuSign. The terms were replaced months ago. A week later your AI assistant is still quoting the old payment terms, completely sure of itself.

Nothing crashed. The assistant just remembers a contract that is gone.

For Mergetober I built a DocuSign connector for Cognee to fix that. The code is in PR for issue #5716.

What it does

Cognee turns documents into a knowledge graph an agent can search. The connector feeds it from DocuSign:

  • Each envelope becomes one document. A short header says who sent it, who signed as what and when, plus custom fields and what signers typed into form fields. The text of the signed PDF follows.
  • You choose which envelopes: by status, by folder, or by a start date.
  • Later runs only look at envelopes whose status changed.
  • Envelopes that are voided, deleted or purged are forgotten. 
source = docusign_source()  # DOCUSIGN_* from the environment
await cognee.remember(source, dataset_name="docusign",
                      write_disposition="merge")
Enter fullscreen mode Exit fullscreen mode

It signs in with a JWT grant (integration key, user id, RSA key) and only ever sends GET requests. The API host comes from DocuSign's userinfo call, not a hardcoded demo.docusign.net, because production accounts live on different hosts.

The catch with "deleted"

DocuSign does not really delete envelopes. And the obvious signal does not work.

I sync with from_date and from_to_status=changed, which lists envelopes whose status changed. But a completed envelope cannot be voided. Removing it means moving it to the Deleted folder, or purging it. Neither one changes the status date. A cursor never sees them.

So every run does a second check. Any envelope the connector ingested earlier but the listing did not return is looked up again by id, with its folders and purge state. If it is binned, purged, gone or out of scope, it becomes a delete. dlt removes the row and Cognee's orphan cleanup removes it from the graph and the vectors.

I added rules because a wrong delete is worse than a missed one. A failed page, a 401 or a long rate limit stops the run before anything is deleted or the cursor moves. An empty listing never wipes the dataset.

The real test

First I built a fake DocuSign that sits under a real HTTP session, so the JWT exchange, paging and retries all run. 59 tests pass, including one that runs the whole Cognee pipeline with a mocked LLM and shows that an unchanged re-sync makes no LLM call. A fake only proves what I thought to fake, though.

So I used a free developer account. I created five agreements through the API, signed them with a headless browser, and ran the example script with a real LLM.

The question "Who signed which agreements, and when?" came back with all five signers and the right times.

Then I moved the Deltacorp offer letter to the Deleted folder. Its status stayed completed, as I expected. The next sync reported forgotten: 1 and the dataset went from 5 documents to 4. Asking about the salary in that letter now returns "the context does not state it".

I purged another envelope. The status stayed completed there too. Only purgeState changed, to documents_queued. One sync later: forgotten: 1, 4 documents down to 3.

The real account did not find a bug this time. It confirmed four things I had only read in docs: the by-id lookup works without from_date, the form data has the shape I assumed, the folder types are sentitems and recyclebin, and neither a binned nor a purged envelope moves the status date.

What I did not test

  • Scanned PDFs. The demo stamps text onto every page, so my "scan" was never empty. That path has fake tests only.
  • A purge that removes the envelope from every listing.
  • Production. I ran the demo host only.
  • Distinct signer emails. All five signers used my own address.

One more limit. The text comes from the PDF's text layer. There is no OCR.

Try it

The example is packages/connector/docusign/examples/example.py. You need a free developer account, an integration key with an RSA key pair, and one consent click. The README has every step.

If you point it at real agreements, keep the API user's permissions small and the dataset separate. I'd like to hear what breaks.

Top comments (0)