What a 14-day Freshdesk trial taught me about building a cognee connector

During Mergetober I picked up topoteretes/cognee#4756: build a Freshdesk connector for cognee, the open-source AI memory engine. The connector pulls support tickets, their replies and knowledge-base articles into cognee so you can ask questions like "What problems are customers reporting?" and get answers from a knowledge graph.

The issue had a clear checklist:

• Connect with an API key and choose what to ingest
• Ingest tickets, conversations and solution articles
• Incremental sync with updated_since: only fetch what changed
• Forget on delete: if it's deleted in Freshdesk, it's gone from the graph on the next sync
• Tests covering the ingest path and the incremental cursor
• "Rate limits are per-plan and low on entry tiers. Back off politely."

I could have written the connector straight from the API docs. Instead I opened a 14-day trial, made test data by hand in the UI, and checked the API after every click. Almost every design decision in the final PR came from something the docs don't say.

Here's what I found.

---

Step 0: "Just use dlt's verified source"

Two other people in the issue thread suggested reusing dlt's verified Freshdesk source. So before building anything I read its code, not just its README:

• Endpoints: agents, companies, contacts, groups, roles, tickets. No conversations, no solution articles, no delete handling. That's three of the five requirements missing.
• The incremental cursor starts at initial_value="2022-01-01T00:00:00Z", so tickets last updated before 2022 are never loaded.
• It stops at Freshdesk's 300-page cap (its own comment says so) with no way to walk past it.
• Rate limiting is a while True retry on 429 only. No pacing, and requests.get has no timeout.
• It isn't a pip package. It's a template you copy in with dlt init.

So I borrowed the idea (merge on id with an updated_at cursor) and not the code.

Lesson: when someone says "just reuse X", spend 20 minutes reading X's source. The README won't list what's missing.

---

Setting up the trial

Freshdesk's signup is now called "Freshdesk Omni". The trial matters: the Free plan has no API access.

Two small things that cost me time:

• Your helpdesk domain is in the address bar (<domain>.freshdesk.com/a/...). If you see *.myfreshworks.com, that's the Freshworks org login, not your helpdesk.
• The API key is under profile picture → Profile settings → View API key. New keys look like fwapi_... and still work with Basic auth as key:X.

My first read-only call returned something important in the headers:

X-Ratelimit-Total: 50
X-Ratelimit-Remaining: 49
X-Ratelimit-Used-CurrentRequest: 1

50 requests per minute on a trial. If you fetch conversations with one request per ticket, you burn through that fast. So pacing has to be part of the design from the start.

---

Finding 1: A reply moves the ticket's updated_at (good news)

I made three test tickets (FDTEST-1/2/3) against a fake example.com contact so Freshdesk's auto-emails go nowhere.

Then I snapshotted updated_at, added a public reply, and checked again. Then added a private note, and checked again.

Action	Ticket updated_at
Before	20:26:05
Public reply	20:28:57 (reply created at 20:28:54, ticket bumped ~3 s later)
Private note	20:29:50

Both bump the ticket. That means conversations don't need their own cursor: if a ticket shows up in updated_since, refetch its conversations. No scanning every ticket every run, which matters a lot at 50 requests a minute.

Private notes come back with private: true. They're internal text ("legacy SSO tenant, check auth logs"), so the connector leaves them out unless you set include_private_notes=True. Internal notes shouldn't end up in a searchable graph by accident.

---

Finding 2: updated_since is inclusive

Querying with since=20:27:43Z returned a ticket whose updated_at was exactly 20:27:43Z. So the ticket sitting on the cursor comes back every single run.

Merging on id makes that harmless, but it still means wasted work and wrong stats. The connector remembers which ids sat on the cursor and skips them next time unless they changed again:

if ticket_id in on_cursor and ticket["updated_at"] == since:
    stats["skipped"] += 1  # already synced on the cursor boundary last run
    continue

---

Finding 3: Deleted tickets just disappear (the big one)

This is where following the docs would have produced a broken connector.

I deleted a ticket, restored it, and marked another as spam:

Action	updated_at	Normal list with updated_since	Special filter	GET /tickets/{id}
Delete	moves	gone, even though it changed after the cursor	filter=deleted shows it	200, deleted: true
Restore	moves again	back	gone from filter=deleted	200
Mark as spam	moves	gone	filter=spam shows it	200, spam: true

If you only follow updated_since, you never see a delete. The ticket just silently drops out of the list. Your graph keeps it forever.

The fix: filter=deleted and filter=spam both combine with updated_since. So every sync does three listings:

listed = _list_tickets(client, since)          # 1. what changed
gone: set[int] = set()
if not full:
    for name in ("deleted", "spam"):           # 2 + 3. what was deleted or spammed
        items, truncated = client.list_all(
            "/tickets", {"filter": name, "updated_since": since}
        )
        gone |= {t["id"] for t in items}

The deleted and spam results become _deleted tombstones, and cognee forgets them. Restore needs no special code: the ticket shows up in the normal listing again and the merge brings it back.

The one case it can't see: "Delete forever"

I also tested permanent deletion from Trash. After "Delete forever", the ticket is gone from filter=deleted too, and GET returns 404.

So a ticket that is trashed and purged between two syncs leaves no trace for an incremental run. A full listing (first run, or after changing a rendering option) does catch it. Rather than pretend otherwise, it's written in the README's limitations.

---

Finding 4: Articles in subfolders are invisible

Knowledge-base articles live in a tree: category → folder → subfolder → article. I built exactly that, with the only published article inside a subfolder.

GET /solutions/categories/{id}/folders returns top-level folders only. Subfolders are only reachable through /solutions/folders/{id}/subfolders. Skip that call and in my test the connector would have ingested zero published articles.

So the walk recurses, with a visited set and a depth cap:

def _walk_folder(client, folder: dict, seen: set[int], depth: int) -> Iterator[dict]:
    if folder["id"] in seen:
        return
    if depth > MAX_FOLDER_DEPTH:
        # Skipping would forget every article below, so stop instead.
        raise RuntimeError(f"Freshdesk folders nest deeper than {MAX_FOLDER_DEPTH} levels.")
    seen.add(folder["id"])
    yield from _list_or_empty(client, f"/solutions/folders/{folder['id']}/articles")
    # The category listing only returns top-level folders; articles in
    # subfolders are reachable only through this call.
    for sub in _list_or_empty(client, f"/solutions/folders/{folder['id']}/subfolders"):
        yield from _walk_folder(client, sub, seen, depth + 1)

Note the raise instead of return on the depth cap. If the connector silently stopped walking, the next sync would think every article below that level was deleted and forget them all. Errors should never delete.

Articles also have no updated_since and no deleted filter. A deleted article simply returns 404. So articles are walked in full every run and compared with the ids from the last run: missing ids become tombstones.

---

Finding 5: A years-old forum bug is still there

I found an old developer-forum thread claiming that a published article with unsaved-but-not-published edits returns the draft text from the API. I tested it:

1. Published an article with the text "VERSION ONE".
2. Edited it to "VERSION TWO UNPUBLISHED EDIT" and only saved (didn't publish).

The API (both the list and the single GET) returned status: 2 (published) with the VERSION TWO text. Meanwhile the public customer portal still showed:

Still there. And no field in the response flags pending edits. I checked every key. Worse, the editor auto-saves while you type, so an agent halfway through an edit already changes what the API returns. And updated_at moves on both save and publish, so you can't tell "draft saved" from "went live".

I didn't fake a fix (scraping the portal HTML is fragile and fails on private portals). The README says plainly that the API returns the latest saved text, which may include unpublished edits.

---

Finding 6: A rate-limit header that breaks int()

Early in the session X-Ratelimit-Remaining came back as "48". A few minutes later it came back as "38.0".

>>> int("38.0")
ValueError: invalid literal for int() with base 10: '38.0'

A connector that paces itself with int(headers["X-Ratelimit-Remaining"]) would crash partway through a sync. The fix is one line, plus a test so nobody "cleans it up" later:

def _header_int(headers, name: str) -> int | None:
    # Freshdesk sends rate-limit counters both as "48" and as "38.0".
    try:
        return int(float(headers.get(name)))
    except (TypeError, ValueError):
        return None

On top of that: 429 waits for Retry-After, 5xx backs off, and every request has a timeout.

---

The bug only a live run could catch

With the design settled, I wrote the connector following the layout of cognee's Google Calendar connector (merge + _deleted tombstones, errors never delete). 36 sync tests passed on the first try.

That made me suspicious. So I broke the code on purpose 12 ways (removed the deleted/spam passes, removed the subfolder walk, used int("38.0"), dropped the boundary skip, used the 30-day default list, leaked private notes, ignored Retry-After, removed timeouts...) and checked that a test failed for each one. 12 out of 12 caught.

Then I ran the real connector twice against the trial. The rows were right. But the saved cursor was the spam ticket's timestamp.

That's a race:

1. The normal listing runs.
2. An agent edits ticket A (time T2).
3. Ticket B gets deleted (time T3).
4. The deleted pass sees B, and the cursor jumps to T3.
5. The next run asks for updated_since=T3, and A's edit is skipped forever.

The fix: only the normal listing moves the cursor.

# Only the normal listing moves the cursor.  A ticket deleted after that
# listing can carry a later updated_at than a ticket edited in between,
# and jumping to it would skip the edit forever.
cursor = max((t["updated_at"] for t in listed), default=since)

I added test_a_delete_during_the_run_does_not_move_the_cursor_past_an_edit, put the old code back, and confirmed the test failed with "the edit must not be skipped".

Lesson: my fake API passed every test because I wrote both the fake and the code with the same wrong assumption. Real data broke the assumption.

One design choice helped a lot here: the test fake replaces only the HTTP session. The real client (link-header paging, retries, Retry-After, the "38.0" header, the 300-page restart) runs in every test.

---

Running it end to end (and the Windows detours)

The example runs the full pipeline: sync from Freshdesk → cognee builds a graph with a local LLM (Ollama, llama3.1:8b + nomic-embed-text) → ask a question. It took five runs to get a clean one, and none of the failures were the connector's fault:

1. OllamaException - 404 page not found. cognee 1.6.3 appends /api/... to LLM_ENDPOINT, and my .env had http://localhost:11434/v1 (correct for older cognee versions). So requests went to /v1/api/generate. The fix was to drop the /v1.
2. The fix didn't apply. Setting LLM_ENDPOINT in the shell does nothing, because cognee loads .env over the environment. I used a separate .env for the run instead.
3. LanceDB: "failed to persist". The file path was 279 characters. By default cognee keeps its databases inside the venv's site-packages, and Windows' MAX_PATH is 260. Pointing SYSTEM_ROOT_DIRECTORY and DATA_ROOT_DIRECTORY to a short path fixed it. (Windows contributors: set these first.)
4. The run worked, but the log had no results. Python block-buffers stdout when it's redirected to a file, so the final prints died with the process. Use python -u.
5. Clean run. exit=0.

The second run, with nothing changed in Freshdesk, is my favorite screenshot:

tickets: 0, skipped: 2. Nothing was reprocessed. The two skipped items were the draft article and the ticket sitting exactly on the cursor, which is the inclusive-boundary rule showing up on real data. And the recall answer lists the real test tickets (login failure, a double-charge refund) while the spam ticket's "cheap watches" text is nowhere in it.

I also got something wrong here: I first said skipped: 1 in an earlier run was the spam ticket. Reading the code showed it only counts drafts and boundary tickets. Check the code before explaining a number.

---

Final checks

• 57 tests passing. All 43 warnings come from installed libraries, 0 from the connector.
• ruff check and ruff format --check clean, the same commands CI runs.
• A forget test runs the real cognee pipeline with a mocked LLM and checks that a deleted ticket, a spam ticket and a deleted article leave the graph while an untouched ticket stays.

Before opening the PR I re-read my own plan comment on the issue and found a mismatch: I'd promised "no requester email", but the code wrote Requester: Name <email> into every ticket document. I fixed it (now the name only, and a name containing @ is dropped), with a test for 3 cases. Maintainers assign issues based on your plan, so they may check the PR against it.

The PR is up: topoteretes/cognee-community#417.

Using it looks like this:

await cognee.remember(
    freshdesk_source(),  # reads FRESHDESK_DOMAIN + FRESHDESK_API_KEY
    dataset_name="support",
    primary_key="id",
    write_disposition="merge",
    max_rows_per_table=0,
)

---

What I'd tell anyone writing an API connector

1. Test on a live account before you design. My final design came out of six findings, and the docs mentioned none of them.
2. Test every way data disappears. Delete, restore, spam, purge, unpublish. "Follow updated_since" sounds complete until a deleted ticket silently drops out of the list.
3. Check whether your cursor is inclusive. Then write a test for the boundary.
4. Parse headers defensively. "38.0" is not an int.
5. Errors should never delete. If you're unsure whether something is gone, stop. Don't forget it.
6. Break your code on purpose. If your tests pass on the first try, check that they can fail.
7. Fakes share your assumptions. Run against the real thing at least once.

Thanks to the cognee maintainers for running Mergetober. If you're building connectors for cognee too, I hope this saves you a trial account or two.