I wanted eleven years of my own notes as plain Markdown files, so I could grep them. Two sources: a phone notes app going back to 2015, and Youdao Note, a Chinese note service with an official command-line client.
The plan was one script: walk the folder tree, read every note, write it to disk with a little YAML frontmatter. The first run finished without an error. It was also wrong in three different ways, and none of them raised anything.
1. A flaky error that looked like "no more pages"
The CLI lists a folder 50 entries at a time; you pass the last id as a cursor to get the next page.
The server also drops the socket fairly often. My first version treated a failed list call as "this folder doesn't paginate" and moved on with what it had. Every folder with more than 50 notes was silently cut at 50. The walk finished, the totals looked plausible, nothing complained.
The fix is boring: retry the transient failure, and only stop paginating when a page comes back short.
for attempt in range(RETRIES):
code, text = run(["list", "-f", folder_id, *cursor_args])
if code == 0:
break
time.sleep(1.5 * (attempt + 1))
# ...parse entries into `page`...
if len(page) < PAGE: # a short page is the only real "end"
break
The rule I wrote into the docstring: a transient error and a terminal condition must never share a code path.
2. Exit code 0, body = the error message
Reading a note is youdaonote read <id>. For some notes the server answers with {"text": "获取笔记内容失败", "isError": true} ("failed to get note content"). The CLI ignores isError, prints the text to stdout, and exits 0.
So the script wrote files whose entire body was an error string. By exit code, by file count, by "did it write something": all success.
The guard has to look at the content:
ERRORS = ("获取笔记内容失败", "SSE error", "socket connection was closed")
ok = (code == 0 and text.strip()
and not (len(text.strip()) < 400
and any(e in text for e in ERRORS)))
The length cap is there so a real note that happens to quote one of those strings isn't thrown away.
Same place, second trap: empty notes. For a note with no body, the CLI throws an SSE connection error instead of returning an empty string. I checked each of those through a second client; they really were empty. So "failed" had to be split into empty, unreadable, and actual failures. Otherwise --only-failed keeps retrying notes that will never have content.
3. The API was never going to work for most of them
With both guards in place, the numbers were finally honest, and bad:
| status | notes |
|---|---|
| readable through the API | 504 |
server returns isError
|
1,292 |
| empty | 78 |
| duplicates of the phone-app notes | 80 |
The 1,292 weren't random. They were the older notes in the service's legacy .note format; everything from before 2021 came back this way. Retrying doesn't help. The server's text extraction just doesn't handle them.
What worked was the unclever path: the app's own "export everything" button. It renders legacy notes to PDF, and the text inside those PDFs is intact. So the second importer reads the export folder instead of the API:
def extract(path):
name = path.name.lower()
if name.endswith(".pdf"):
import pymupdf
with pymupdf.open(path) as doc:
return "\n".join(pg.get_text() for pg in doc).strip(), "pdf"
if name.endswith(".mindmap"):
return mindmap_to_outline(path.read_text("utf-8")), "mindmap"
return path.read_text("utf-8").strip(), path.suffix.lstrip(".")
Mind maps are JSON node trees; mindmap_to_outline walks from the root and emits an indented bullet list, which greps fine.
Where it can match on (folder, title), the importer reuses the note id from the API pass, so both passes line up; otherwise it derives a stable id from a hash of the path.
Final count from the export: 1,853 notes with a body. 1,652 from PDF, 103 mind maps, 94 Markdown. Plus 234 from the phone app: 2,087 notes, about 3.5 million characters, all greppable.
One more: trust the disk, not the manifest
The exporter runs 12 workers and takes a while, so it has to resume. The manifest of finished notes is only written at the end of a run. Hit Ctrl-C and the manifest says nothing is done, while the disk holds hundreds of files. So the resume check asks the disk:
def already(item):
if done.get(item["id"], {}).get("status") == "ok":
return True
p = note_path(item)
return p.exists() and p.stat().st_size > 0
What I'd do first next time
- Pick one note you know is broken and one you know is empty. Look at exactly what the tool returns for each before writing the loop.
- Count by content, not by exit code or file count.
- Check the official export before the API. It was the less clever path and the only complete one.
Once everything was on disk, the payoff was immediate. The first thing I counted was punctuation: about half of my short notes don't contain a single period. That part is a different story, written up here: https://medium.com/@cxtao203/i-exported-eleven-years-of-my-own-notes-and-found-the-same-sentence-every-time-64b04dbdfc60
Top comments (1)
number 2 is the one that gets everyone. exit 0 with the error text as the body means every check downstream says success, file count included. the length cap so a real note that quotes the error string doesn't get thrown out is a nice touch.