I built a schemaless intake for my CRM. It silently dropped files three times.

I dropped a file into my own intake folder, ran the sync, and watched it finish clean. No error. No warning. The file was never there again — not in the index, not in any downstream output, not anywhere the system could tell me about.

The risk of a freeform intake was never "anything can get in." It's that nothing tells you what got dropped.

TL;DR: A schema-less intake that lets you dump any file into a folder doesn't fail loudly when a file falls out of the pipeline — it just finishes normally without it. I hit that exact hole three times while prototyping a personal CRM over a weekend. The fix wasn't more validation at the intake. It was a deterministic index layer, sitting between intake and interpretation, that records what it saw and nothing else — so "did this file get picked up" becomes a question you can check mechanically instead of one you have to trust.

Quick answer: why does a freeform data intake lose files silently, and how do you catch it?

• A freeform intake (drop any file, decide the schema later) doesn't reject bad input — it just never validates the input at all. An unrecognized directory name, a missing metadata field, or a naive downstream reader can all cause a file to disappear from processing with zero errors and zero warnings.
• I hit three separate versions of this while building a personal case-tracking intake: an unknown client directory that passed through silently, a missing date field that silently fell out of a downstream calculation, and a keyword-matching classifier that misfired on my own test data — twice.
• The fix is a deterministic index layer between the freeform intake and any interpretation step (LLM or human): it records path, date, and a content hash for every file it sees, does zero interpretation, and therefore produces the identical output for the identical input every time. "What got picked up and what didn't" becomes something you diff, not something you assume.
• Four follow-up experiments (Plan A vs. Plan B for where to stop a bad sync, replacing keyword matching with an LLM, catching downstream code that ignores a health flag, and caching interpretation results) show where that determinism can quietly break — and how to keep it from breaking.
• This is a hypothesis-stage design. It has been run end-to-end against a local mock repository, not against production case data.

I want to be upfront about scale before anything else: this is a weekend prototype for my own case-tracking workflow, not a production data platform. But the failure mode underneath it — a permissive intake that can drop input without telling you — shows up at any scale where "let the AI read whatever's in the folder" sounds like a reasonable starting design. I built one of those. It looked like it worked. It quietly wasn't.

Why I built a schemaless intake in the first place

My case notes, meeting minutes, and Slack threads for each client pile up scattered across folders and chats. Answering "what phase is this client in right now" meant opening files and re-reading them by hand, every time. That manual lookup was the whole problem I was trying to solve.

I'd tried the opposite approach before: design the schema first. I started from "what fields does every client record need," and ran straight into the fact that clients are different enough that no single schema covered all of them. I stalled out at the design stage and never shipped anything.

This time I deferred the schema decision instead of trying to resolve it up front. I prototyped a freeform intake over a weekend — drop case notes into a folder with no fixed shape, and let a later interpretation layer decide what they mean. The reference point was Palantir's ontology model — specifically just the idea of separating meaningful data units (Object), the connections between them (Link), and the operations that change state (Action). I only borrowed that three-way split; nothing else about the ontology system.

What I learned building it: accepting "anything" without a schema wasn't the dangerous part. The dangerous part was that a file I thought I'd successfully dropped into the intake could fall out of processing somewhere downstream, with no path back to noticing it had happened. I hit that same shape of hole three separate times once I started actually running the thing.

To be clear about where this stands: this is still a hypothesis-stage design with no production case data run through it. It's been exercised end-to-end against a local mock case repository — not more than that. If you're the kind of person stuck at the schema-design step of a system meant to feed an LLM everything, that's who this is written for. It's who I was a few months ago.

The design: a deterministic index layer that never interprets anything

My first instinct was something close to "drop files in a per-client folder, let an AI read the contents and figure out the client's state." That design has one property that kills it: nobody notices the file the AI failed to read. LLM interpretation drifts run to run, misses things, and doesn't throw an exception when it misses.

So I put a layer between intake and interpretation that does zero interpretation of its own. It's close in spirit to a small data lake with schema-on-read — nothing is validated or typed at write time. The system splits into three layers:

flowchart TD
    A["Intake<br/>data/intake/&lt;client_id&gt;/<br/>drop files, no schema enforced"]
    B["Deterministic index layer<br/>records path, date, source_id, health<br/>zero interpretation"]
    C["Interpretation layer (LLM/human)<br/>client phase, issues, next action<br/>judgment happens here"]
    D["last_contact_at<br/>computed mechanically from frontmatter dates"]
    E["Interpretation cache<br/>separate file, keyed by path<br/>never written back to the index"]

    A --> B
    B --> C
    B --> D
    D --> C
    C --> E

• Intake: drop files under data/intake/<client_id>/. The directory name is the client ID, so there's no separate entity-resolution step to write.
• Deterministic index layer: records every file's path, date, and source_id (a content hash) to JSONL. No interpretation happens here, so the identical input always produces the identical index. Whether a file was picked up or not is something you can verify by reading this layer's own record.
• Interpretation layer: an LLM or a human decides the client's phase, open issues, and next action. The one exception is "last contact date" — that's computed mechanically from the index layer's frontmatter dates, not left to the LLM. Handing date freshness to an LLM invites hallucinated recency judgments, so that stays on the deterministic side.

How the deterministic index layer is actually built (path, date, source_id)

The code below is a generalized, simplified version of the real implementation — it won't run as-is. What I want to convey is the pattern, not a drop-in library.

import hashlib
from pathlib import Path
import frontmatter  # standing in for python-frontmatter

INTAKE_ROOT = Path("data/intake")

def build_index(known_client_ids: set[str]) -> list[dict]:
    entries = []
    for client_dir in sorted(INTAKE_ROOT.iterdir()):
        client_id = client_dir.name
        if client_id not in known_client_ids:
            print(f"[warn] unknown client_id '{client_id}'. "
                  f"Add it to config or check the directory name.")
            continue

        for path in sorted(client_dir.rglob("*")):
            if not path.is_file():
                continue
            post = frontmatter.load(path)
            date = post.get("date")
            if date is None:
                print(f"[warn] {path} has no date key. "
                      f"It will not be reflected in last_contact_at.")

            entries.append({
                "client_id": client_id,
                "path": str(path.relative_to(INTAKE_ROOT)),
                "date": str(date) if date else None,
                "source_id": hashlib.sha256(path.read_bytes()).hexdigest()[:12],
            })
    return entries

known_client_ids is held up front, and any directory name not in it triggers a warning. A file with no date key is not crashed or skipped — it goes into the index as date: None, but with a warning attached. Neither of those two warnings existed in the first version. I added both after stepping in the holes below.

Local end-to-end verification (idempotency checked with sha256, not eyeballing)

Before touching anything close to real case data, I built a mock case repository locally and ran the whole thing through:

1. Repository scan → copy files into intake (a sync script)
2. Index generation (the code above)
3. Reading the index down to "today's tasks"

The sync script compares content by sha256 whenever a same-named file already exists at the destination, and skips the write if there's no difference. Run the same script twice in a row and the second run finishes with zero diffs. Confirming idempotency by hash comparison instead of scanning a log by eye was a small thing, but it was reassuring.

def sync_file(src: Path, dst: Path) -> bool:
    """Skip the write if the destination already has identical content."""
    if dst.exists() and (
        hashlib.sha256(dst.read_bytes()).digest()
        == hashlib.sha256(src.read_bytes()).digest()
    ):
        return False  # unchanged
    dst.write_bytes(src.read_bytes())
    return True

I only decide whether to run this against real case data after this whole rehearsal passes.

Hole 1: an unknown client_id directory passed through with zero errors

The first version treated client_dir.name as the client ID unconditionally. My assumption was: as long as the scan finishes without an error, the intended client's files must have been picked up correctly.

I typo'd a client ID, made a directory with that wrong name, dropped a file in it, and ran the sync. It finished cleanly — no error, no warning. Opening the index file, that file wasn't in it. As far as the system was concerned, a file I'd physically placed had never existed.

I'd been treating "finished without an error" and "was processed as intended" as the same statement. They weren't. The scan never validated the directory name at all — an unknown name passed through exactly like a known client_id would. The missing piece was a check against the known set of valid client IDs, and that check simply wasn't in the original design. I added the known_client_ids check and the warning to build_index in response: something like unknown client_id 'mystery-client'. Add it to config or check the directory name.

Hole 2: files with no date key silently never reached the freshness calculation

I'd defined the index layer's job as "record that a file exists." A file missing its date key, recorded as date: null, felt sufficient — whoever read the index could make the call themselves. Being in the index at all was supposed to be proof the file had been "properly picked up."

I dropped a file with no date key in its frontmatter and checked. As expected, it landed in the index as date: null — no crash, no skip. But it was excluded from the last-contact-date (last_contact_at) calculation, and looking at the index gave no way to notice that exclusion had happened.

I'd been conflating "is in the index" with "is used by the interpretation layer's calculations." They aren't the same state. The index is a record of existence; it makes no promise about how downstream calculations will use that value. I hadn't been thinking of those as two separate things, and that's exactly what I'd missed. I added a warning for date-less files — "will not be reflected in the last_contact_at calculation" — so the distinction is visible the moment you look at the index.

Hole 3: a keyword-matching phase classifier misfired on my own test data — twice

I'd built a mock extractor for client phase using plain keyword matching. I wrote a test-data note that said "this is dummy data for a PoC verification test," and the string "PoC" tripped the classifier into misjudging phase=prototype/demo.

I changed the wording to "not a production log" and re-tested — this time the string "production" tripped it into misjudging phase=full production. The second attempt was supposed to be a lesson learned from the first, and I fell into the identical shape of hole again.

This is the design principle I'd already committed to — never let keyword matching do interpretation, keep index and interpretation separate — proving itself twice against my own test data. Phase classification is exactly the kind of judgment that belongs to an LLM or a human, not string matching; the mock extractor was underbuilt on that front.

What a freeform intake actually risks (the working hypothesis)

None of the three holes were the kind of bug that crashes and stops the system. They were the opposite: processing completed with no error and no warning, in a state that wasn't actually correct. Building this made the real risk concrete for me — a freeform intake's danger isn't "anything can get in." It's that the processing result gives you no way to tell what got picked up and what got dropped.

The deterministic index layer exists to make that indistinguishability mechanically checkable. Because it does no interpretation, identical input always produces an identical result. That means "were this client's files picked up" and "is this date reflected in the freshness calculation" are both questions you can answer by reading the index file.

That said, everything above is a mock-repository trial. Real case data will be messier in format than anything I've tested, and I expect there are holes in here I haven't found yet.

Experiment: where should a warning threshold stop the sync?

An earlier draft of this design left an open concern: logging warnings from the index layer doesn't change the fact that you still have to go look at the log. Here I actually ran two candidate designs for what happens once the warning count crosses a threshold, and compared them directly. Plan A aborts the scan mid-way and exits with exit(1) the moment the threshold is crossed. Plan B lets index generation run to completion, sets an unhealthy flag, and leaves it to the downstream sync step to check that flag and refuse the handoff.

Prediction: Plan A, which stops processing the instant a warning is detected, should be the safer of the two.

Result: with the threshold set at 2, a third warning (an unknown client_id) triggered mid-scan. Plan A aborted right there. Two normal, zero-warning files belonging to a different client — files that would have been scanned later — never appeared in the index at all. Opening the index file on its own gives you no way to tell whether it reflects every file or stopped partway through. Plan B kept scanning to the end with the same three warnings, wrote all six entries to the index, and recorded unhealthy: true in a meta.json sidecar. The downstream sync refused the handoff on seeing that flag — but the index itself still held a complete record of every file.

$ wc -l index.jsonl
$ cat meta.json

6 index.jsonl
{"warning_count": 3, "unhealthy": true}

Analysis: stopping the "process" of index generation and keeping the index's "record" complete turned out to be incompatible goals. Once the index layer is defined as zero-interpretation-plus-complete-record, Plan A — which injects a judgment call mid-generation and cuts it short — contradicts that definition. A warning-threshold "stop" decision belongs at the boundary between the index and downstream processing, not inside the index layer itself.

def build_index(intake_root, known_client_ids, max_warnings):
    entries, warnings = [], []
    for client_dir in sorted(intake_root.iterdir()):
        ...  # scan logic runs to completion, never aborted mid-way
    return entries, {"warning_count": len(warnings), "unhealthy": len(warnings) > max_warnings}

def sync(index_path, meta):
    if meta["unhealthy"]:
        raise RuntimeError("index unhealthy: refuse to sync")  # the stop decision lives at the boundary

Experiment: does swapping keyword matching for an LLM break the index's determinism?

The other open concern: the moment keyword-based phase detection gets replaced with an actual LLM call, does the whole index lose its claim to reproducibility?

Prediction: index generation and interpretation (phase detection) are separate functions, but since both get called inside the same pipeline, I expected the non-determinism on the interpretation side to bleed into the index side too.

Result: running index generation twice, as an independent process, against identical input, and comparing the sha256 of the two index.jsonl outputs — they matched exactly. Then I swapped in a deliberately non-deterministic mock that reads the index and returns a phase (standing in for an LLM call, using an unseeded random number to change its answer every run) and ran that twice: all three subjects' phase results changed between runs. The index didn't change. The interpretation did.

Analysis: the index records the LLM's input (file contents) as-is; the interpretation layer reads that record and never writes back into the index. Data only flows from index toward interpretation, never the other way, so no matter how wide the interpretation side's randomness swings, there's no path for it to reach the index. My concern — "putting an LLM in the loop breaks reproducibility" — was correct, but what breaks is the reproducibility of the interpretation, not of the index. Because the index completely records what was handed to the LLM, even if the interpretation's result changes on every run, which input produced which judgment can still be verified after the fact by reading the index.

def build_index(intake_root) -> list[dict]:
    ...  # no randomness, no LLM call. records the read content as-is
    return [{"client_id": cid, "path": p, "content": post.content, ...}]

def interpret(index_entries: list[dict]) -> list[dict]:
    # stands in for the LLM call. never writes back into the index
    return [{"path": e["path"], "phase": call_llm_phase(e["content"])} for e in index_entries]

Experiment: how do you catch downstream code that ignores the unhealthy flag?

Plan B, from two sections up, assumed downstream processing would always check the unhealthy flag. I tested what happens when an implementation slips through that doesn't — and whether that gap itself is detectable.

Prediction: even with the current design of recording unhealthy in a sidecar meta.json, an implementation that doesn't reference it should still get caught somehow — at minimum, the index file's shape not matching expectations should throw an error somewhere.

Result: I wrote a naive downstream reader that never references meta.json at all, just parses each line of index.jsonl as an entry directly, and ran it. It completed normally with zero warnings and printed out the healthy-looking last-contact date as if nothing were wrong. unhealthy: true never surfaced anywhere in this downstream code's output. Then I added a single sentinel record — {"_sentinel": true, "_health": "unhealthy", ...} — as the first line of the index JSONL and fed the same naive reader the same file. It crashed with a KeyError on entry["client_id"].

Analysis: a sidecar meta.json only does its job for an implementation that intentionally chooses to reference it. For an implementation that doesn't go looking, the record living there might as well not exist. A sentinel record instead mixes health information into the index itself — the one file downstream code is guaranteed to read. An implementation that ignores the flag doesn't get to "successfully ignore" it anymore; it gets stopped by a record shape it wasn't expecting, and that failure to ignore becomes visible on its own. Carrying health status as part of the index's own data, instead of a sidecar, doesn't depend on how carefully downstream code was written.

# A naive downstream reader unconditionally reads entry["client_id"].
# A meta.json sidecar completes without noticing.
# Feeding it an index with a sentinel record
# ({"_sentinel": True, "_health": "unhealthy"}) as the first line
# raises a KeyError on that line and crashes instead.
for line in index_path.read_text().splitlines():
    client_id = json.loads(line)["client_id"]

Experiment: does caching interpretation results back into the index break determinism?

One more thing I wanted to confirm: if phase-detection results get cached by writing them back into the index, does the mechanism itself — "an sha256 match on the index proves the input is identical" — stop working?

Prediction: index generation and interpretation are separate functions, but writing interpretation results back into each index entry and keeping everything in one file should make the latest interpretation visible just by looking at the index, and should be more convenient to work with.

Result: with an implementation that adds a phase field to each index entry and overwrites the file, running twice against identical input produced two completely different sha256 values for index.jsonl. Diffing the two showed content unchanged — the phase value written back had simply changed between runs. I then separated phase into its own file (phase_cache.jsonl), with the index never written to at all, and ran it twice: index.jsonl's sha256 matched exactly both times, while phase_cache.jsonl's sha256 differed between runs.

Analysis: writing interpretation results back into the index makes it impossible to tell, from a sha256 match or mismatch alone, whether the index's own contents (path, date, source_id) changed, or whether only the interpretation changed while the input stayed the same. Once the index is defined as zero-interpretation-plus-deterministic, importing interpretation's non-determinism breaks that definition the moment it happens. Separating the cache into a file independent of the index keeps the index's sha256 usable as a signal of input identity.

There was a second hole hiding in this one, though: keying the cache on source_id alone. Two files with byte-identical content (a reused template, for instance) placed under different client directories produce the same source_id by construction. Look that up in the cache and one client's interpretation result can silently overwrite the other's. Keying on path instead — which is always unique within the index — avoids that collision entirely.

# 4a: written back into the index (index.jsonl's sha256 changes every run)
entry["phase"] = interpret(entry["content"])  # overwrites index.jsonl itself
# 4b: kept in a separate file, keyed on path (always unique within the index)
cache[entry["path"]] = interpret(entry["content"])  # index.jsonl is never touched

Conclusion: a three-layer design that makes freeform intake verifiable

Across the three holes and the four experiments above, the design settled into this shape. Intake (data/intake/<client_id>/) is a schema-free layer — drop a file and it's tied to a client ID by directory name alone. The index does zero interpretation and completely records every file's path, date, and source_id; health status (unhealthy) lives inside the index's own data, as a sentinel record, not in a sidecar. Interpretation is where an LLM or a human judges client phase and next actions, and that result never gets written back into the index — it goes into a separate cache file keyed on path, which is always unique within the index.

These three layers, plus two boundary rules (health status lives inside the index's data; interpretation results never write back into the index), are where this design landed. It has not yet been run against real case data. Operational monitoring and handling the format variance real case data will inevitably have are both out of scope for this post — but the chain of verification that started from "the hole in a schema-free intake" feels, to me, like it's made one full pass through the question of what belongs at which point across intake, index, and interpretation.

Copy-paste checklist: before you build a freeform intake

# does anything in your intake pipeline finish "success" without validating
# the identifier it just consumed?
grep -rn "iterdir\|listdir\|glob(" your_intake_script.py

# does your index (or equivalent record) get written to by more than one layer?
# if interpretation results and raw intake facts share a file, you've lost the
# ability to tell "input changed" from "interpretation changed" by hashing it
grep -rn "\.jsonl\"\|\.json\"" your_index_script.py

For each hit, ask one question: if this component silently drops or silently reinterprets something, is there any file you could open afterward that would tell you? If the answer is no, you have the same hole this post describes.

If you're tempted to skip the index layer because "the LLM can just read everything directly" — that was my first design too, and it's the design that hides exactly this failure. A misread file produces no exception. Nothing tells you it happened, ever, unless something downstream of the LLM keeps its own zero-interpretation record of what went in.

FAQ

Why not just let the LLM (or a human) read every file directly, and skip the index layer?

That was my first design, and it's the one that hides this exact failure. A file the LLM fails to read produces no exception — its interpretation just quietly doesn't happen. Without a layer that records "this file existed and was seen" independently of interpretation, there's no way to notice a miss after the fact.

Doesn't validating known_client_ids defeat the point of calling this a freeform intake?

No — "freeform" here only applies to file content and schema, not to the directory namespace. The intake still needs a closed, known set of valid client IDs, because without one, a typo in a directory name is silently treated as "there is no such client" rather than as an error. That's Hole 1 in this post.

If the index never interprets anything, how does it produce last_contact_at?

Mechanically, from the date field in each file's frontmatter — not from an LLM judgment call. Handing "is this recent" to an LLM risks an inconsistent, hallucinated read on freshness across runs, so that one calculation stays on the deterministic side of the boundary, even though everything else about client state is interpretation.

Is any of this running against real client data yet?

No. Everything described here — the three holes and the four experiments — was run against a local mock case repository, end-to-end, not production data. Real case files will vary in format more than anything I've tested so far, and I expect to find holes this design doesn't yet handle.

How I verified this

Every claim above came from actually running the code, not from reasoning about what it should do. The three holes were reproduced by hand: creating a mistyped client_id directory, dropping a file with no date field, and feeding the mock phase classifier test data containing "PoC" and then "production" as literal strings. The four follow-up experiments were run as described — sha256 comparisons on index.jsonl across repeated runs, a deliberately non-deterministic mock standing in for an LLM call, a naive downstream reader tested both with and without a sentinel record, and two cache designs (written-back vs. separate file) diffed by hash. All of it ran against a local mock case repository built for this purpose; none of it has touched real case data. Client names, directory structures, and specific case content are my own test fixtures, not real client information.

References

• Ontology – Core concepts | Palantir — source for the Object / Link / Action split used to separate meaning from process in this design

---

Sho Naka (nomurasan). Originally written in Japanese; the English here is an AI-assisted re-adaptation of my own text, not a literal translation. The design, the code, and the experiment results are mine.