A schemaless file intake never crashed. It silently dropped good files three times.

info: The final draft was polished with AI assistance. The facts and the wording were checked by the author.

TL;DR

• The problem: an intake that accepts files without a schema can drop a file from processing with no error and no warning, and nobody notices. My weekend prototype fell into the same kind of hole three times.
• The fix: put a layer that does zero interpretation between the intake and the interpretation step (an LLM or a human). The same input always produces the same record, so "what was picked up and what was dropped" becomes something you can check mechanically.
• Why it matters: you can start collecting data today without finishing a schema first. Swap the interpretation for an LLM later; the record layer keeps guaranteeing that nothing was silently skipped.

Why I chose a schemaless intake

My notes and work logs pile up across folders and chats, one per topic. To answer "where did I leave this?" I had to open the files and re-read them myself. Tracking state was entirely manual, and that was the starting point.

I had tried the schema-first approach before. I started from "the fields everything should share", but the topics were too different from each other, I never managed to freeze a schema that fit all of them, and the design stalled.

This time I did the opposite and postponed the schema. I prototyped, over a weekend, an intake where you drop files into a folder without declaring a type, and left the meaning-making to a downstream interpretation layer (an LLM or a person).

What I learned after building it is that "accepting anything" is not the real problem. The scary part is that a file you think was accepted can fall out of processing midway, and there is no path that detects it. When I actually ran it, I hit the same kind of hole three times.

This is still a hypothesis. It has never run on real data; I only ran it end to end on a mock dataset on my laptop. The reader I have in mind is someone building a data store that an AI will read in full, who is stuck at the schema-design step. That was me a few months ago.

Three layers: intake, record, interpretation

My first idea was simple: drop files in a folder and let the AI read them and judge the state. But in that design nobody can notice "the file the AI failed to read". The AI's interpretation wobbles slightly every run, and it does not raise an exception when it misses something.

So I inserted a layer between intake and interpretation that does no interpretation at all. It is close in spirit to a tiny data lake with schema-on-read: nothing is enforced at write time.

flowchart TD
    A["Intake<br/>drop files in a folder<br/>no schema"]
    B["Record layer<br/>which files exist, their dates, a content hash<br/>zero interpretation"]
    C["Interpretation layer<br/>summary, classification, next step<br/>LLM / human"]
    A --> B --> C

• Intake: drop a file in a folder. The folder name is the classification key, so there is no entity-resolution step to write.
• Record layer: mechanically records every file's path, date and content hash. No interpretation goes in, so the same input always yields the same record. Whether a file was picked up is something you check against this layer.
• Interpretation layer: summaries, classification and "what next" belong to the LLM or a human. But values a machine can settle, like "when is this file from", come from the record layer. Hand that to an LLM and its judgement of newer-versus-older drifts, and you get freshness hallucinations.

The skeleton of the record layer

The code is a simplified sketch of the pattern and will not run as-is.

import hashlib
from pathlib import Path
import frontmatter  # standing in for python-frontmatter

ROOT = Path("inbox")

def build_records(known_keys: set[str]) -> list[dict]:
    records = []
    for folder in sorted(ROOT.iterdir()):
        key = folder.name
        if key not in known_keys:
            print(f"[warn] unknown folder '{key}'. Add it to config or check the name.")
            continue
        for path in sorted(folder.rglob("*")):
            if not path.is_file():
                continue
            post = frontmatter.load(path)
            date = post.get("date")
            if date is None:
                print(f"[warn] {path} has no date. It will not count toward freshness.")
            records.append({
                "key": key,
                "path": str(path.relative_to(ROOT)),
                "date": str(date) if date else None,
                "digest": hashlib.sha256(path.read_bytes()).hexdigest()[:12],
            })
    return records

Hold the list of known keys up front and warn when a folder name is not in it. A file without a date is recorded rather than crashed on, but it gets a warning. These 2 checks were added after the holes described below. The first version warned about neither.

The sync step that copies files in compares content hashes when a file of the same name already exists at the destination, and skips the write if nothing changed. Running it twice in a row ends with zero diff the second time. Confirming idempotence by hash comparison instead of eyeballing logs was a small but real comfort.

Hole 1: an unknown folder name passed straight through

The first implementation treated any folder name as a classification key. I assumed that if the scan finished without an error, the files I intended to include had been picked up.

I created a folder with a typo in its name, put a file in it and ran the sync. It finished normally, with no error and no warning. The file was nowhere in the record. From the system's point of view, a file I had placed had never existed.

The cause was equating "finished without error" with "processed as intended". The scan never validated folder names, so an unknown name passed exactly like a known one. The check against a list of known keys was missing from the design itself.

Hole 2: a file without a date silently fell out of the freshness calculation

I had defined the record layer's job as "record that a file exists", so recording a dateless file as "no date" seemed sufficient. Whoever looked at it could decide. Being in the record was supposed to be proof that the file had been picked up.

I placed a file with no date and checked. As expected, it appeared in the record with no date. No crash, no skip. But it was excluded from the "when was this last touched" calculation, and nothing in the record told me that it was not being counted.

The cause was conflating "is in the record" with "is used by the downstream calculation". The record layer only proves existence; it says nothing about how downstream uses the value. Dateless files now get a "will not count toward freshness" warning, so the distinction is visible the moment you look at the record.

Hole 3: keyword-based classification reacted to the wording of my test data

My mock classifier was a plain keyword match. I wrote "dummy data for the PoC verification test" in a test file's note, and the string "PoC" made it classify the file as "prototype stage".

I changed the wording to "this is not a production log" and ran it again. This time "production" made it classify the file as "in production". I had changed the phrasing after the 1st failure and fell into the same structural hole a second time.

This was my own design rule, keep record and interpretation apart and never let keyword matching interpret, demonstrated twice against my own test data. Classification is interpretation and belongs to the LLM or a human, not to a keyword match. The mock was simply too crude.

What the real risk of a schemaless intake is

None of the three holes was the kind of bug where the system breaks and stops. They were the opposite: processing "completed" with no error and no warning, in a state that was probably wrong. The real risk of a schemaless intake is not "anything can get in". It is "you cannot tell from the output what was picked up and what was dropped". I only felt that while implementing it.

The interpretation-free record layer exists to make that distinction checkable by machine. Because it does not interpret, the same input gives the same result every time. So "were this folder's files picked up?" and "is this date being counted?" are questions you answer by looking at the record.

Three rules that keep the record layer honest

After plugging the holes I set three boundary rules so that the record layer's definition, zero interpretation and complete coverage, does not erode. Each was checked with a small experiment on my laptop.

The decision to stop lives outside the record layer. I wanted processing to halt once warnings passed a limit. But if record generation is cut short, the healthy files that would have been scanned afterwards never appear, and the record alone cannot tell you whether it reflects everything or was interrupted. Write the record to the end, mark it unhealthy, and let the downstream reader decide whether to stop.

Embed the health mark in data the downstream will always read. Put the unhealthy mark in a side file and any downstream that does not bother to read it behaves as if the mark did not exist: it runs to completion with zero warnings. Mix the mark into the record itself and an implementation that ignores it does not "get away with it"; it stops on a row of the wrong shape. Ignoring becomes visible.

Never write interpretation results back into the record. Do that and the record's hash changes on every run, so a hash match can no longer tell "the input changed" from "only the interpretation changed". Keep interpretation results in a separate file, keyed by the path, which is unique within the record. Keying by content hash goes wrong as soon as two folders hold files with identical content: their results get swapped.

Conclusion

The intake is a layer where you just drop files, with no schema. The record layer does not interpret; it writes every file's path, date and hash through to the end and carries its own health mark. The interpretation layer is where an LLM or a human decides, and its output goes to a separate file rather than back into the record. That is where the design settled. It has still never run on real data, which will surely be messier than my mock, and I expect there are holes I have not found yet.