DEV Community: JEONSEWON

I open-sourced the product that IS the business. Here's the cold logic.

JEONSEWON — Sun, 05 Jul 2026 09:05:41 +0000

My repo is public under MIT. The detector, the frozen parameters, the eval harness, even the log of where it fails (the E3 limitation) — all of it. For a solo founder whose product is that code, people reasonably ask if I've just given away the company. I don't think I have, and the reasoning is colder than "open source is nice."
First, the practical driver: my entire bottleneck is getting real traces from real systems, and traces are sensitive data. The single most effective thing I can say to someone hesitant is "the data never has to leave your machine — run the tool locally and read every line of what it does." That sentence only works if the code is genuinely open. A closed binary asking for your production logs is a much harder sell than an auditable repo. Openness isn't generosity here; it's the price of the trust my ask requires.

Second, the honest assessment of what's actually defensible. The detector is a structure-then-semantics cascade — a competent engineer who read my posts could rebuild the idea in weeks. If my moat were the code, I never had one. What can't be forked in a minute: a public track record of pre-registered criteria, published negatives, and a README that documents the failure modes on the same page as the wins.

That's slow to build and impossible to copy retroactively — a fork gets my code, not my history of not lying about it.
Third, open code makes my honesty claims falsifiable. "Zero false positives on synthetic, semantic layer weak on real data" isn't something you have to take my word for — you can clone it and check. For a brand built on anti-self-deception, being checkable is the feature.

What I'm not claiming: that this guarantees a business model. Monetization (hosted, team features, whatever it becomes) is an open question I'll answer with real users, not in advance. But the order of operations feels right: trust first, traces second, revenue model third — and open code buys the first two.

Repo: github.com/JEONSEWON/Clew-by-Custos

BuildInPublic #AIAgents #OpenSource #DevTools

I opened my tool to every framework. The disciplined next move was writing zero more code.

JEONSEWON — Tue, 30 Jun 2026 14:17:08 +0000

After I generalized my tool's input to read standard trace files, the obvious next step screamed at me: keep going. Build adapters for n8n, for Dify, for people who wrote their agents in plain Python. Cover everything. More coverage, more reach — right?
I stopped, because that instinct is exactly the one that killed my first project: building features nobody had asked for, polished to perfection, used by no one.

The split that made the decision for me: there's a standard ecosystem and a non-standard one. LangGraph, CrewAi, AutoGen, LlamaIndex all emit OpenInference traces — so one entrypoint opened all of them at once. Building that was demand-driven: I'd hit the real friction of "I can't get your trace in" with actual people. That's a wall the market put in front of me, so removing it is justified.

The non-standard side — n8n, Dify, hand-rolled Python — uses bespoke formats, and each needs its own adapter. Here's the trap: I could build all of them now. It would feel productive. Tests would go green. And not one of them is blocking a real user today, because I don't have a real user on those frameworks asking. Building them up front isn't coverage — it's speculation with a compiler.

So the rule I'm holding: build the door when someone is actually standing at it. The standard ecosystem had people at the door. The non-standard adapters get built the day a real user shows up with that format and that pain — not a day before. Their trace will also tell me exactly what their format looks like, which beats guessing.
"What's the disciplined thing to build next?" often has the answer "nothing yet." The hard part of solo building isn't writing code. It's not writing the code that feels productive but isn't wanted.
Code: github.com/JEONSEWON/Clew-by-Custos

BuildInPublic #AIAgents #DevTools #SoftwareArchitecture

I almost rewrote my engine. It was never the thing that was broken.

JEONSEWON — Thu, 25 Jun 2026 16:10:56 +0000

My tool could only ingest traces from one framework, and that narrow door was strangling adoption. The obvious fix screamed at me: the detection engine must be too framework-specific — rewrite it to be agnostic. Days of work, touching the most sensitive code I have (the frozen detector). I was ready to start.

Instead I spent one hour doing nothing but reading. Read-only: no edits, no plan, just trace exactly where the framework-coupling actually lived.

It wasn't in the engine. The detector was already framework-agnostic — waste looks the same whether it came from LangGraph or CrewAI, and the core only ever saw normalized spans. The coupling was a single thing: there was no entry point that read a trace file. The "rewrite" I was about to do would have churned days of risky changes through code that was already correct, to fix a problem located somewhere else entirely.
The actual fix was adding one entrypoint — hours, low risk, the frozen engine untouched. The hour of diagnosis didn't just save time; it saved me from confidently solving the wrong problem, which is the more expensive mistake. A rewrite that "works" but addressed a non-issue still leaves you with the original bottleneck and a pile of new risk.

The discipline I'm taking from this: when something feels broken, the instinct is to open the file you suspect and start changing it. The cheaper move is to first prove where the problem is, read-only, before a single edit. Most of the time the thing you were about to rewrite was never the thing standing in your way.

Code: github.com/JEONSEWON/Clew-by-Custos

My test suite said "pass." It was lying by 19 tests.

JEONSEWON — Tue, 23 Jun 2026 16:30:06 +0000

Before adding a feature, I ran the suite to confirm a clean baseline. It came back 139 passed, 3 skipped. Green-ish, no failures. The easy move is to nod and start building.
I didn't, because the number was wrong. My known-good baseline is 158. A run that comes back at 139 isn't "close enough" — it's 19 tests short, and I have no idea whether those 19 are passing, failing, or simply not there.

The cause turned out to be boring: a fresh virtualenv hadn't installed the adapter extras, so every OpenTelemetry-dependent test silently skipped or never got collected. Nothing errored. The run looked healthy precisely because the missing tests couldn't fail — they just weren't run. That's the most dangerous kind of green: not "everything passed," but "the things that would have caught a problem weren't even asked."
Why stop the whole feature for this? Because my entire way of working is "compare before and after my change to see what broke." If the starting point isn't the real 158, that comparison is meaningless — I'd later have no way to tell whether my new code broke something or whether it was already broken. A baseline you can't trust poisons every result built on top of it.

Installed the extras, reran: 158 passed, 0 skipped. Now there's a real starting line. Only then did I let myself build.
The lesson isn't about pytest. It's that "no failures" and "everything was actually checked" are different claims, and the gap between them is exactly where false confidence lives. Count what should be there, not just what's red.
Code: github.com/JEONSEWON/Clew-by-Custos

BuildInPublic #AIAgents #Testing #DevTools

I almost shipped a parser that couldn't read a single real file

JEONSEWON — Sat, 20 Jun 2026 11:32:58 +0000

I needed my tool to accept trace files in a standard format (OTLP / OpenInference) so any framework could feed it. I read the spec, pictured the JSON structure, and was about to write the parser against that mental model. Then I made myself do one annoying thing first: dump the actual output and compare.
Good thing I did. Almost every assumption was wrong.

I'd assumed uppercase-hex trace IDs, UnixNano string timestamps, and a nested resourceSpans structure — straight from the spec. The real SDK output had 0x-prefixed IDs, ISO datetimes, and a flat array. The "standard OTLP-JSON" I'd designed against turned out to be a spec example that no tool actually emits. If I'd written the parser against my mental model, it would have passed my own hand-made fixtures and then failed on every real file. The generalization would have been fake.

The second dump taught me something bigger. I'd assumed users have trace files sitting on disk. They mostly don't — OpenInference's own examples ship spans to a dashboard over HTTP. So the premise of "send me your trace file" was itself off. The most valuable output of this whole task wasn't the parser — it was a 3-line snippet that lets a user produce a file in the first place. A parser with no "here's how to make the file" guide is a tool nobody can use.

So the scope quietly shifted: support the one format real SDK output actually uses, reject the others with a clear "convert it like this" error instead of failing silently, and ship the copy-paste snippet — then verify that snippet actually runs end to end, not just the parser's unit tests.
The discipline here is the same one I keep relearning: a result that's correct against data you imagined tells you nothing about data the world produces. Verify the real shape before you build for it.

(Caveat kept explicit: one export format supported today; dashboard/proto exports get a conversion message, not silence. The detection engine and its known limits are unchanged — this was about the door, not the detector.)
Code: github.com/JEONSEWON/Clew-by-Custos

"Why don't you just use a better embedder?" — the most reasonable suggestion I keep refusing

JEONSEWON — Fri, 19 Jun 2026 08:57:05 +0000

Every time I show the E3 finding — that my semantic layer barely separates genuinely-distinct outputs on real, same-topic data — someone gives the same sensible advice: just swap in a stronger embedding model. It's a good instinct. I keep saying no, and the reason is more interesting than the suggestion.

The semantic layer's job is to confirm whether two outputs are really redundant. On synthetic data it worked. On real same-topic text, outputs share so much vocabulary that almost everything scores "similar," and my threshold (φ=0.514) can't tell waste from normal progress. So yes — a different embedder might separate them better.

Here's the trap. I have exactly three real pairs that expose this. If I start swapping embedders and tuning until those three separate cleanly, I haven't fixed the detector — I've fitted it to three examples. The number would look great and mean nothing, because I'd have chosen the instrument after seeing what makes the result pretty. That is the precise mistake that killed my first project: a signal that looked strong because it was quietly shaped to the data in front of me.

Freezing the embedding model isn't stubbornness about this particular model. It's refusing to change the ruler after seeing the measurement. The honest fix for E3 isn't a better embedder chosen against three examples — it's real traces across many topics and domains, enough to see the actual distribution, and then a pre-registered recalibration I commit to before looking.

Which is why the bottleneck isn't a model on Hugging Face. It's traces from people running real multi-agent systems. The better embedder might genuinely be the answer — but I only get to find that out honestly once, so I'm not spending it on n=3.
Code, the E3 log, and the frozen params:
github.com/JEONSEWON/Clew-by-Custos

The 4-step ritual I use so an AI coding agent can't hand me a green checkmark that lies

JEONSEWON — Thu, 18 Jun 2026 15:51:38 +0000

I built my whole product with an AI writing the code, and the scariest failure mode isn't bugs — it's a test suite that passes for the wrong reason. Here are all four steps, and the real moment each one saved me.

Pre-register the pass/fail criteria, frozen in git, before I see any result. If I define "success" after seeing the output, I'll unconsciously pick the definition the output already satisfies. My first project died exactly here: a signal that "passed" was secretly measuring trace length, not failure. Writing the bar down first is the cheapest insurance there is.
Commit the criteria and run the baseline tests. A known-good starting line the agent can't silently move. If a test goes green, I want to know it went green today, not that it was already green.
Ask the agent for a PLAN, not code. Most "the AI wrecked my codebase" stories begin with approving a 400-line diff nobody read. A plan is reviewable in two minutes. This is also where I catch the dangerous "fixes" — once, my agent's fix made the numbers go green by quietly deleting the hard case that exposed a real weakness. I caught it in the plan, not in production.
Review the plan, push back, then approve — manually. The agent never writes code I haven't already read a plan for. Auto-approve is how you wake up to a system that's confidently wrong.
It feels slow. It's the opposite: an hour here saves a week of debugging something that looked correct the whole time. Code's public: github.com/JEONSEWON/Clew-by-Custos

BuildInPublic #AIAgents #ClaudeCode #DevTools

I spent a day teaching my tool to make the problem look smaller

JEONSEWON — Wed, 17 Jun 2026 10:38:02 +0000

Every "we found $X of waste!" tool has a quiet incentive: the bigger the number, the more impressive the demo. So when I built the report that turns a trace into a waste summary, I deliberately wired in four rules whose only job is to keep that number honest:

1.Count only the re-run, never the original. If an agent does real work once and then redundantly repeats it, only the repeat is waste. Charging the legitimate first run to the "waste" column is the easiest way to double your headline number — and a lie.

2.One row per wasted span (dedupe). When one repeat pairs with several earlier runs, the naive report lists it multiple times and the waste visually balloons. I assert in a test that rows = actual wasted spans.

3."unknown" instead of a guess. No token count captured? The report says unknown. It does not invent a plausible number to fill the cell.

4.The report prints its own frozen parameters (φ, N, embedding model) in the header — so anyone reading it knows exactly which settings produced it.

None of this makes the demo flashier. That's the point. In a category where every competitor is incentivized to inflate, a report that refuses to exaggerate is the differentiator — and it's enforced in tests, not in good intentions.

Code's public: github.com/JEONSEWON/Clew-by-Custos

BuildInPublic #AIAgents #LLMOps #DevTools

I passed all 5 of my real-AI tests. The most useful thing I found is that half my detector barely works.

JEONSEWON — Tue, 16 Jun 2026 14:50:16 +0000

I finally pointed my multi-agent waste detector at real AI output for the first time — five scenarios, real Claude traces, no synthetic data. Four of the five came back exactly as expected on the first run. The fifth (a "looks like a repeat but isn't" case) threw a false positive.

The tempting move there is to nudge my similarity threshold until the false positive goes away. I didn't touch it. I diagnosed first — and the culprit turned out to be my test design, not the detector. I rebuilt the test properly; the second run passed 5/5, with the threshold (φ=0.514) untouched from start to finish.

But passing 5/5 isn't the story. This is:
I measured the similarity of output pairs that are genuinely not redundant — different content that should look distinct. On real, same-topic output, 100% of them scored above my "this is redundant" threshold. Not a near-miss band like my synthetic work predicted — everything, comfortably above the line.

Here's what that means, and it's uncomfortable: my zero false positives right now are entirely the work of the structural layer (which simply isn't raising candidates). The semantic layer — the part that's supposed to confirm "yes, this really is redundant" — has almost no separating power on real same-topic text, because outputs on one topic share so much vocabulary that they're all similar by default. The moment the structural layer surfaces one borderline candidate, the semantic layer rubber-stamps it.

My synthetic mock-exam hid this completely. Clean separation between "planted near-identical" and "unrelated clean" is exactly the artificial sharpness real data doesn't have.

The honest boundary, updated: four detection paths work on real traces with zero false positives from the structural layer. The semantic layer's separating power on real data is not demonstrated — the opposite is. This is one topic and five traces, so it's a strong signal to act on, not a verdict.

What I'm explicitly NOT doing: redesigning the semantic layer to fix this now. Tuning it against one topic and five traces is just overfitting with extra steps — the same trap that killed my first project. The real fix needs real traces across several topics and domains, which only come from people running actual systems.

The whole thing — code, the synthetic GO, the real-probe log, and this E3 limitation written out in full — is public: github.com/JEONSEWON/Clew-by-Custos. If you run a multi-agent system and can share a trace, that's the one input that turns this from a suspicion into an answer.

"How do you build a test your detector can't game?"

JEONSEWON — Sun, 14 Jun 2026 08:53:08 +0000

If you build a waste detector and then build your own test set, there's a trap waiting: you can pass without your detector actually working.
Here's how. My "waste" examples have repeated steps; my "clean" examples don't. Looks reasonable — until you realize a dumb detector that just flags any repeat now scores perfectly, and the semantic layer (the hard part, the part that decides if a repeat is genuinely redundant) never has to do anything. You'd ship a "0.85 F1" that's really measuring "did the trace repeat," which is the exact mistake that killed my first project.

The fix is paired design: every clean example shares the identical structure as a waste example — same nodes, same repeats, same handoffs. The only difference is whether the repeated work is semantically redundant or real progress. That single constraint forces the semantic layer to be the actual judge, because structure alone can no longer tell the two apart.

Building the test you can't cheat is harder than building the detector. It's also the only reason the detector's score means anything.

My AI-agent waste detector scored zero false positives. Then I ran it on a real trace.

JEONSEWON — Sat, 13 Jun 2026 06:27:34 +0000

My detector passed every synthetic test with zero false positives. Then I pointed it at one real trace and found a crack.
This is the honest version of where I am. I'm building Clew — a tool that finds the redundant loops, re-queries, and handoffs that silently burn tokens when multiple AI agents work together. No crash, no error, just two agents quietly re-doing each other's work while the token bill climbs.
I build in public, and I publish the negatives. So here's the whole arc, including the part that isn't working yet.

First, I killed my own hypothesis

The original idea wasn't waste detection at all. It was failure prediction: watch the behavior between agents and forecast multi-agent failures before they happen. The differentiator was a single metric built on two signals — structural cycles in the inter-agent message graph, and the decay of novelty in embeddings.

Before I ran anything, I pre-registered the success bar: AUC ≥ 0.80. I numbered every change and kept the signal code physically separated from the labels so I couldn't leak my way to a good number. Then I ran it on MAST-Data — UC Berkeley's dataset of 1,600+ real multi-agent traces across 7 frameworks[(Cemri et al., arXiv:2503.13657)](url)

Result: AUC ≈ 0.455. A coin flip.

It got worse. The signal correlated with trace length at r ≈ 0.86 — it was mostly measuring how long a trace was, not whether it failed. Correcting for that dropped AUC to 0.42 and reversed the direction: successful traces actually showed more decay (p ≈ 0.013).

The honest read: not disproven, but unvalidated. On this implementation, on this data — negative. So I shut it down. And I counted it as a win, because I got a fast, honest answer in weeks instead of building a dashboard on a metric that secretly measures string length. That experiment became the DNA of everything since: design the experiment that's allowed to kill the idea.

The pivot: from predicting failure to cutting waste

The intuition behind v1 — that you need structure and meaning — turned out to be right. The implementation was wrong. An external paper confirmed the shape of the fix: an unsupervised cycle-detection framework that runs structure first, then semantics [(George et al., IBM Research, arXiv:2511.10650)](url). On their benchmark of 1,575 LangGraph trajectories, the cascade hit F1 0.72 — versus 0.08 for structure alone and 0.28 for semantics alone.

To be clear: that 0.72 is IBM's result on IBM's data, not mine. I keep that line bright in everything I publish. But it told me what I'd gotten wrong (I'd summed the signals instead of cascading them, and looked at global trends instead of local repeats), and it pointed at a sharper wedge: stop predicting failure, start detecting the redundant loops and handoffs that burn tokens. That's measurable. It speaks in dollars.

Building it so I couldn't fool myself

Before writing a line of detection logic, I built the thing that makes the validation trustworthy. Pre-registered GO/KILL criteria, frozen in git before I looked at any result. A leakage guard enforced not as a policy but as a failing test: the detection code physically cannot import or read the label files. Parameters chosen on a dev split only; the evaluation split touched exactly once, after freezing.

Two moments from that build are worth telling, because they're the actual product.

The fix that passed but was a lie. During calibration, a clean case kept tripping the detector: two lookups with the same schema but different values (think customer A vs customer B). My first "fix" diversified the data so the numbers turned green — except that didn't solve the limitation, it deleted the case that exposed it. Shipped as-is, the detector would have flagged legitimate work as waste. The real fix routed the decision from the semantic layer (which can't tell those apart) to the structural layer (which can): a re-query only counts if the input is identical. Then I put the hard case back in to prove the gate worked.

The audit that caught a blind spot before launch. A recall check right before freezing found that one of four waste patterns — regenerative handoffs — scored 0/10. Diagnosis: it's structurally identical to a normal handoff, so there's no candidate path for it. Rather than quietly drop it to protect the score, I explicitly de-scoped it, left it in the dataset, and let those 10 misses count against my aggregate F1.

Carrying that penalty, the single-shot evaluation came back GO: F1 0.857, zero false positives, 100% recall on the three in-scope patterns. The out-of-sample numbers reproduced the dev numbers exactly — so no overfitting to my own dev split.

And every one of those numbers is synthetic, held-out, three patterns. That matters for what comes next.

Then reality showed up

A GO on synthetic data isn't a product. So I pointed the detector at real LangGraph instrumentation for the first time. The structural machinery held up better than I expected — but it surfaced three things my synthetic tests had never touched:

Spans collapse weirdly. Real LLM calls instrument under a single model-class name, so the detector saw "the same node three times" and cried wolf. Fixable: fold the LLM sub-spans into their parent node while preserving tool spans.
Routers fake repeats. LangGraph's conditional-edge functions instrument as spans too, and a repeating router looks like a repeating node. Fixable, and principled: a span that burns zero tokens is, by definition, not token waste — so exclude it.

Those two are confirmed engineering fixes. The third one I can't fix from my desk:

The threshold might not transfer. My similarity threshold was calibrated on synthetic data, where "redundant" and "distinct" separate cleanly. On real output, same-domain results cluster in a middle band that sits right on top of my threshold. Stripping the JSON scaffolding accounts for some of it (~0.2 of the similarity), but not all. With a sample size of three, this is a crack to take seriously — not a verdict. And there's exactly one way to find out.

What's true, and what isn't

I think the honesty boundary is the whole point, so here it is plainly.

True today: a structure-then-semantics cascade catches three planted waste patterns in held-out synthetic traces with zero false positives, passing a pre-registered bar. On real instrumentation, the structural mechanism is confirmed (spans collapse correctly, genuine repeats still fire, router false-positives are removed). And there's a working tool: feed it a trace, get back a waste report in minutes.

Not true yet: that it works on real production traces. That it saves real tokens — I have zero measured savings. That the threshold generalizes — unverified, and currently my biggest open question. Zero users.

The tempting move is to nudge the threshold up by a hair so the borderline cases fall the right way. That's exactly the post-hoc overfit that killed v1. The threshold has to be re-derived honestly, from real distributions — which I don't have.

The ask
That's where you come in, and it's a genuine ask, not a pitch.

If you run a multi-agent system — LangGraph, CrewAI, AutoGen, something custom — and your token bill has been creeping up: send me a trace. I'll run it through Clew and send back a free report — where the redundant work is, and what it's costing you in tokens. If your data can't leave your environment, I'll send you the tool to run locally and you just share the numbers.

I'm not selling anything. The one question my synthetic tests genuinely cannot answer is whether this holds on real output distributions — and I'd rather find that out honestly than pretend it's already settled.

The most valuable thing I've built so far isn't a clever detector. It's a way of working that doesn't let me lie to myself about whether the detector is any good. If real traces break it, you'll read about that here too.