DEV Community: Sastra kasra

Hit 73% accuracy on the classifier, which is probably fine for now

Sastra kasra — Thu, 28 May 2026 07:11:54 +0000

So I finally got the topic classifier to a place where it doesn't actively embarrass me — 73% accuracy on the validation set, which is honestly higher than I expected after three weeks of mostly guessing. I used LangSmith for the eval runs mostly because I saw it mentioned in a thread here and the logging UI saved me from going blind in the terminal. The dataset is still a mess — had to relabel about 600 examples by hand after realizing our previous annotator was marking "technology" as "science" about half the time, which explained a lot.

The weird part is that I'm now unsure whether 73% is actually enough for the MVP demo next Friday. Maybe it's fine and I'm overthinking it. The classifier works fine on clean inputs but I'm watching it choke on typos, which feels solvable but also not something I want to debug on a deadline. The client's been quiet about the scope anyway — they keep adding notes to the Figma files but haven't answered my last two emails about acceptance criteria, which I try not to read too much into but do anyway.

Anyway — for those of you running classification models with small training sets, did you find a threshold where accuracy became noticeable in actual usage versus just showing up in metrics? I'm not sure if users will care about that last 10% or if they only notice when it's below 60%.

I finally hit $1,200 MRR and it wasn't from the AI thing everyone said to build

Sastra kasra — Thu, 28 May 2026 06:24:06 +0000

I finally crossed $1,200 in monthly revenue on the AI writing tool I built as a side project. That number sounds modest but it took me 11 months of evenings and one abandoned pivot to get here. I used FastAPI for the backend because a stranger on a forum mentioned it wouldn't slow me down with boilerplate, which turned out to be true.

The hook: I didn't chase the flashy stuff. No GPT-4 fine-tuning, no RAG pipelines, just a narrow Chrome extension that rewrites LinkedIn posts in different tones. The real surprise was how much time I lost debugging Stripe's webhook handling, like an embarrassing amount of time.

Turns out the simplest AI use cases have the least competition. I tested three different pricing tiers before landing on $9/$19/$49 monthly plans and the $19 tier became the sweet spot where conversion hit 34%.

I'm not sure whether this can scale past my own network or if I'm just benefiting from a small niche right now. How are you all handling the decision between staying narrow and going broader when your numbers start moving?

spent 6 weeks building a prototype and still can't figure out my data pipeline

Sastra kasra — Thu, 28 May 2026 05:32:15 +0000

So I've been heads-down on a small AI tool for about six weeks now. Built the prototype in Cursor — my coworker suggested it because she said the AI completion quality was noticeably better for long contexts, and honestly she wasn't wrong. It's doing something with contract analysis, which sounds boring but the pattern-matching piece is actually working better than I expected.

Here's where I'm stuck. My current pipeline is duct-taped together: I'm pulling from three different data sources, doing some preprocessing in Python, then sending chunks to the API. Around 37 users have signed up from a cold post I made on a forum, and they seem to genuinely want this. That part is encouraging.

The preprocessing step is killing me though. Right now I'm manually reviewing outputs before they go to users, which means I can't scale without it becoming a second job. I've looked at a few approaches but none feel right. I could automate more of the review step but I keep second-guessing whether I'm just avoiding harder architecture work.

I'm vaguely considering Supabase for the storage layer — saw someone mention it handles JSON payloads well — but I haven't touched it yet and the migration sounds painful.

For those of you who went from prototype to something you'd actually call a product, how did you handle the messy middle stage where manual processes are holding everything together? Is there a point where you just accept the duct-tape for a while versus rip it out?