I built SkillPilot over about three weeks with Claude Code, a set of workflow skills, and a lot of arguing. It is a real product: two Next.js apps on Cloudflare Workers, Supabase for auth and entitlement, Stripe as merchant of record, 24 sellable items, real money.
This is what I actually learned, including the parts that make agentic coding look worse than the marketing does.
The initial build is genuinely fast, and the token bill is brutal
The first 80% arrived at a speed I have not seen before. Schema, auth, content pipeline, checkout, webhook, entitlements — days, not weeks.
The cost is not subtle. Long agentic sessions burn tokens at a rate that will surprise you if you are used to autocomplete. A single deep session on the payments layer cost more than a month of my previous tooling. That is a real number in a real budget, and nobody's demo mentions it.
It is worth it for the initial build and it stops being worth it for small changes. Knowing which of the two you are in is most of the skill.
It rewards knowing exactly what you want
The single biggest predictor of a good session was how precisely I could state the outcome before starting.
"Add coupons" produced something plausible and wrong. "A coupon is either a gift that writes an enrolment directly, or a discount that forwards to a Stripe promotion code and is only consumed once the webhook confirms payment — and a gift must never go through the gateway, because a 100%-off Checkout Session settles to no_payment_required and our webhook only accepts paid" produced
something I shipped.
The second version is not a longer prompt. It is a design decision I had already made. The model was fast at building it and would not have chosen it.
Which brings me to the uncomfortable part.
It does not replace engineering experience
Not yet, and not close. Three examples from this codebase, all of which needed someone who has been burned before:
The write order in the payment webhook. Purchase first, enrolment second. A crash between them leaves someone who paid without access — recoverable by hand.
The other order leaves access granted with no accounting record, which is not.
Nothing in the prompt implies that ordering; you have to have thought about which failure you would rather explain to a customer.
The gift coupon's opposite order. For a gift, the redemption is written before the enrolment — the reverse of the purchase path — because there the redemption is what rations the access. Write the enrolment first and you hand out the gift, then discover it was not available. I only found this because a security review made me ask why the two paths differed.
A database function instead of application code. Checking a coupon's limit and recording the redemption has to be one atomic step. Read the count in the worker, decide, then write, and a code posted to a forum hands out as many enrolments as requests arrive in between. A forum link is a burst of concurrent requests. Verified: 40 simultaneous redemptions against a cap of 10 grant exactly
10.
An agent will happily write the racy version. It reads fine.
AI interfaces are weak until you give them something concrete
Every UI I got with a generic prompt looked like every other AI-generated UI:
rounded cards, gradient buttons, an emoji, and no point of view.
What changed it was two things.
A design skill in the loop — an actual document describing type scale, spacing, colour roles and when to use which chart form, rather than "make it look good".
Concrete references. "Take the density and restraint of Linear's marketing site" moved the output further in one prompt than a paragraph of adjectives ever did. Naming a real product you both can look at replaces a lot of hand-waving.
With those two, I got a usable design on roughly the first attempt instead of the fifth. Not finished — usable. Then a few rounds of "this section is 12 screens deep on mobile, fix the hierarchy" got it the rest of the way.
That refinement loop is where the last 20% lives, and it is slower and less fun than the first 80%.
And then the part that actually matters
Here is the thing I would tell anyone shipping something agentic-built and taking money for it.
On the day I made the first real purchase, the suite was green. tsc was clean. The checkout had been dead in production for days.
1,805 unit tests. 103 test files against 306 source files. All passing. Three bugs, and not one of them could have appeared in a unit test:
-
A Content-Security-Policy header killed the payment.
form-actiongoverns the redirect to Stripe, and ours did not allow it. "Continue to payment" did nothing, in every Chromium browser. No error, no exception, no failing test. - The dashboard gave away the whole pack to anyone who bought a single course, because the access check matched on pack and ignored the course.
- One of four purchase screens quoted the wrong price, because it rendered before the coupon was validated.
I found all three by buying my own product with a real card.
And then it happened again. Weeks later I discovered analytics had been silently dead: the tracker loaded from one domain and sent events to another, and the CSP allowed only the first. Script loads, window.umami exists, page perfect, dashboard empty — which reads as "no traffic", which is exactly what a new site expects to see. Same class of bug. Second time.
The lesson is not "write more unit tests". It is that a certain kind of bug lives entirely outside the unit: browser policy, a real session, a real redirect, a real payment, a real card. The only thing that catches them is exercising the actual path in the actual browser against the actual production config.
So: end-to-end tests, and more of them than feels reasonable. Not because the agent writes bad code — the code was fine — but because agentic development moves fast enough that you accumulate integration surface faster than your intuition about it.
What I would tell myself three weeks ago
- Decide the hard things yourself. Failure modes, data ownership, what happens when the money and the access disagree. The agent builds; it does not choose.
- Budget for tokens like a line item. Expensive up front, then taper.
- Give the design system real references. Name a site.
- Test the browser, not the function. Your suite being green is evidence about your functions, and almost none about your product.
The productivity gain is real and large. It is not "no engineers needed". It is closer to: the typing stopped being the bottleneck, and every decision I was already bad at is now the whole job.
I run SkillPilot — researched tech career roadmaps where every salary figure cites its source and the date it was retrieved. The roadmaps are free.
Top comments (1)
1,805 tests green and checkout was dead. CSP blocked the Stripe redirect and the suite never opened a browser.
Green on functions doesn't mean much if it never touched the part that was broken. Lock the done check first, the click a person actually hits with a real card on the real domain, then a separate go to build.