DEV Community: Artem Kireev

Black Friday code freeze: what design can still change

Artem Kireev — Fri, 13 Nov 2026 16:31:36 +0000

Black Friday is November 27 this year, and most freeze guides put the lock on theme, app and checkout changes between November 13 and 20. Today is the 13th. If your team treats "freeze" as "stop thinking about the storefront", you are about to spend the busiest two weeks of the year leaving known checkout problems in place.
This article argues that a freeze should lock code, not design. It shows which checkout and product page fixes are safe because they are content, not code, ranks them by what the research says costs you orders, and gives a plan for the 14 days left.

A freeze protects code, and that is all it protects

The reason for a freeze is simple. A new bug in browsing or checkout costs the most when traffic peaks. The Good's holiday code freeze guide says it plainly. Feature changes that need code wait until the freeze ends.

Look at how those guides define the line, though. The Shopify-focused Craftshift 2026 plan sorts work by how fast you can undo it. Theme code, app installs, checkout customizations and Shopify Functions are red, because rollback takes hours. App settings and styling are amber, with rollback in minutes. Prices, banners, product copy, images and collection membership are green, and you can undo them in seconds. Its dates are one vendor's recommendation, not Shopify policy, but the logic holds. Risk follows rollback time, not whether a designer touched it.

The same Craftshift plan locks configuration on Friday, November 13 and code on November 20. So you have a week where content and data stay open, and a few days where only critical fixes ship. Too many teams spend that week waiting.

That is a mistake, because checkout problems are mostly not exotic. They are well documented, and several of them live in copy and data.

Rank the fixes by what actually loses orders

Start from the evidence instead of the backlog. Baymard's cart abandonment research averages 50 studies at a 70.22% abandonment rate. Among shoppers who intended to buy, it lists the reasons. 40% cite extra costs such as shipping, tax and fees, 20% slow delivery, 19% distrust of the site with card details, 18% forced account creation, 17% a long or complicated checkout, 12% a total cost that was not visible up front, and 13% an unsatisfactory returns policy.

Now sort those by rollback speed. Extra costs and a hidden total are a data and copy problem for many stores. Think shipping thresholds, delivery estimates, a line on the product page. Delivery speed is partly a promise you can state more clearly. Returns policy is a page. Trust is wording, badges and payment method labels. Account creation is a setting on Shopify and most other platforms. Only a long checkout is a real code problem.

Baymard also says that fixing solvable checkout usability issues could raise conversion by 35.26% for a large site. Treat that as an upper bound for sites with many unfixed problems, not a forecast. Baymard has also said it is retesting its checkout research and expects new findings in the second quarter of 2027, so the exact percentages may move. The ranking of reasons has been stable for years, which is why I trust it for a two-week decision.

The point is that most of the top reasons for abandonment can be addressed with changes your freeze policy already calls green or amber.

Show the total cost before the cart

Since extra costs are the top reason, start there. A shopper who sees a $9 shipping fee for the first time on the payment step reads it as a trick, however fair the fee is.

The no-code fixes are boring and effective. Put the free shipping threshold in the announcement bar and on the cart drawer with a live "you are $14 away" line, which is a content block on most themes. Add estimated delivery dates by region to the product page, because for Black Friday orders the question "will it arrive by December 24?" is the one on people's minds. State your holiday returns window in one sentence near the buy button. Make guest checkout the default and hide account creation until after payment.

If you cannot show exact tax, show a range and say it is an estimate. A range that is roughly right beats a surprise that is exactly right.

Check each change against one question. Could this break a payment? Copy, banners and threshold messages do not touch the payment flow. A new fee calculator does, so it waits until December.

Test less, but do not stop looking

Many guides argue about whether to run A/B tests during the holidays. The honest answer is that most stores should not start new ones now. Convert's guide says Black Friday through Cyber Monday is a two to four day window of hyper-shopping, and that conversion rate, average order value and revenue per visitor swing sharply, with flash deals, ad spend and competitor offers all changing the conditions. A test that starts on November 24 and ends on November 30 tells you about Black Friday, not about your store in March.

The Good takes the opposite line. More traffic means faster significance, and "your biggest risk is doing nothing". Both can be right. That article is from 2021 and gives no data, so I read it as a reminder, not as proof. Traffic does speed things up, but only for questions that matter in a normal week.

My rule is this. If a test can reach a readable result by about November 20, let it finish. If not, ship the obviously better version to everyone only when you have a documented reason, not a hunch, and write it down. Baymard's reasons are such a reason. A hidden shipping cost is not a hypothesis to test during Black Friday.

What you should do is watch. Heatmaps, session recordings and customer service tickets cost nothing in risk terms. Set up your conversion tracking before traffic arrives, as The Good advises, so that you have clean data for the first week of December.

A plan for the next 14 days

Here is the sequence I would run, assuming a Shopify store and a team of one to three people.

Today and tomorrow, take a snapshot. Duplicate the live theme as a restore point, export your settings and list who can publish. The Craftshift plan recommends the same, plus one monitoring dashboard and a rollback rule agreed in advance, such as "revert if checkout completion drops by a set amount over an hour".

By Monday, audit the green list against Baymard's reasons. Open your store on a phone, add a product to the cart and go through checkout as a stranger. Write down every moment where you learned a cost, a date or a policy for the first time. Each one is a copy or data fix.

By Wednesday, ship the fixes. That means a threshold message, delivery dates, returns line, guest checkout default, and payment method icons. Log each change with a time and the person who made it, so a bad hour is easy to explain.

From November 20, touch nothing in code. Content stays open, including prices, stock, banners and emails. Keep the thaw date at December 1 or later, since Cyber Monday orders arrive until midnight.

Last, keep a list of the red items you wanted and could not ship, such as a faster checkout step, a new cart design or an app. That list is your January plan, and it will be better informed because you watched real peak traffic first. For help judging whether your conversion numbers are healthy when peak week ends, my Shopify conversion benchmarks piece explains why published figures disagree.

Originally published on kireev.us.

Click to cancel for SaaS: redesign your cancellation flow

Artem Kireev — Tue, 10 Nov 2026 15:19:54 +0000

New York City's click-to-cancel rule took effect on October 1, 2026. It says that if a customer signed up online, they must be able to cancel online. Most coverage is written by lawyers for lawyers, and it ends at "review your terms of service."

This article is for the people who own the screens. It argues that the real compliance work is a design audit of every path into your subscription, and that the federal rule being dead does not take you off the hook. It helps SaaS founders, product managers and designers who sell to individuals or small teams.

The federal rule died, the obligation did not

A lot of product teams relaxed in 2025. The Eighth Circuit vacated the FTC's click-to-cancel amendments in July of that year on procedural grounds, which put the old 1973 Negative Option Rule back in place. According to a summary of the current federal position, the FTC opened a new rulemaking in March 2026, comments closed in April, and nothing has replaced the vacated rule.

That is true and mostly irrelevant. The same summary points out that the Restore Online Shoppers' Confidence Act (ROSCA) was never affected. It still requires clear disclosure, informed consent to charges and a simple way to stop recurring charges. Section 5 of the FTC Act still bans unfair and deceptive practices. State auto-renewal laws, with California's the most developed, apply whatever Washington does.

The enforcement record shows how seriously this is taken. In September 2025 Amazon agreed to pay $2.5 billion to settle FTC claims over Prime: a $1 billion civil penalty plus $1.5 billion in refunds. The complaint described a cancellation path of four pages, six clicks and fifteen options. The order requires a cancellation method that matches how the customer signed up and is not difficult, costly or time-consuming.

Now add a city. On October 1, 2026 the NYC Department of Consumer and Worker Protection rule took effect. The city's own page says businesses must offer a "straightforward cancellation process in the same method as sign up", with civil penalties starting at $525 per violation and possible refund liability. One city, one rule, and it is the first municipal version in the country according to the National Law Review.

What "same method as sign up" means on a screen

Read that sentence as a designer, not a lawyer. It is a parity test between two flows. If a customer needed three steps and a card form to start, they should not need a phone call and a support ticket to leave.

The part the legal write-ups skip is that most SaaS products have more than one front door. A customer can start a trial on the marketing site, from inside the product, through a sales-assisted checkout, through an app store, or through a partner marketplace. Each door is a "method". Each one needs a matching exit, or a clear and visible route to wherever the exit lives.

This is where teams get caught. Self-serve signup works beautifully. Cancellation lives in a billing portal that only the original purchaser can see. Annual plans bought through a reseller can only be cancelled by emailing an account manager. None of these were designed as dark patterns. They are the residue of how billing was wired, and a regulator reads the outcome, not the intent.

My advice is to map it as a table. Rows are every way a person can become a paying customer. Columns are the number of steps to start, the number of steps to stop, the channel for each, and who can do it. Any row where stop is longer than start, or sits in a different channel, is a defect. It takes an afternoon, and I would run it before reading a single clause of the rule.

Where the rule stops and your exposure starts

Be honest about scope. The rule covers automatic renewal and continuous service offers sold to consumers in New York City. Pure enterprise contracts negotiated by procurement are a different animal. The National Law Review's account of the proposal describes it as applying to any subscription that qualifies as an automatic renewal or continuous service offer, and the public sources I could read do not say how it treats self-serve plans bought by a freelancer on a company card. I do not know, and I would not bet the answer is favorable.

That ambiguity is the argument for designing to the strictest reading. Self-serve SaaS with a monthly card plan sits closest to the consumer line. Prosumer tools, creator software and small-team plans all live there. If you bill people with a credit card and a pricing page, assume you are in scope somewhere.

Two facts make the cost of getting this wrong higher than the $525 headline. First, the city's page says liability can include refunding the customer. A refund exposure scales with your customer base, not with a per-violation fine. Second, a flow that is clean in New York but painful elsewhere is a flow you still have to defend under ROSCA and your state law. Building a second, hidden "New York version" is more work than fixing the original.

And do not treat a phone number as a fix. A support line is a separate channel. If signup was a web form, a call centre is not parity.

Save offers and the line between persuasion and obstruction

Retention teams will ask whether the exit survey, the pause option and the discount offer all have to go. The sources I found do not address retention offers directly, so this is judgment, not law. My view is that offers are fine and obstruction is not.

The test I would use is whether the customer can finish cancelling without engaging with the offer. A discount shown on the confirmation screen, with a clear "Cancel anyway" button of equal weight, is a choice. A discount that must be declined across three modals, or a "Are you sure?" that moves the real button, is friction by design. The Amazon order is explicit that declining must be possible without manipulative phrasing, and that standard is a good proxy for where regulators look.

Pause and downgrade are the most useful saves because they are real alternatives. A customer who cancels because of a budget freeze may take a three-month pause. A customer who uses one feature may take the cheaper plan. Offer them once, on one screen, in plain language, and let the primary action stay cancellation.

One more measurable point. Do not bill after the first cancellation attempt. Charges that land after a customer has tried to leave are the clearest way to turn a clumsy flow into a violation. Show an unambiguous confirmation on screen, send the same by email with the date access ends, and make the end date match what billing will actually do.

A one-week plan

Day one, build the entry-and-exit table described above and mark every mismatch. Include app store and reseller channels, even when you do not control the billing, because you control what your product tells the customer about them.

Day two, walk the cancel flow with a stopwatch and a fresh account, on mobile as well as desktop. Count taps and screens. If the path to "Cancel subscription" is more than two clicks from account settings, or the word "cancel" is not on the first screen, rewrite the labels before you rewrite anything else.

Day three, strip the flow down to its legally necessary parts: terms of the cancellation, the end date, a final confirm. Then add back at most one save offer and one optional reason question. The reason question must be skippable. Your churn data is more honest when it is.

Day four, fix the account-ownership gaps. If only the billing owner can cancel, say so at the point of need and give a self-serve way to request transfer. Remove every "contact sales to cancel" on plans that were bought without talking to sales.

Day five, test the confirmations. Cancel, wait, and check that the email arrives, the access end date is correct and no invoice is generated afterward. Write the findings down and put the table in a place where whoever changes billing next will see it.

Then repeat it each time you add a plan, a marketplace listing or a new way to buy. Parity is not a one-time audit. It is a property you have to keep, and the cheapest moment to keep it is when the new door is being built.

Originally published on kireev.us.

Figma MCP design system prep for AI agents

Artem Kireev — Fri, 06 Nov 2026 15:20:57 +0000

Figma now lets coding agents write straight to the canvas, and on October 29 it explained how its own editing model learns to do the same. Both posts say agents work "using your components and variables." Neither says what your files need to look like for that to be true.

This article argues that the gap is where teams lose the benefit. The agent follows the rules you write down and the structure you actually built, not the system in your head. It is for design leads and product managers who are about to let an agent touch a production library.

The announcement skips the part you have to do

In March, Figma opened the canvas to agents. Its launch post says the use_figma tool works with your components and variables, and that auto layout, components and variables are the structures agents operate on. Claude Code, Codex, Cursor, Copilot and others can connect.

The same post gives no guidance on preparing a file. It points to the help center and developer docs for the rest. Figma's MCP server FAQ fills in some practical limits. You need a Full seat to write outside your drafts. Fonts must be uploaded to your Figma account. Components must be published before Code Connect mappings can be completed. And the output "is beta quality and may need manual review and cleanup."

That last line matters most. Figma is telling you the agent is a fast first pass, not a finished one. How much cleanup you face depends on inputs you control.

On October 29, Figma's VP of engineering described the model behind the agent in How we're training a Figma-fluent editing model. The post is candid about one thing worth repeating. Post-training improves the model's part of a larger system that also includes canvas context, design system context and tools. Two of those three are your files. A better model does not rescue a messy library.

What goes wrong when you just point an agent at a library

Bitovi ran the agent against real components and wrote up the result in Figma just opened the canvas to agents. The good news first. From an existing Button, the agent produced a component set with correct variants, styles and clean layer names.

The agent "doesn't pick up on a linked design system unless you tell it to. Every time." It used plain frames for some components while using real library components for others. Left to defaults, it hardcoded colors, spacing and type. Bitovi's summary is the line I would put on a wall: the agent optimizes for the literal goal you gave it, not the implicit standards in your head.

Hardcoded values are not a model flaw you wait out. A designer who never opened the variables panel produces the same output. The agent is a very literal junior, so every gap in your system gets reproduced, quickly.

Treat one test with care. Bitovi's piece is one team's experience, and output varies between runs. The pattern is useful. No one has published a failure rate.

Five file problems that make agents guess

Practitioner guides, such as the Figma file best practices in the camoa dev guides and a uxdesign.cc practical guide, agree on the problems. They are blog advice, not Figma documentation, so read them as heuristics. I rank them by cost.

Hardcoded fills and spacing. If a button's fill is a hex value instead of a variable, the agent has nothing to bind to and copies the hex. Bind every fill, stroke, radius and gap to a variable first. This single fix removes the failure Bitovi saw most clearly.

Primitives and semantic tokens matter here. A variable named blue-500 tells the agent a color. A variable named action-primary tells it a purpose. Agents choose by purpose when you ask for "a primary action", so semantic names carry the instruction.

Generic layer names come next. "Frame 234" and "Group 1" give the agent no hint about what a thing is, and the code it produces inherits the vagueness. Name layers the way an engineer would name a component, and match component and property names to your code (a "Style" property in Figma against a "variant" prop in code is enough to break mapping).

Then structure. Agents read auto layout as intent. If a card has fixed positions instead of padding, gap and direction, the agent cannot tell what should grow and what should wrap. Guides also warn against nesting auto layout very deeply, though I have not seen a measured threshold, so treat any specific number as folklore.

Finally, missing descriptions. Guides report that component descriptions reach the agent as context. A two-line note on a star rating component, saying what its states mean and when not to use it, costs nothing and is read every time.

Why rules matter more than a better prompt

Figma's answer to inconsistency is skills, which are markdown files that spell out workflow steps, sequencing and conventions. The foundational /figma-use skill teaches the agent how the tool works, and teams can add their own. Figma says skills make output more predictable because the same prompt can yield different results from a non-deterministic model.

Read that as a statement about where your effort belongs. A prompt lives for one request. A skill lives for every request after it. Bitovi's suggested directives show the right grain. Check for a linked design system before generating anything, and use variables for color and spacing.

This is the logic behind approval UX for agents. You cannot review every action at agent speed, so judgment moves into rules written in advance.

My first skill would be short. Name the library to use and require the agent to load it before drawing anything. State that every color, space and radius comes from a variable, and that a missing token should be reported, not invented. Say which components to use for buttons, inputs and cards, and that plain frames are never acceptable as stand-ins. Describe the naming pattern for new layers. End with what to report: any value the agent could not bind, and any component it had to build.

An agent that lists its own exceptions turns cleanup from a hunt into a checklist.

Where the model gains help, and where they do not

Figma's own numbers show real progress. Typical tasks are about 20% faster than in beta. Against expert designers in head-to-head testing, the new agent beats or matches the previous production agent 62% of the time. User ratings are 33% higher and users undo the agent's work 60% less often.

Those are Figma's internal results, measured on its own tasks, and the post itself says one evaluation is only one input among many. They show the model improving. They do not show your library behaving.

The post also names errors that remain, such as overlapping icons and text and clipped arrows. These are visual faults that structure alone will not prevent. They are why Figma's agent takes a screenshot of its own output and iterates on mismatches, and why you should still look at what it makes.

A better model helps everyone equally. An agent-ready file helps only you, and it makes each model upgrade mean less cleanup.

A one-week plan for a production library

Do this on a duplicate of your library, as Figma recommends, never on the live file.

On day one, audit variables. Find every fill, stroke, radius and spacing value in your core components that is not bound to a variable. Start with the 20 most used components.

On day two, rename. Replace generic layer names in those components, and align component and property names with the code names your engineers use.

On day three, check auto layout in your cards, forms and navigation. Every container should express direction, gap and padding rather than fixed positions.

On day four, add descriptions to components that have states or usage rules. Keep each to two or three lines.

On day five, write your first skill using the template above and run five fixed prompts against the duplicate, such as a settings page, a card variant, a form with errors, an empty state and a mobile screen. Record how many values were hardcoded and how many components were faked with frames. Run the same five after any change to the skill.

That count is your metric. If you already test your system visually, the method in AI agent visual regression testing for design systems pairs well with it, since it catches the clipped and overlapping output Figma admits still happens.

If the first week shows twenty hardcoded values, you have learned something real about your library, and you learned it before the agent shipped the same flaws into a product. That is the point. The agent is a mirror, and it is more honest than a design review.

Originally published on kireev.us.

Shopify script tags deprecation: what breaks on March 1, 2027

Artem Kireev — Tue, 03 Nov 2026 15:24:32 +0000

On March 1, 2027, Shopify stops injecting script tags into storefronts. Since October 1, 2026, apps can no longer create or update them. Anything an old app added to your store through that mechanism, such as a review widget, a popup, a sticky bar or an upsell, will vanish without an error message.

This article is for store owners and e-commerce leads, not app developers. It explains why the quiet failure is the real risk, how to find out in an afternoon which parts of your storefront depend on script tags, and what to do before the deadline.

A deadline that comes with no warning banner

The dates are simple. According to Shopify's developer documentation, from October 1, 2026 the create and update calls for script tags return a permission error on every API version. Existing tags keep running. On March 1, 2027 Shopify stops loading them, and "any dependent features stop for that store."

Here is what makes this different from a normal deprecation. Shopify planned a warning banner for app users and then dropped it. In a September post on the Shopify developer community, a Shopify staff member said the install warning would not ship on October 1, that an earlier email had been overstated, and that the dates had not changed. Shopify said it would watch script tag usage through the fourth quarter and revisit the question in January.

Read that as a merchant. The platform will not tell you which of your apps still depend on the old mechanism. The developers who wrote those apps are complaining in public that they cannot reach their own customers. One developer in the thread said 70 to 75 percent of their users still rely on script tags. I can't verify that number, but even if it is half right, a lot of stores will hit March with live features that quietly stop.

Your analytics will not flag it either. A missing review widget is not a 500 error. It is a page that looks slightly emptier, and the conversion rate drifts down over a few weeks.

Which storefront features usually depend on script tags

Script tags were the easy way for an app to put JavaScript on every page of a store without touching the theme. That makes them common in the parts of a storefront that sit on top of the design rather than inside it. Think review stars on product cards, exit-intent popups, free-shipping progress bars, countdown timers, currency or language switchers, cookie banners, size guides, chat bubbles and cross-sell drawers.

These are also the features that carry conversion weight. Reviews are the clearest case. Northwestern's Spiegel Research Center report found that a product with five reviews is 270 percent more likely to be purchased than one with none. Treat that number carefully. It comes largely from one retailer's data and is higher for expensive products. The same report says nearly all of the lift arrives within the first 10 reviews. So the loss from a vanished review widget depends on how many reviews you have, how pricey the product is, and whether shoppers can still find the reviews anywhere else on the page.

Few stores will lose anything like 270 percent. The risk is that these features are ones a design review never checks, because they live in the browser, injected after load, not in the theme code.

A related shutdown has already happened at checkout. A vendor write-up from Aimerce reports that Additional Scripts on the Thank You and Order Status pages stopped for non-Plus stores after August 26, 2026, and that conversion pixels for Meta, Google or TikTok stop firing with no error in the admin. It is a vendor blog selling its own fix, so weigh its advice accordingly. Its audit trick is still good. Compare conversions reported in each ad platform against orders in Shopify, and a growing gap means lost tracking.

The replacement is not automatic

Shopify's documented replacement is an app embed block, shipped inside a theme app extension. For tracking only, the replacement is a web pixel. The two behave very differently for you.

A web pixel needs no action from the merchant. You approve permissions when you install or update the app, and it runs. An app embed block is the opposite. In Shopify's words, "app embed blocks are inactive until an app user turns them on." The developer is supposed to send you to the theme editor with the embed switched on, and confirm it is active on the published theme before deleting the old script tag. Otherwise, as the docs put it, "you leave a gap where neither works."

That creates three failure modes you can plan for.

First, the app updated, but nobody told you to flip the toggle. The developer thread says there is currently no programmatic way to enable an embed for existing merchants, so some developers are relying on emails that merchants don't read. Second, embeds are per theme. If you duplicate your theme for a holiday redesign and publish the copy, the embed has to be turned on again. Third, both versions can run at once. Shopify warns that doing so loads the script twice, which can double-count analytics or show a popup twice.

Stores on legacy themes have a further problem. App blocks need JSON templates, though Shopify says embed blocks work on vintage and Online Store 2.0 themes. Developers in the thread asked what happens to merchants who cannot move, and the thread shows no Shopify commitment to an exception. If your theme is more than a few years old and heavily edited, this is the most likely place for surprises.

A one-afternoon audit before the deadline

You do not need a developer to find out how exposed you are. You need a list of apps, a browser and a few hours.

Start in Shopify admin and list every installed app. For each one, write down what it shows shoppers and where. If an app has no visible storefront element, such as a pure back-office tool, cross it out. What remains is your risk list.

Next, open your live store in a private window with the browser's developer tools on the Network tab, and load the home page, a collection page, a product page and the cart. Filter requests to JavaScript. Files served from cdn.shopify.com with an app's name in the path, or from the app vendor's own domain, tell you which apps inject code. Write down the app, the page and the visible feature it controls. This gives you a map from feature to app, which is the document most stores don't have.

Then contact each vendor on the risk list with one precise question. Ask whether the app still uses script tags on your store, and if so, what you need to do and by when. Ask them to confirm in writing that the app has moved to an app embed block or a web pixel. If the answer is vague, treat it as a no.

Finally, work out your fallback. For any feature that drives revenue, such as reviews, upsells or shipping thresholds, decide now what you will do if it disappears in March. That might be a different app that already uses theme extensions, or a native theme section. Shopify also describes a route for admin-created custom apps. Move the script into the theme itself, ideally as a custom Liquid block. The catch is that the script stays after the app is removed, and Shopify notes that editing theme code takes the theme out of automatic upgrades.

What I would do this month

Treat the deadline as a release date for your own store. Work backward from March 1 and leave a buffer, because app vendors will be answering the same question from thousands of merchants at the same time in February.

In November, build the feature-to-app map and email vendors. In December, avoid redesigns and new theme publishes if you can, since each one resets the embed activation question. In January, check the Shopify developer changelog, because Shopify has said it will revisit the situation then and may announce targeted notices. Between mid-January and mid-February, have each vendor activate its embed on a staging copy of the theme, then do the same on the published theme, with a before-and-after screenshot of every template that carries an app feature.

After each change, compare the page with your screenshots. Watch for a doubled popup, a missing badge, a layout shift where a widget used to load, and a conversion pixel that fires twice. Compare ad platform conversions against Shopify orders for a week afterward.

One habit for later. At every app demo, ask whether it runs through a theme app extension and web pixel, or still injects a script tag.

Originally published on kireev.us.

AI agent visual regression testing for design systems

Artem Kireev — Fri, 30 Oct 2026 16:12:35 +0000

AI coding agents now edit interface code all day, and they do it without looking at the screen. The code compiles, the tests pass, and a button three screens away has quietly changed color. Trail of Bits found the same pattern in security patches this month, and the UI version is easier to miss because nobody greps for "slightly off".

This article argues that visual regression testing is the review step product teams are skipping, and that the hard part is not the tooling but deciding what counts as a change. It is written for designers, product managers and founders who let agents touch a shared component library and want a setup they can put in place this week.

Agents fix the thing you asked and change the things you did not

On October 21, Trail of Bits published a review of its own merged patches. Across more than 130 merged pull requests, it confirmed 21 regressions: 16 functional, 1 performance and 4 in build or tests. A first agent pass flagged 58 candidates, and only 21 were real, about 36% precision. The lesson is not that agents write bad patches. It is that a change is judged against the bug it fixes, and nobody checks everything the diff touched.

That is a security team looking at server code. Interfaces have it worse. A functional test asks whether the checkout button submits the form. It does not ask whether the button is still the same green as the one on the pricing page.

Augment Code's guide to testing AI-generated UIs describes the failure plainly. Agent edits can pass linting, type checks and functional tests while still shipping layout shifts, broken CSS and styling drift. It is a vendor guide, so treat its benchmark numbers with care. The mechanism it describes matches what any team with a shared component library has seen.

My view is that this is a review problem, not a model problem. Better models will drift less, but they will also be given bigger tasks. The check has to live outside the author.

Why code review does not catch visual drift

A reviewer reading a diff sees --primary: #1a7f4b changing to --primary: #1b8250. Both are valid. Neither looks wrong. The reviewer cannot see that the token feeds hover states, disabled states, focus rings and the dark theme.

In shadcn/ui style setups the component source is copied into your repo, which means an agent can edit it directly. Augment notes that a change to --primary reaches every component that references it. One token, many consumers, one screen checked by the person who wrote the prompt.

Agents also vary run to run. Similar prompts can produce different spacing, different token choices and different responsive behavior. A spacing value of 14px is numerically fine and still off a 4px grid. A linter will not object. A person scanning a pull request will not either.

The fix is to stop asking humans to compute rendered output in their heads. Render it, compare it with the last approved render, and let the human judge only the differences.

What a visual test actually checks

The mechanism is simple. Playwright's snapshot testing saves a reference screenshot on the first run with expect(page).toHaveScreenshot(), then compares every later run against it. Storybook's visual testing does the same per story. Once enabled, every story becomes a test, and the first run creates the baselines.

Two details decide whether this helps or annoys your team.

First, thresholds. Playwright's per-pixel threshold defaults to 0.2 on a 0 to 1 scale, and maxDiffPixels is unset by default. That default will pass a subtle color shift on a large button. Tighten it for token-level components, loosen it for pages with photography.

Second, environment. Playwright's docs warn that rendering varies with the operating system, browser, hardware and headless mode, and baselines are named per browser and platform. A baseline made on a designer's laptop will fail in CI. Generate baselines in the same container image that runs the checks, or every run opens with false alarms and people learn to ignore the check.

Pixel diffing has a blind spot too. It tells you something changed, not whether the change was intended. That judgment is the part worth spending human attention on.

Test the components, then the flows that earn money

Teams usually start by screenshotting whole pages. I would start one level down. Shared components carry the blast radius, so cover each state of each one: default, hover, focus, disabled, loading, error. Storybook makes that cheap because the states already exist as stories.

Then add a short list of flows where a visual break costs revenue. For a SaaS product that is sign-up, the first-run screen and billing. For a store it is the product page, the cart and checkout. Five or six screens at two viewport widths is a better start than sixty screens nobody maintains.

Keep the noise out. Disable animations, wait for fonts, and mask regions that change by design, such as timestamps, rotating banners and ads. Playwright lets you move the mouse off interactive elements so hover styles do not leak into the shot, and a custom stylesheet can hide volatile content. Each of these removes a reason for someone to mute the check.

Where agents come in, give them the screenshots too. An agent that can see the rendered result can catch its own mistake before a human does. It should still not be the one to approve a baseline.

The approval rule that makes it work

The weak point of every visual test suite is the update command. In Playwright it is npx playwright test --update-snapshots. In Storybook you accept the change in the panel. Both are one click, and an agent that sees a red check will happily click it for you.

That turns the test into a rubber stamp. So write one rule down: an agent may propose a baseline change, a person who owns that component approves it. Make the UI check a required status on pull requests, which Storybook's Chromatic integration supports, and put the component owner on review for any file that changes a baseline.

The second half of the rule is about intent. Augment calls the failure "intent drift", where a change is visually clean but diverges from what the component is for. No screenshot diff finds that. Only someone who knows why the button is that size can, which is where a designer earns the review.

There is a catch worth admitting. Visual tests also report your own design lag. If the design file was not updated after a scope change, the test will flag the code as a regression when the design is stale. Fix the source of truth first, then the baseline.

A setup you can start on Monday

You do not need a platform purchase to begin. Here is a sequence that fits in a week.

Day one, list the ten components and five flows where a visual mistake would cost the most. Day two, write screenshot tests for them in Playwright, or turn on Storybook visual tests if you already keep stories. Day three, pin the browser and container image, and generate baselines there.

Day four, set thresholds. Start strict on tokens and buttons, looser on image-heavy pages, and review the first week of failures to see which are noise. Day five, make the check required, name an owner per component, and write the rule that agents propose baselines while people accept them.

After that, add one habit. When an agent task touches a token file or a shared component, the pull request description must list the screens it expects to change. If the diff shows others, the reviewer has a concrete question to ask.

None of this makes agents safer on its own. It makes their side effects visible, which is what Trail of Bits was trying to do for code. The cost is a few hours of setup and a review step that sometimes says no.

Originally published on kireev.us.

AI agent approval UX: show the side effects, not a yes/no

Artem Kireev — Tue, 27 Oct 2026 15:46:41 +0000

Anthropic reported this year that users accept 93% of the permission prompts its coding agent shows them. That is not a sign of trust. It is a sign that the prompt has stopped carrying information, and every SaaS team shipping an "Approve" button for an agent is building the same prompt.

This article argues that approval screens fail because they ask about the command instead of showing the consequence. It covers what to put on the screen, how to tier actions so people see fewer prompts, and how to tell when your own gate has turned into a rubber stamp. It is for product designers and PMs adding agents to a SaaS product.

A 93% approval rate is a design failure

The default way to make an agent safe is to ask the human first. Before it sends the email, edits the record or runs the command, a dialog appears with two buttons. It feels responsible, and it is easy to ship.

The data on how that goes is not kind. In its write-up of Claude Code's auto mode, Anthropic says that "in practice users accept 93%" of manual prompts, and that this leads to "approval fatigue, where people stop paying close attention to what they're approving." Anthropic's own answer was to automate most of the decisions with a classifier. That is a reasonable move for a developer tool. A customer-facing SaaS product has a different problem. You cannot hand the decision to a classifier and tell a finance manager it was fine.

A write-up on permission prompts as a UX bug makes the useful point that "the user approved it" should not end a post-mortem. The question is whether the interface made an informed approval possible. If the dialog shows UPDATE invoices SET status = 'void' WHERE ..., the answer is no, and the approval means nothing.

A high approval rate is the symptom, not the goal. Teams often read it as proof the agent is accurate. It can equally mean nobody is reading.

Agents change more than you asked

The bigger reason a bare confirmation fails is that the thing being confirmed is rarely the whole change. Trail of Bits looked at every patch its engineers wrote with AI agents and that maintainers merged upstream. In their review, over 130 merged pull requests contained 21 confirmed regressions, 16 of them functional. The agent fixed the reported bug and broke something nearby.

That is code, but the pattern carries over to product work. Ask an agent to "clean up duplicate contacts" and the stated action is a merge. The side effects are a changed owner on 40 deals, a deleted email thread and a broken automation that keyed on the old ID. The person approving sees "Merge 212 contacts?" and says yes.

Trail of Bits found that the fix on their side was evidence, not more confirmation. Their regression tool traces the change to real interfaces and demands a live reproduction for every claim. If a step cannot produce proof, the result is marked inconclusive. Strip the security language out and the design principle is simple: do not ask the human to trust a summary.

For approval UX, that means the screen should show the second-order effects the agent already knows about or can compute. Which records change, which other objects reference them, what stops working, and what cannot be undone.

Put the consequence on the screen

Four elements turn a confirmation into a review. None needs new technology. They need the backend to expose a dry run, which is the real cost of doing this well.

First, a diff. Show before and after for the fields that change, the way a code review does. "Status changes from Active to Void" on 212 rows is readable. Group identical changes and let the reviewer expand one example.

Second, the blast radius. Say how many records, which customers, which downstream systems. A count that stands out ("1 of 212 is a customer with an open dispute") does more than a paragraph of explanation.

Third, reversibility, stated plainly. "Undo available for 30 days" and "Cannot be undone" should look different, because they call for different levels of attention. The Amazon Bedrock confirmation pattern that AWS documents returns only the function name and parameter values for a yes/no. AWS's own guidance lists "the proposed action only, without the reasoning chain, data sources, or potential consequences" as an anti-pattern that turns review into a formality. Read that as a warning about default framework behavior. The stock approval step is the thing to design past.

Fourth, the reason. One line on why the agent chose this action, linked to the evidence. If the reason is weak, the reviewer should be able to see that it is weak.

I would also drop the bright green Approve button. Make the primary action "Apply 212 changes" so the button repeats the scope.

Fewer prompts, tiered by reversibility

Richer screens only work if people see fewer of them. The tianpan write-up estimates, as the author's own rough figure rather than a measured one, that people give roughly 10 to 20 genuinely attentive approvals per session before they start pattern-matching. Treat attention as a budget and spend it where it matters.

AWS gives a workable baseline: read-only operations proceed on their own, low-risk writes need one reviewer, and financial transactions, data deletion and external communications need stricter approval. A product designer can sort any agent action into three bins.

Reads and drafts need no gate. Show the result and let the user edit it. Reversible changes run, then appear in an activity feed with an undo button. This is review after the fact, and for most actions it beats a blocking dialog because the user sees the real result rather than a prediction. Irreversible or external actions block. Think sending to customers, refunding money, deleting data and changing permissions. These get the full diff, blast radius and reason.

Two cautions. Trust should not leak across categories. A user who approved ten report exports has told you nothing about whether they want the agent to issue refunds. And AWS warns against wildcard grants such as "always allow this type of action", which remove oversight for a whole class. Bound any "don't ask again" to a specific object, parameter shape or time window, and let people revoke it from one screen.

The classifier that sorts actions into tiers should be deterministic rules, not another model reading the same untrusted input. AWS makes this point directly, because adversarial content can talk a model into labeling a request low-risk.

Measure whether your gate is real

You can tell if an approval step works by instrumenting it. Most teams log approvals for audit and never look at them as a design signal.

Track three numbers per action type. The approve rate. A flat 100% means the gate is decorative or the action should not be gated. Time to decision. The tianpan article suggests alerting on approvals faster than about 800 milliseconds, which is a heuristic, but a click that quick cannot include reading. And the trend within a session, because latency that falls as the session goes on is fatigue showing up in your data.

Then run one cheap test this week. Take your three most common agent actions and plant a deliberately wrong proposal in a staging build, such as a refund to the wrong customer or a merge that includes a record that should not be touched. See how many reviewers catch it, with the current dialog and with a diff version. Anthropic's classifier testing found that humans caught a disguised dangerous command only 13.6% of the time, a figure reported in a DevOps.com summary of the auto mode work that I could not confirm against Anthropic's own post, so treat it as indicative. Your own catch rate is the number that counts.

A checklist for the sprint. List every action the agent can take. Mark each as read, reversible or irreversible. Delete the prompt on the first two and replace it with an activity feed and undo. For the third, build the dry run and render the diff, blast radius, reversibility and reason. Ship the instrumentation with it. If catch rates do not improve, the screen is not doing its job yet.

The same logic applies when the agent is not yours. Stores are already deciding how to handle outside shoppers' bots, as I covered in AI shopping agents at checkout. Whether the agent is inside your product or knocking at its door, the question is the same. What will a human be able to see before it acts?

Originally published on kireev.us.

Shopify conversion rate benchmarks: why the numbers disagree

Artem Kireev — Fri, 23 Oct 2026 15:11:53 +0000

Search for the average Shopify conversion rate and you get 1.4%, 1.74%, 2.5% or "2 to 3%", all published in 2026, all stated with confidence. A store at 1.6% is either below average, above average or right in the middle, depending on which page you read.

This article explains where the gaps come from, why a benchmark is a weaker tool than it looks, and how to build a comparison you can trust with the data you already have. It is for store owners and e-commerce leads who are about to set a conversion target.

Four benchmarks, four answers

Start with what the sources actually say. Shogun's benchmark page covers 803 active Shopify stores from January through September 2026 and reports a median conversion rate of 1.74% and a mean of 2.54%. Littledata, which pulls data from 433 stores that send events to Google Analytics 4, reports a median of 1.4% and says 2.7% puts a store in the top 20%. Cartylabs, a Shopify app vendor, gives a median band of 2.0 to 3.0% and calls 3.5 to 5.0% the top quartile.

That is a spread of more than a full point on the median, and the top-tier thresholds differ even more. Shogun's 75th percentile is 3.32%. Cartylabs' top quartile starts at 3.5% and runs to 5.0%.

None of these numbers is fake. They answer different questions. The first mistake teams make is treating "Shopify conversion benchmark" as one number with a few sources that disagree by rounding. It is several numbers measuring different populations with different rules.

Why the same metric gives different answers

Three things move a benchmark before your store enters the picture.

The first is who is in the sample. Littledata only counts stores that use its app and feed GA4. Shogun only counts stores in its own customer base, and it drops stores below 0.1% or above 25% conversion as "non-representative." Both filters are sensible, and both shape the result. A cohort of stores that pay for analytics tooling is probably more mature than the Shopify average. A cohort that excludes the 0.1% tail looks healthier than one that keeps it.

The second is the measurement. Littledata defines conversion as the share of sessions with at least one purchase, counted in GA4. Your Shopify admin and your GA4 property may not count sessions identically, and I could not find either tool documenting that they do. A store comparing one against a benchmark built on the other should check before trusting the gap. Cartylabs notes a trap on the checkout side: Shop Pay accelerated checkout can skip the checkout_started event, which makes checkout completion look higher than it is.

The third is who wrote the page. Cartylabs does not disclose its sample sizes per segment, the time window or how it computed percentiles, and it sells a checkout app. That doesn't make its numbers wrong. It means you should weight them below a benchmark that states its sample, period and exclusions.

I'd rank these by transparency: a page that tells you the store count, the date range and the filters beats a page that reports a band with no method. By that rule, the Shogun and Littledata figures are the ones to use as a rough anchor, and the 2 to 3% "median" should be read as marketing.

The mean is the wrong number to chase

Look at Shogun's own figures again. The median is 1.74% and the mean is 2.54%. The page describes the mean as "pulled by top performers." Its middle 50% of stores run from 0.83% to 3.32%, so the typical store sits in a band that is wider than the median itself.

That has a practical consequence. If you set a goal of "reach the average," you may be aiming at a number that most stores in the sample never hit. Half the stores are under 1.74%. A handful of very good stores drag the mean up.

Industry matters even more. Shogun's industry medians range from 3.22% for health and wellness and 3.15% for beauty down to 1.39% for consumer electronics and 1.20% for sports and outdoors. A sporting goods store at 1.5% is above its category median. The same 1.5% in beauty is less than half of it.

Price does the same thing. Cartylabs' checkout-completion bands fall from a 50 to 62% median for orders under $30 to 15 to 28% above $750. Those bands come with the caveats above, but the direction is plausible and matches what you would expect: the more a shopper spends, the more they hesitate. A store selling $900 items that benchmarks itself against the whole platform is measuring its catalogue, not its checkout.

Conversion rate hides where the leak is

Site-wide conversion divides orders by all sessions. That includes visitors who arrived from a podcast mention, bounced from a blog post or never meant to buy. Many of those sessions cannot convert.

Checkout completion is a cleaner signal because it only counts shoppers who already started. Even here the sources diverge. Littledata's median is 51%, with 67% at the top 20%. Cartylabs puts the median at 38 to 48%. Both describe the same metric in plain words, and they are about ten points apart.

The better instrument is the reason shoppers leave. Baymard's compilation of 50 studies puts average cart abandonment at 70.22%. That figure alone tells you little, because Baymard points out that many abandonments are just browsing, and it attributes 42% of US abandonments to "I was just browsing / not ready to buy." It then removes that group and lists what is left: 40% cite extra costs, 20% slow delivery, 19% a lack of trust with card details, 18% a forced account, 17% a long checkout and 17% site errors.

Those are fixable. A benchmark tells you that you are slow. These reasons tell you what to change. If a shopper abandons because shipping appears at the last step, a better button will not help.

How to build a benchmark you can use

Skip the platform-wide average. Build three comparisons instead.

Compare against yourself first. Pull 12 months of data and split by device, traffic source and new versus returning visitors. Cartylabs suggests treating weekly swings under three percentage points in checkout completion as noise, and I agree with the spirit of that: a store with a few hundred checkouts a week sees swings that size from chance alone. Compare the same quarter year over year, because Q4 and Q1 behave differently.

Then pick one outside benchmark that publishes its sample, period and filters, and match it on category and average order value. Use it as a range, not a target. If you sit inside the middle 50% for your category, you have no emergency. If you sit below the 25th percentile, you have a reason to dig in.

Finally, check your own tracking before you trust any comparison. Confirm that checkout_started fires on Shop Pay and on mobile wallets. Confirm that B2B and draft orders are excluded from the numerator. Make sure the session definition in your admin matches the one in the benchmark you borrowed. A mismatch here can shift your result by more than most CRO tests will.

Once those three are in place, spend your effort on the abandonment reasons. Start with cost visibility: show shipping and tax estimates in the cart, before the shopper commits. Offer guest checkout. List the payment methods that Baymard flags as missing, and put the returns policy where people can see it. Each of these maps to a reason with a real share behind it. A benchmark can say you are behind. Only the reasons can tell you what to fix on Monday.

Originally published on kireev.us.

AI shopping agents at checkout: allow, block or verify?

Artem Kireev — Wed, 21 Oct 2026 19:24:18 +0000

Stores are starting to shut their doors on AI shopping agents. Amazon reportedly removed Meta's Muse assistant in September, Delta keeps outside agents from booking flights, and the big US banks keep bots out of customer accounts. Most of these blocks are blunt, and many catch customers who are simply using an assistant.

This article argues that "block or allow" is the wrong question for an e-commerce team. The useful one is which steps of your checkout an agent may touch, and how you would know it is the agent your customer sent. It is for store owners, product managers and designers who have to decide before an agent shows up in their analytics.

The blocks are arriving faster than the rules

In mid-October, Android Headlines summarized the latest wave: Amazon pulling Meta's Muse assistant off its shopping platform, Delta stopping external agents from booking flights, and JPMorgan Chase, Bank of America and Citigroup keeping automated bots out of accounts. The same piece notes there is no clear industry standard for telling an authorized user bot from a malicious scraper.

That missing standard explains most of the mess. Anti-bot systems built to stop scrapers see a headless browser clicking through a cart and treat it the same way. The article points out that Cloudflare-style challenges often throw CAPTCHAs that stop an agent mid-checkout. The shopper never learns why the assistant failed. They just see it fail.

Platforms already tried to draw a line once. In 2025, Modern Retail reported that Shopify added language to the default robots.txt on merchant storefronts saying that "any end-to-end flow that completes payment without a final review step is not permitted." Shopify's Ilya Grigorik said the change added or removed no rules and mainly pointed integrators to Checkout Kit. Analyst Juozas Kaziukėnas read it as a note for developers, and merchants can override the file.

Put those together and you get a policy vacuum. The platform wrote a sentence, the CDN is guessing, and the merchant, who owns the customer relationship, usually hasn't decided anything.

Blocking all agents has a real cost

The reflex is to keep everything out. It feels safe, and for banks it often is. For a store it is a bet that agent-assisted shoppers are worth less than the risk.

Consider what the customer sees. Baymard's cart abandonment benchmark puts the average at 70.22% across 50 studies, and among reasons shoppers give, 17% say checkout was too long or complicated and 17% cite site errors or crashes. A shopper who hands the chore to an assistant is trying to skip exactly that friction. If your bot wall breaks the assistant halfway, the failure lands on your brand, not on the assistant.

Agents also compare. If one store lets them read prices, stock and shipping cost, and the next store throws a challenge page, the first store is the one that gets considered. I think this will matter first in repeat-purchase categories where the shopper already knows what they want and only wants it ordered.

None of this means "open the gates". Fraud, price scraping and inventory hoarding are real. It means the blanket block deserves the same scrutiny as any other decision that removes a path to purchase.

Three tiers instead of a yes/no switch

A better frame splits the journey into three tiers and sets a policy for each.

Read is the first tier: product pages, prices, availability, shipping and return terms. These are public already. Make them machine-readable with clean structured data and plain-text policies, and rate-limit rather than block. An agent that can read your return policy accurately is better for you than one that guesses.

Act is the second tier: adding to cart, applying a discount, choosing a shipping option. Here you want to know who is asking. Allow it for identified agents, and keep it behind your normal bot defenses for anonymous ones.

Pay is the third. This is where Shopify's wording about a "final review step" makes sense, and I would copy the principle. Let the agent assemble the order, but have the human confirm the total, delivery date and return terms before money moves. That also protects you from the angry-customer case where an agent bought the wrong size.

Tiers give a design team something concrete to work from. Each one maps to screens, states and error messages, rather than to a firewall rule nobody in product ever sees.

Verified identity is the missing piece

Tiering only works if you can tell agents apart. The most credible attempt is Web Bot Auth, where an operator signs each request with a private key and the site verifies it against a public key the operator publishes. The IETF draft describes it as an architecture for identifying automated traffic using HTTP Message Signatures.

Be honest about its status. That draft is an individual submission, now expired and replaced by a newer one, and the IETF has not endorsed it. It also settles who is calling, not what they may do. Whether to allow, throttle or block a verified agent stays your decision, which is precisely why you need the tiers above.

On the transaction side, Stripe describes the Agentic Commerce Protocol, co-developed with OpenAI and released as open source in September 2025, as keeping the business as the merchant of record. Merchants decide what is sold, how it is displayed, how payment is processed, and can accept or decline any order on fraud signals. The relevant point for design is that the protocol assumes you keep control of the decision points. Sources disagree on how much live volume ChatGPT's own checkout has seen, so I would treat adoption claims with caution and the protocol's control model as the useful part.

What to do this week

You don't need a protocol to start. You need an owner and a list.

First, find out what is happening. Pull a week of bot-challenge logs from your CDN and look for challenges served on cart and checkout URLs. If you see them, you are already blocking someone's assistant, whether you meant to or not.

Second, write the three-tier policy on one page: what agents may read, what they may do with a verified identity, and what always needs a human confirmation. Get product, engineering and whoever owns fraud to sign it. Fifteen minutes in a room beats a year of defaults.

Third, fix the read tier. Check that prices, stock, shipping cost and return terms appear in plain HTML and structured data, without a script that must run first. This also helps ordinary search and accessibility, so it pays off even if no agent ever arrives. Baymard's top abandonment reason is extra costs at 40%, so show total cost early for humans and machines alike.

Fourth, design the review step. Decide what the confirmation screen shows when an agent built the cart, and what the shopper sees if the agent is stopped. Write that error copy now. A clear "your assistant couldn't finish, here is your cart" message turns a failure into a handoff.

Then revisit the policy each quarter. The standards will move, and the first version of yours only needs to be deliberate, not final.

Originally published on kireev.us.