DEV Community: Doug silkstone

A Practical Holdout Pattern for Coupon Extension Rules

Doug silkstone — Mon, 05 Oct 2026 16:35:23 +0000

Coupon extensions can change a checkout experience and the way a discount is attributed. A detected extension is not proof that it caused a bad order, though. Before applying a global block, a merchant needs a comparison they can explain.

I use a simple control loop when building Benson: observe the session, assign a stable holdout, measure the same outcomes on both sides, then choose the smallest rule the evidence supports.

1. Start with an event contract

Keep detection, treatment, and outcome data separate. A useful session record can include:

a first-party scoped session identifier;
the detected surface, such as a coupon extension or shopping agent;
the rule state: observe, treatment, or holdout;
code attempts and the response from the storefront;
order, conversion, discount, and order value fields when they exist.

This contract lets a team answer two different questions: “What appeared in the session?” and “What changed after we applied a rule?” Combining those questions into one event makes later analysis hard to audit.

2. Keep detection separate from the decision

An extension can try a code without changing the final order. An AI shopping agent can request several offers without telling you whether the shopper would have paid full price, used a first-party offer, or abandoned checkout. Detection is the beginning of the investigation.

In observe mode, record what a proposed rule would match while leaving the existing experience in place. This is a safe way to find an overly broad selector, a valuable partner, or a code that the merchant intended to keep.

3. Assign a stable holdout

Use a stable first-party identifier so the same kind of visit does not move between groups on every reload. The exact split depends on the store and the experiment, but the assignment must be decided before reading the outcome.

An illustrative assignment looks like this:

bucket = hash(experiment_id + stable_session_id) % 100
rule_state = "treatment" if bucket < 50 else "holdout"

The holdout keeps the current experience. The treatment applies the proposed block, cap, or replacement. Keep the time window, offer conditions, and traffic definition visible beside the result; otherwise a clean-looking comparison can hide a campaign or merchandising change.

4. Compare outcomes side by side

Read the treatment and holdout together. Useful measures include conversion rate, order value, discount cost, commission or attribution cost, and code attempts per session. A treatment group with fewer discounts is not automatically better if it also loses valuable orders.

Avoid turning an illustrative dashboard or a before-and-after estimate into a customer result report. A concurrent holdout is stronger evidence because both groups experience the same period and store conditions. If the sample is small or traffic changes halfway through, label the result as a signal and keep the test running.

5. Put guardrails around the rule

A rule should have a narrow match, a clear rollback, and a visible holdout. Check the rule against the sessions it would affect before enabling it. Watch for changes in traffic mix, first-party promotions, inventory, and campaign terms that could explain an outcome difference.

If conversion stays comparable while discount or attribution cost moves in the desired direction, the merchant has evidence for a controlled rollout. If conversion falls, keep the holdout and investigate the trade-off instead of expanding the block.

6. Make the decision reproducible

Store the experiment name, assignment rule, time window, storefront conditions, and the version of the control in the readout. That record turns “the extension looked expensive” into a decision another operator can inspect and repeat.

This is the gap Benson is built for: showing online stores which coupon extensions and AI shopping agents appear in their sessions and what they cost, then supporting hide, cap, or replacement rules with a holdout behind each change. I am the founder of Benson. This article describes a measurement method, not a customer result report. Learn more at https://trybenson.com.

Spin up a License Key API on new Railway Functions In Less Than Two Minutes!

Doug silkstone — Tue, 11 Feb 2025 23:07:26 +0000

Railway's new Functions feature makes it incredibly easy to deploy single-file TypeScript code without the overhead of managing infrastructure or repositories.

In this article, we'll walk through building a license API using Hono, Redis, and the Bun runtime—all in a single file. No more excuses, it's time to build software!

Note: Railway Functions are perfect for small tasks like handling webhooks, cron jobs, or simple APIs. What makes them even better is how they interact with the rest of the Railway ecosystem.

Customizing Your Prefixes

Before diving in, make sure to edit the prefix values in the code to suit your needs. The default prefixes, such as WTHSEISMIC_, are placeholders and should be replaced with values relevant to your project to ensure uniqueness and avoid conflicts.

Prerequisites

A Railway account.
Basic knowledge of TypeScript.
Familiarity with serverless concepts.

Step 1: Set Up a New Project

Create a New Project:

Head over to your Railway dashboard and create a new project. Choose the Functions service to get started.
Select the Functions Environment:

Railway will automatically set up the environment using the Bun runtime for your function. No need for additional configuration!

Step 2: Provision Redis

Add a Redis Service:

In your Railway project, provision a Redis service. This will be used to store license data.
Share Environment Variables:

After provisioning Redis, click on the Redis service, then head to Variables. Use the "Share Variables" feature to expose the following environment variables to your function:

REDISHOST
REDISPORT
REDISUSER
REDISPASSWORD
(Optional) REDIS_TLS if your Redis requires TLS.

Step 3: Create Your Function

Create a New Function:

In the Railway dashboard, click New Function. This opens the built-in editor which supports TypeScript using Bun.
Set Up Additional Environment Variables:

Add any additional variables your API requires. For example, generate unique values (e.g., using a UUID generator) for:

ADMIN_KEY
HMAC_SECRET
PAYMENT_SECRET
Optionally, set PORT (defaults to 3000 if not provided).

Step 4: Paste and Save the Code

Copy and paste the following complete code into the Source Code tab of your Railway Function. Save your changes using ⌘+S (or Ctrl+S on Windows). Railway will automatically deploy your changes!

// index.ts
import { Hono } from "hono";
import { cors } from "hono/cors";
import { rateLimiter } from "hono-rate-limiter";
import Redis from "ioredis";
import crypto from "node:crypto";

// Custom prefix configuration
const LICENSE_PREFIX = "WTHSEISMIC_";
const LICENSE_SET = "WTHSEISMIC_LICENSES";
const WEBHOOK_SET = "WTHSEISMIC_WEBHOOKS";

// Updated LicenseTier type to include "unlimited"
type LicenseTier = "basic" | "pro" | "enterprise" | "unlimited";

... (full code here)

Step 5: Test Your Deployment

Once deployed, Railway provides a URL for your function. Use the following curl commands to test each endpoint:

5.1: Test the Root Endpoint

Returns the current timestamp.

curl -X GET https://<your-function-url>/

5.2: Create a New License

Creates a new license key for a user.

curl -X POST https://<your-function-url>/api/licenses \
  -H "Content-Type: application/json" \
  -d '{"email": "user@example.com", "tier": "pro", "durationDays": 90}'

5.3: Validate a License

Validates an existing license key.

curl -X GET https://<your-function-url>/api/licenses/WTHSEISMIC_xxx

5.4: Handle Payment Webhook

Simulates a payment webhook to extend a license's duration.

curl -X POST https://<your-function-url>/webhooks/payment \
  -H "Content-Type: text/plain" \
  -H "X-Payment-Signature: <signature>" \
  -d '{"licenseKey": "WTHSEISMIC_xxx", "durationDays": 30}'

5.5: Extend License via Admin Endpoint

Extends the license duration using an admin key.

curl -X POST https://<your-function-url>/admin/licenses/WTHSEISMIC_xxx/extend \
  -H "Content-Type: application/json" \
  -H "X-Admin-Key: your-secret-key" \
  -d '{"durationDays": 30}'

Use these examples to verify that your function is working as expected.

So there you have it, in less than two minutes, we've put together a license key server on Railway using Redis, Hono and Bun - Demonstrating how headache-free the whole process is, and how (not so) small snippets of code can live within your Railway ecoystem within minutes.

Great feature, great platform.