DEV Community: David Sert

Testing AI Models with Feature Flags: LLM Prompt Optimization

David Sert — Sat, 18 Jul 2026 15:32:49 +0000

Modern AI applications face a unique challenge: how do you A/B test something as dynamic and complex as a large language model? Whether you're choosing between Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro, optimizing prompt engineering, or balancing cost and quality across model tiers, traditional experimentation approaches fall short. Feature flags provide an elegant solution for testing AI configurations without deploying new code.

AI teams struggle with critical decisions: Which frontier model performs better for customer support—Claude Opus 4.8's coding expertise or GPT-5.5's reasoning prowess? Should you pay $30 per million tokens for Opus 4.8 or use Claude Haiku 4.5 at $6 for simpler queries? Can prompt caching reduce costs by 90%? Without proper experimentation, these decisions are made on gut feeling rather than data, potentially costing thousands in wasted API spend.

This guide demonstrates how to use Optimizely Feature Experimentation to A/B test the latest AI models, prompts, and cost optimization strategies. You'll learn how to set up feature flag variations that route traffic to different AI configurations, leverage cost-saving features like prompt caching and batch APIs, measure quality metrics, and make data-driven decisions that can reduce AI costs by 50-67% while maintaining quality.

The 2026 AI Model Landscape

The AI model landscape has matured significantly with three dominant providers releasing flagship models in late 2025. Each model excels at different tasks, making intelligent model selection critical for both quality and cost optimization.

OpenAI GPT-5.5

GPT-5.5 is OpenAI's most capable model series for professional knowledge work. It comes in three variants: Instant (optimized for speed), Thinking (extended reasoning), and Pro (maximum capability).

Key capabilities include a 400,000-token context window, 128,000-token output capacity, 65% fewer hallucinations compared to GPT-4, and 100% accuracy on the AIME 2025 mathematics olympiad. The model is priced at $5 per million input tokens and $30 per million output tokens, with a 90% discount on cached inputs.

GPT-5.5 is best suited for complex reasoning tasks, mathematics, and general-purpose applications where cost efficiency matters.

Anthropic Claude Opus 4.8 and Sonnet 4.6

Claude Opus 4.8 is described as "the best model in the world for coding, agents, and computer use." It achieves 80.9% on SWE-bench Verified (real GitHub issues), making it the leader for autonomous coding tasks. The model delivers flagship performance at 67% lower cost than the previous Opus 4.

Pricing for Opus 4.8 is $5 per million input tokens and $25 per million output tokens. With prompt caching, cache reads cost only $0.50 per million tokens, representing a 90% savings.

Claude Sonnet 4.6 offers the best balance of intelligence, speed, and cost for most use cases at $3 per million input tokens and $15 per million output tokens. Anthropic recommends starting with Sonnet 4.6 for general-purpose applications.

Claude Haiku 4.5 provides a budget option at $1 per million input tokens and $5 per million output tokens, ideal for high-volume, simple queries.

Google Gemini 3.1 Pro

Gemini 3.1 Pro features the longest context window among the three providers at 1 million tokens, native multimodal understanding for images, video, and audio, and wins user preference rankings for helpfulness.

Pricing for contexts under 200,000 tokens is $2 per million input tokens and $12 per million output tokens. For larger contexts, pricing increases to $4 input and $18 output.

Gemini 3.1 Pro excels at multimodal tasks, longest context requirements, and daily assistance scenarios.

Model Selection Decision Matrix

The following table summarizes when to use each model:

Model	Cost per 1M+1M Tokens	Best For	Key Benchmark
Claude Opus 4.8	$30.00	Coding, agents, autonomous tasks	SWE-bench: 80.9%
GPT-5.5	$35.00	Reasoning, mathematics	AIME 2025: 100%
Gemini 3.1 Pro	$14.00	Multimodal, long context	1M token context
Claude Sonnet 4.6	$18.00	General-purpose, balanced	Production default
Claude Haiku 4.5	$6.00	High-volume, simple queries	Fast, cost-efficient

Intelligent routing can save 40-67% by using the right model tier for each query type.

Why Feature Flags for AI Testing

Feature flags transform how teams deploy and test AI models in production. Rather than deploying new code for every model change, feature flags allow instant configuration changes, gradual rollouts, and rapid rollback if issues arise.

Benefits of the Feature Flag Approach

Testing multiple frontier models simultaneously becomes straightforward. You can run Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro in parallel, routing traffic based on your experimental design.

Experimenting with cost optimization features like prompt caching and batch APIs requires no code changes. Simply update the flag configuration to enable or disable these features for specific user segments.

Instant rollback provides a safety net. If a model underperforms or costs spike unexpectedly, you can switch back to your baseline within seconds, not hours.

Gradual traffic ramps minimize risk. Start with 1% of traffic, validate metrics, then progressively increase to 5%, 25%, 50%, and finally 100%.

Segmentation by query complexity or user tier enables intelligent routing. Send simple queries to Haiku 4.5 and complex queries to Opus 4.8, optimizing both quality and cost.

What You Can Test

Model providers and versions offer the most impactful testing opportunities. Compare Claude Opus 4.8 versus GPT-5.5 versus Gemini 3.1 Pro to determine which performs best for your specific use case.

Model tiers within a provider family also warrant testing. The quality difference between Opus 4.8 ($30), Sonnet 4.6 ($18), and Haiku 4.5 ($6) may or may not justify the cost difference for your workload.

Prompt templates significantly impact output quality. Test system prompts, few-shot examples, and structured output formats.

Model parameters like temperature, top_p, and max_tokens affect response creativity and consistency.

Cost optimization strategies including standard API calls, prompt caching, and batch API usage can reduce costs by 50-90% for eligible workloads.

Intelligent routing rules that classify queries by complexity and route to appropriate model tiers can yield 40-67% cost savings.

Feature Flags as AI Configuration Routers

The core pattern for feature flag-based AI testing routes each request through a decision point that determines the AI configuration:

User Request → Feature Flag Decision → Model Selection → Cost Optimization → LLM API Call → Metrics Tracking

flowchart TB
    A[User Request] --> B[Optimizely Feature Flag]
    B -->|50% Control| C[Claude Opus 4.8]
    B -->|25% Var1| D[GPT-5.5]
    B -->|25% Var2| E[Gemini 3.1 Pro]
    C --> F[Apply Caching]
    D --> F
    E --> F
    F --> G[Track Metrics]
    G --> H[Optimizely Results]

When a user triggers an AI interaction, the feature flag's decide() method returns a variation key such as "opus", "gpt55", or "gemini". The application maps this variation to a complete AI configuration object containing the provider, model, prompt, and parameters. Cost optimization logic checks for cache hits or batch eligibility. Finally, the appropriate LLM API is called and comprehensive metrics are tracked back to Optimizely.

Implementation Guide

This section provides complete code examples for implementing AI model testing with Optimizely feature flags.

Setting Up the Feature Flag

Create a new feature flag in Optimizely named ai-model-selection-2026 with the following variations:

The control variation uses Claude Opus 4.8 with prompt caching as your baseline. Add challenger variations for GPT-5.5, Gemini 3.1 Pro, and an intelligent routing option that dynamically selects models based on query complexity.

Set initial traffic allocation to 40% control, 20% GPT-5.5, 20% Gemini 3.1 Pro, and 20% intelligent routing. This distribution provides sufficient data for each variation while maintaining a substantial control group.

Define a custom event named ai_response_generated with event properties for latency_ms, cost_usd, accuracy_score, cache_hit, and tokens_used.

Configuration Map

Define your AI configurations as a map that the feature flag variations reference:

const AI_CONFIGS = {
  'control': {
    provider: 'anthropic',
    model: 'claude-opus-4-8',
    temperature: 0.7,
    max_tokens: 2048,
    useCache: true,
    systemPrompt: 'You are an expert customer support agent...',
    pricing: { input: 5.00, output: 25.00, cache_read: 0.50 }
  },
  'gpt55': {
    provider: 'openai',
    model: 'gpt-5.5',
    temperature: 0.7,
    max_tokens: 2048,
    systemPrompt: 'You are an expert customer support agent...',
    pricing: { input: 5.00, output: 30.00, cache: 0.50 }
  },
  'gemini': {
    provider: 'google',
    model: 'gemini-3.1-pro',
    temperature: 0.7,
    max_tokens: 2048,
    systemPrompt: 'You are an expert customer support agent...',
    pricing: { input: 2.00, output: 12.00 }
  },
  'haiku': {
    provider: 'anthropic',
    model: 'claude-haiku-4-5',
    temperature: 0.5,
    max_tokens: 1024,
    useCache: true,
    systemPrompt: 'You are a concise customer support agent...',
    pricing: { input: 1.00, output: 5.00, cache_read: 0.10 }
  }
};

Main Request Handler

The main function handles the feature flag decision and routes to the appropriate model:

import Anthropic from '@anthropic-ai/sdk';
import OpenAI from 'openai';
import { GoogleGenerativeAI } from '@google/generative-ai';
import { createInstance } from '@optimizely/optimizely-sdk';

const optimizelyClient = createInstance({
  sdkKey: process.env.OPTIMIZELY_SDK_KEY
});

const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const googleAI = new GoogleGenerativeAI(process.env.GOOGLE_API_KEY);

async function getAIResponse(userId, userMessage, userAttributes = {}) {
  const startTime = Date.now();

  const user = {
    id: userId,
    attributes: {
      user_tier: userAttributes.tier || 'free',
      query_complexity: classifyQueryComplexity(userMessage)
    }
  };

  const decision = optimizelyClient.decide(user, 'ai-model-selection-2026');
  const variation = decision.variationKey;
  const config = AI_CONFIGS[variation];

  let response, tokensUsed, cost, cacheHit = false;

  try {
    if (config.provider === 'anthropic') {
      const result = await callClaude(config, userMessage);
      response = result.response;
      tokensUsed = result.tokensUsed;
      cost = result.cost;
      cacheHit = result.cacheHit;
    } else if (config.provider === 'openai') {
      const result = await callGPT(config, userMessage);
      response = result.response;
      tokensUsed = result.tokensUsed;
      cost = result.cost;
      cacheHit = result.cacheHit;
    } else if (config.provider === 'google') {
      const result = await callGemini(config, userMessage);
      response = result.response;
      tokensUsed = result.tokensUsed;
      cost = result.cost;
    }

    const latencyMs = Date.now() - startTime;

    optimizelyClient.track('ai_response_generated', userId, {
      $opt_event_properties: {
        latency_ms: latencyMs,
        cost_usd: cost,
        tokens_used: tokensUsed,
        cache_hit: cacheHit,
        model_provider: config.provider,
        model_name: config.model,
        variation: variation
      }
    });

    return { response, metadata: { variation, latencyMs, cost, tokensUsed, cacheHit } };

  } catch (error) {
    optimizelyClient.track('ai_error', userId, {
      $opt_event_properties: {
        error_type: error.name,
        model_provider: config.provider,
        variation: variation
      }
    });
    throw error;
  }
}

Claude API Call with Prompt Caching

The Claude implementation enables prompt caching for significant cost savings:

async function callClaude(config, userMessage) {
  const requestParams = {
    model: config.model,
    max_tokens: config.max_tokens,
    temperature: config.temperature,
    messages: [{ role: 'user', content: userMessage }]
  };

  if (config.useCache) {
    requestParams.system = [
      {
        type: 'text',
        text: config.systemPrompt,
        cache_control: { type: 'ephemeral' }
      }
    ];
  } else {
    requestParams.system = config.systemPrompt;
  }

  const result = await anthropic.messages.create(requestParams);

  const inputTokens = result.usage.input_tokens;
  const cachedTokens = result.usage.cache_read_input_tokens || 0;
  const outputTokens = result.usage.output_tokens;
  const cacheHit = cachedTokens > 0;

  const inputCost = (inputTokens - cachedTokens) * (config.pricing.input / 1_000_000);
  const cacheCost = cachedTokens * (config.pricing.cache_read / 1_000_000);
  const outputCost = outputTokens * (config.pricing.output / 1_000_000);
  const totalCost = inputCost + cacheCost + outputCost;

  return {
    response: result.content[0].text,
    tokensUsed: inputTokens + outputTokens,
    cost: totalCost,
    cacheHit: cacheHit
  };
}

GPT-5.5 API Call

The GPT-5.5 implementation with automatic caching detection:

async function callGPT(config, userMessage) {
  const result = await openai.chat.completions.create({
    model: config.model,
    max_tokens: config.max_tokens,
    temperature: config.temperature,
    messages: [
      { role: 'system', content: config.systemPrompt },
      { role: 'user', content: userMessage }
    ]
  });

  const inputTokens = result.usage.prompt_tokens;
  const outputTokens = result.usage.completion_tokens;
  const cachedTokens = result.usage.prompt_tokens_details?.cached_tokens || 0;
  const cacheHit = cachedTokens > 0;

  const uncachedInput = inputTokens - cachedTokens;
  const inputCost = uncachedInput * (config.pricing.input / 1_000_000);
  const cacheCost = cachedTokens * (config.pricing.cache / 1_000_000);
  const outputCost = outputTokens * (config.pricing.output / 1_000_000);
  const totalCost = inputCost + cacheCost + outputCost;

  return {
    response: result.choices[0].message.content,
    tokensUsed: inputTokens + outputTokens,
    cost: totalCost,
    cacheHit: cacheHit
  };
}

Query Complexity Classification

Intelligent routing requires classifying queries by complexity:

function classifyQueryComplexity(userMessage) {
  if (userMessage.length < 50) return 'simple';
  if (userMessage.length > 200) return 'complex';

  const simpleKeywords = [
    'track order', 'shipping status', 'return policy',
    'hours', 'location', 'price'
  ];
  const complexKeywords = [
    'refund', 'damaged', 'defective', 'unauthorized',
    'fraud', 'complaint', 'manager'
  ];

  const lower = userMessage.toLowerCase();

  if (complexKeywords.some(kw => lower.includes(kw))) return 'complex';
  if (simpleKeywords.some(kw => lower.includes(kw))) return 'simple';

  return 'medium';
}

Testing Strategies

Several testing strategies address different optimization goals.

Strategy 1: Model Provider Comparison

Test the hypothesis that Claude Opus 4.8 matches or beats GPT-5.5 on accuracy at a lower blended cost ($30 versus $35 per 1M+1M tokens).

Configure a 50/50 split between Claude Opus 4.8 with prompt caching and GPT-5.5 with prompt caching. Run the test for two weeks with all customer support queries.

Measure accuracy via thumbs up/down rate as the primary metric. Track cost per query, latency P95, and cache hit rate as secondary metrics.

If Claude accuracy matches or exceeds GPT-5.5, keep Claude as the default—it delivers equal or better quality at a lower blended cost ($30 versus $35 per 1M+1M tokens).

Strategy 2: Cost Optimization with Prompt Caching

Test whether enabling prompt caching reduces costs by 40-50% without degrading quality.

Split traffic 50/50 between Claude Opus 4.8 standard and Claude Opus 4.8 with prompt caching enabled.

The primary metric is cost per query, with an expected 40-50% reduction. Accuracy must be maintained at baseline levels to confirm caching does not impact quality.

Expected savings with a 50% cache hit rate: cost drops from $0.015 to $0.008 per query. For 100,000 monthly queries, this represents $700 in monthly savings.

Strategy 3: Intelligent Model Routing

Test the hypothesis that routing simple queries to Haiku 4.5 reduces costs by 70% while maintaining acceptable quality.

In the control group, use Claude Opus 4.8 for all queries. In the variation group, route simple queries (estimated 70% of traffic) to Haiku 4.5 and complex queries (30%) to Opus 4.8.

Expected cost comparison per 100,000 queries:

Approach	Simple Queries (70K)	Complex Queries (30K)	Total Cost
All Opus 4.8	$1,050	$450	$1,500
Intelligent Routing	$210 (Haiku)	$450 (Opus)	$660

The intelligent routing approach yields 56% cost savings. Validate that Haiku accuracy on simple queries reaches at least 90% of Opus accuracy.

Strategy 4: Batch API for Non-Urgent Queries

For workloads that can tolerate 1-24 hour latency, such as email support responses, test batch API pricing.

Split email support queries 50/50 between real-time GPT-5.5 at $35.00 per million input and output tokens and batch API GPT-5.5 at $17.50 per million tokens.

The 50% cost savings from batch processing applies only to workloads where delayed responses are acceptable.

Measuring Success

Define success metrics before launching any experiment.

Example OKR: Cost Optimization

Objective: Reduce AI support costs while maintaining quality.

Key Result 1: Reduce cost per query by 40%, from $0.015 to $0.009.

Key Result 2: Maintain accuracy score at or above 85% (thumbs up rate).

Key Result 3: Keep P95 latency under 3 seconds.

Results Analysis Framework

After gathering sufficient data, analyze results across multiple dimensions.

The winner on accuracy may differ from the winner on cost. A model that costs twice as much but delivers only marginally better accuracy may not be the right choice for cost-sensitive applications.

Consider segment performance. A variation may win overall but underperform for specific user segments or query types.

Calculate quality per dollar: accuracy divided by cost. This composite metric helps identify the best value proposition.

Decision Matrix

Scenario	Recommended Action
Variation wins on cost AND accuracy	Roll out to 100%
Variation wins on cost, loses accuracy by less than 5%	Business decision: evaluate cost vs quality tradeoff
Variation wins on cost, loses accuracy by more than 10%	Reject variation
No statistically significant difference	Extend test or decide based on secondary metrics

Real-World Example: E-commerce Customer Support

ShopCo, an e-commerce company processing 1 million support queries per month, faced unsustainable AI costs of $15,000 monthly using Claude Opus 4.8 for all queries.

Baseline Measurement

During the first week, they measured performance with 100% Claude Opus 4.8 without caching: 87% accuracy, 2.8 second P95 latency, and $0.015 per query.

Phase 1: Enable Prompt Caching

Testing prompt caching with a 50/50 split revealed a 48% cache hit rate, reducing cost to $0.008 per query (47% savings) with no change in accuracy.

Phase 2: Intelligent Routing

The final phase implemented intelligent routing based on query complexity:

Query Type	Volume	Model	Cost per Query	Accuracy	Monthly Cost
Simple	650K	Haiku 4.5	$0.003	85%	$1,950
Medium	250K	Sonnet 4.6	$0.009	88%	$2,250
Complex	100K	Opus 4.8	$0.015	92%	$1,500
Total	1M	Mixed	$0.0057	87%	$5,700

Results Summary

Baseline cost with Opus 4.8 and no caching: $15,000 per month.

Final cost with intelligent routing and caching: $5,700 per month.

Total savings: 62%, representing $9,300 per month or $111,600 annually.

Accuracy remained at 87% with no degradation from the optimization.

flowchart TD
    A[User Support Query] --> B{Classify Complexity}
    B -->|Simple 65%| C[Haiku 4.5 $0.003/query]
    B -->|Medium 25%| D[Sonnet 4.6 $0.009/query]
    B -->|Complex 10%| E[Opus 4.8 $0.015/query]
    C --> F[Apply Caching]
    D --> F
    E --> F
    F --> G[Track Metrics]
    G --> H{Monthly Review}
    H -->|Quality Maintained| I[Continue Strategy]
    H -->|Issues Detected| J[Adjust or Rollback]

Best Practices

Leverage Cost Optimization Features

Cache system prompts and static context that remain consistent across queries. Monitor cache hit rate with a target above 40% for meaningful savings. Avoid caching user-specific data that provides no benefit.

Use batch API for email support, analytics, content generation, and overnight jobs. Avoid batch API for real-time chat where latency matters.

Gradual Rollout Schedule

Week 1: 1% traffic to catch catastrophic failures early.

Week 2: 5% traffic to validate metrics and tune thresholds.

Week 3: 25% traffic to gather statistical significance.

Week 4: 50% traffic to accelerate learning.

Week 5: 90% if winning, otherwise rollback.

Automated Rollback Triggers

Define thresholds that trigger automatic rollback:

const ROLLBACK_THRESHOLDS = {
  error_rate: 0.05,
  latency_increase: 1.5,
  cost_spike: 2.0,
  accuracy_drop: 0.10
};

If error rate exceeds 5%, latency increases by 50%, cost doubles, or accuracy drops by 10 percentage points, trigger an automatic rollback to the control variation.

Model Selection Guidelines

For coding and agent tasks, use Claude Opus 4.8.

For math and reasoning tasks, use GPT-5.5.

For multimodal tasks with images or video, use Gemini 3.1 Pro.

For simple, high-volume queries, use Claude Haiku 4.5.

For general-purpose production applications, start with Claude Sonnet 4.6.

Common Pitfalls

Ignoring Thinking Token Costs

GPT-5.5's Thinking variant generates internal reasoning tokens billed at the output rate of $30 per million tokens. A query with 100 visible output tokens may generate 2,000 hidden reasoning tokens, making the actual cost 21 times higher than expected.

Solution: For cost-sensitive tasks, use the Instant variant. When using Thinking, estimate a 10-20x token multiplier.

Not Tracking Cache Hit Rate

Teams often assume 90% caching savings, but actual cache hit rates may be much lower. Expected cost of $0.005 per query with 90% cache hits becomes $0.013 per query with only 15% cache hits.

Solution: Always track cache_hit as a metric. Investigate low hit rates by examining whether system prompts change too frequently or user queries are too unique for caching to benefit.

Testing Too Many Variables Simultaneously

Changing model, prompt, temperature, and caching in a single test makes it impossible to determine which variable caused observed differences.

Solution: Isolate variables. First test models with identical prompts and settings. Then test prompts with the winning model. Then test caching with the winning configuration.

Conclusion

The 2026 AI landscape offers powerful options for every use case. Claude Opus 4.8 leads for coding, GPT-5.5 excels at reasoning, and Gemini 3.1 Pro dominates multimodal applications.

Feature flags enable safe experimentation without code deployments. Cost optimization through prompt caching and intelligent routing can reduce AI costs by 50-67% while maintaining quality. Data-driven decisions based on proper A/B testing consistently outperform gut-feeling model selection.

For teams deploying AI in production, the recommended path forward is:

Set up your first feature flag comparing two or three flagship models.
Enable prompt caching on system prompts for immediate cost reduction.
Run a two-week A/B test on production traffic.
Implement intelligent routing based on query complexity.
Monitor and iterate, re-testing quarterly as models improve.

With proper experimentation infrastructure, you can confidently deploy the right AI model for each task, optimize costs without sacrificing quality, and adapt quickly as the AI landscape continues to evolve.

Related guides

Build a Custom Analytics Integration for Optimizely Web Experimentation

David Sert — Sat, 18 Jul 2026 15:26:40 +0000

Optimizely Web Experimentation ships with native integrations for popular analytics platforms — Google Analytics, Amplitude, Mixpanel, Segment, and others. For many teams, those integrations are sufficient. But teams that run internal analytics dashboards, use niche behavioral analytics tools, route data through custom event pipelines, or push experiment decisions into data lakes often find that the built-in options do not cover their infrastructure.

The Custom Analytics Integration framework is Optimizely's solution to this gap. It lets you define your own integration using a JSON configuration object with a track_layer_decision callback — a JavaScript snippet that executes in the visitor's browser on every experiment bucketing decision. Your callback has access to the full decision context and can call any JavaScript API available on the page: a fetch call to an internal endpoint, a custom window.__analytics object, a tag manager data layer, or anything else your stack exposes.

This guide walks through the complete framework: how it works, how to create an integration, the full variable and API surface available inside the callback, practical examples, and common debugging patterns.

How Custom Analytics Integrations Work

When a visitor arrives on a page running the Optimizely snippet, Optimizely evaluates all active experiments and personalization campaigns. For each experiment where the visitor qualifies and is bucketed, Optimizely fires a bucketing decision. At that point, it iterates over all enabled analytics integrations for that experiment and executes their track_layer_decision callbacks in sequence.

flowchart TB
    A[Visitor arrives on page] --> B[Optimizely evaluates experiments]
    B --> C[Visitor is bucketed into experiment]
    C --> D[Optimizely iterates enabled integrations]
    D --> E[Fires track_layer_decision callback]
    E --> F[Your code sends data to your analytics tool]
    F --> G[Internal dashboard / data lake / custom pipeline]

Each enabled integration fires its callback independently. If you have three integrations enabled for an experiment, all three callbacks execute. The decision context is the same for each — the callbacks do not share state.

The callback runs synchronously in the browser at the moment of bucketing. This happens once per page load for each experiment the visitor is bucketed into. If the visitor is already bucketed (their variation is stored in a cookie or local storage), the callback still fires on subsequent page loads.

Creating an Integration

There are two ways to create a custom analytics integration in Optimizely Web Experimentation: using JSON directly, or using the visual editor. Both approaches configure the same underlying integration object.

Integrations page with the " width="800" height="442">

Using JSON

The JSON approach gives you the most control and is the recommended method for non-trivial integrations. The full schema is:

{
  "plugin_type": "analytics_integration",
  "name": "My Custom Integration",
  "form_schema": [],
  "description": "Sends experiment data to my analytics tool",
  "options": {
    "track_layer_decision": "// your callback code here"
  }
}

Each field serves a specific purpose:

Field	Required	Description
`plugin_type`	Yes	Must be `"analytics_integration"`. Tells Optimizely what kind of plugin this is.
`name`	Yes	Display name shown in the Optimizely UI under Settings > Integrations and in Manage Campaign > Integrations per-experiment toggles.
`form_schema`	Yes	Array of configurable fields that appear in the Optimizely UI per-experiment. Use `[]` for no custom fields.
`description`	No	Human-readable description shown in the integrations list.
`options.track_layer_decision`	Yes	Your callback code as a string. This is a top-level script, not a function body.

To create an integration using JSON:

In your Optimizely project, go to Settings > Integrations.
Click Create Analytics Integration.
Select Using JSON.
Paste your JSON configuration.
Click Save.

The integration appears in the integrations list immediately. To use it, you must enable it globally (from Settings > Integrations) or per-experiment (from Manage Campaign > Integrations inside the experiment editor).

Using the Visual Editor

Optimizely also provides a visual form editor for creating integrations. The visual editor is useful for simpler integrations where you want to configure fields using a UI rather than writing JSON by hand. Under the hood, the visual editor generates the same JSON structure. For complex integrations — particularly those with non-trivial track_layer_decision callbacks or multiple form_schema fields — the JSON approach is more practical.

The track_layer_decision Callback

The track_layer_decision value is a JavaScript string that Optimizely evaluates as a top-level script when a bucketing decision occurs. Understanding what is available inside this callback, and how to use it correctly, is the core skill for building custom integrations.

Available Variables

The following variables are injected into scope when your callback executes:

Variable	Type	Description
`campaignId`	string	The campaign (layer) ID in Optimizely
`experimentId`	string	The experiment ID within the campaign
`variationId`	string	The variation ID assigned to this visitor
`isHoldback`	boolean	Whether the visitor is in the holdback group (i.e., excluded from the experiment)
`campaign`	object	Campaign metadata: `{ id, name, policy }`
`extension`	object	Values from `form_schema` fields configured for this integration, keyed by field `name`

The campaign.policy field indicates the campaign type: "single_experiment" for standard A/B tests, "random" for multi-armed experiments, or "ordered" for personalization campaigns.

Note that campaignId, experimentId, and variationId are all numeric values formatted as strings. If your analytics tool expects integers, use parseInt() when passing these values.

The State API

Access the State API via window['optimizely'].get('state'). This API provides human-readable experiment and variation names, which are more useful than raw IDs in most analytics contexts.

state.getDecisionObject({ campaignId })

Returns an object with human-readable names for the current experiment and variation:

var state = window['optimizely'].get('state');
var decision = state.getDecisionObject({ campaignId: campaignId });
// Returns: { experiment: "Checkout Button Color", variation: "Blue Button" }
// Returns: null if "Mask descriptive names" is enabled in project settings

Always check for a null return value. If your Optimizely project has Mask descriptive names enabled (a privacy setting that strips human-readable names from the snippet), getDecisionObject returns null. Your callback must handle this case and fall back to numeric IDs.

state.getCampaignStates({ isActive: true })

Returns an object containing all currently active campaigns, keyed by campaign ID. Each entry includes bucketing state, experiment assignment, and variation assignment. This is useful when you need context about the full set of active experiments beyond the current decision.

The Utils API

Access the Utils API via window['optimizely'].get('utils'). The most important utility for custom integrations is waitUntil.

utils.waitUntil(predicateFn)

Returns a thenable (a promise-like object). Optimizely polls the predicate function at regular intervals and resolves the thenable when the predicate returns true.

var utils = window['optimizely'].get('utils');
utils.waitUntil(function() {
  return typeof window.myAnalytics !== 'undefined';
}).then(function() {
  // window.myAnalytics is now available
  window.myAnalytics.track('Experiment Viewed', { experimentId: experimentId });
});

Always use waitUntil when your analytics SDK may not be loaded by the time the Optimizely callback fires. Analytics SDKs loaded asynchronously are common, and calling window.myAnalytics.track() before the SDK initializes throws a runtime error and silently drops the event.

Important Constraints

The callback is a top-level script, not a function body. This is the most common source of errors in custom integrations. The track_layer_decision string is evaluated directly — it is not wrapped in a function. Using a bare return statement causes the error:

Invalid options specified: 'return' outside of function

Use early-exit patterns with if blocks instead of return to guard against missing preconditions.

All callback code is publicly readable. The Optimizely snippet is a public JavaScript file loaded in every visitor's browser. Never include API keys, authentication tokens, secrets, or credentials in the callback. If your analytics endpoint requires authentication, use a proxy endpoint that handles auth server-side, or store a write-only public API key in a form_schema field with the understanding that it will be visible in the snippet.

Keep the callback lightweight. The callback executes synchronously on the main thread. Avoid heavy computation, large loops, or synchronous network requests. The utils.waitUntil pattern using async .then() chains is the correct way to defer work.

Building a Complete Integration: Step by Step

The following walkthrough builds a complete integration that sends experiment decision data to an HTTP endpoint using navigator.sendBeacon. This pattern works for any webhook receiver, internal analytics ingestion API, or event pipeline.

Step 1: Define the JSON Structure

Start with the skeleton:

{
  "plugin_type": "analytics_integration",
  "name": "Webhook Integration",
  "form_schema": [],
  "description": "Sends experiment decisions to a webhook endpoint via sendBeacon",
  "options": {
    "track_layer_decision": "// callback goes here"
  }
}

Step 2: Add form_schema Fields

form_schema fields let you configure the integration differently per-experiment, directly from the Optimizely editor UI. Each field becomes available in the callback via extension.fieldName.

"form_schema": [
  {
    "default_value": "https://ingest.example.com/experiment-events",
    "field_type": "text",
    "name": "endpointUrl",
    "label": "Endpoint URL"
  },
  {
    "default_value": "experiment_viewed",
    "field_type": "text",
    "name": "eventName",
    "label": "Event Name"
  },
  {
    "default_value": "enabled",
    "field_type": "dropdown",
    "name": "trackingMode",
    "label": "Tracking Mode",
    "options": {
      "values": ["enabled", "disabled", "debug"]
    }
  }
]

Field values are accessed inside the callback as extension.endpointUrl, extension.eventName, extension.trackingMode. When a field is not configured per-experiment, the default_value is used.

Step 3: Write the Callback

The callback needs to:

Check the trackingMode to respect per-experiment disable/debug settings
Look up human-readable names from the State API
Wait for navigator.sendBeacon availability (it's available in all modern browsers, but the pattern is instructive)
Send the payload

// Check tracking mode from form_schema
if (extension.trackingMode === 'disabled') {
  // Do nothing for this experiment
} else {
  var utils = window['optimizely'].get('utils');
  var state = window['optimizely'].get('state');

  // Look up human-readable names
  var decision = state.getDecisionObject({ campaignId: campaignId });
  var experimentName = decision ? decision.experiment : 'unknown';
  var variationName = decision ? decision.variation : 'unknown';

  var payload = {
    event: extension.eventName || 'experiment_viewed',
    campaignId: campaignId,
    experimentId: experimentId,
    variationId: variationId,
    experimentName: experimentName,
    variationName: variationName,
    isHoldback: isHoldback,
    timestamp: Date.now()
  };

  if (extension.trackingMode === 'debug') {
    console.log('[Optimizely Custom Integration] Decision payload:', payload);
  }

  utils.waitUntil(function() {
    return typeof navigator !== 'undefined' && typeof navigator.sendBeacon === 'function';
  }).then(function() {
    var endpoint = extension.endpointUrl || 'https://ingest.example.com/experiment-events';
    navigator.sendBeacon(endpoint, JSON.stringify(payload));
  });
}

Step 4: Assemble and Install the Full JSON

Inline the callback into the options.track_layer_decision field. Because JSON strings cannot contain raw newlines, either minify the callback or use \\n for newlines. The complete integration JSON:

{
  "plugin_type": "analytics_integration",
  "name": "Webhook Integration",
  "description": "Sends experiment decisions to a webhook endpoint via sendBeacon",
  "form_schema": [
    {
      "default_value": "https://ingest.example.com/experiment-events",
      "field_type": "text",
      "name": "endpointUrl",
      "label": "Endpoint URL"
    },
    {
      "default_value": "experiment_viewed",
      "field_type": "text",
      "name": "eventName",
      "label": "Event Name"
    },
    {
      "default_value": "enabled",
      "field_type": "dropdown",
      "name": "trackingMode",
      "label": "Tracking Mode",
      "options": {
        "values": ["enabled", "disabled", "debug"]
      }
    }
  ],
  "options": {
    "track_layer_decision": "if (extension.trackingMode !== 'disabled') { var utils = window['optimizely'].get('utils'); var state = window['optimizely'].get('state'); var decision = state.getDecisionObject({ campaignId: campaignId }); var experimentName = decision ? decision.experiment : 'unknown'; var variationName = decision ? decision.variation : 'unknown'; var payload = { event: extension.eventName || 'experiment_viewed', campaignId: campaignId, experimentId: experimentId, variationId: variationId, experimentName: experimentName, variationName: variationName, isHoldback: isHoldback, timestamp: Date.now() }; if (extension.trackingMode === 'debug') { console.log('[Optimizely Custom Integration] Decision payload:', payload); } utils.waitUntil(function() { return typeof navigator !== 'undefined' && typeof navigator.sendBeacon === 'function'; }).then(function() { var endpoint = extension.endpointUrl || 'https://ingest.example.com/experiment-events'; navigator.sendBeacon(endpoint, JSON.stringify(payload)); }); }"
  }
}

Paste this into Settings > Integrations > Create Analytics Integration > Using JSON, then save.

To enable the integration for experiments, either enable it globally (all new and existing experiments) or enable it per-experiment from Manage Campaign > Integrations inside the editor.

form_schema Reference

form_schema fields appear in the Optimizely editor under Manage Campaign > Integrations when the integration is enabled for an experiment. They let experiment owners configure the integration without editing JSON. All values are accessed inside the callback via extension.fieldName.

text

Free-form text input. Use for URLs, event names, property names, or any string value.

{
  "field_type": "text",
  "name": "eventName",
  "label": "Event Name",
  "default_value": "Experiment Viewed"
}

Access in callback: extension.eventName

dropdown

Select from a predefined list of options. Use for mode controls, environment selectors, or any field with a fixed set of valid values.

{
  "field_type": "dropdown",
  "name": "environment",
  "label": "Environment",
  "default_value": "production",
  "options": {
    "values": ["production", "staging", "development"]
  }
}

Access in callback: extension.environment

checkbox

Boolean toggle. Useful for feature flags within the integration — for example, enabling verbose logging or controlling whether holdback decisions are tracked.

{
  "field_type": "checkbox",
  "name": "trackHoldback",
  "label": "Track holdback visitors",
  "default_value": "false"
}

Access in callback: extension.trackHoldback — note that checkbox values are strings ("true" / "false"), not booleans. Use extension.trackHoldback === 'true' for conditional checks.

Field name requirements: Field names must be valid JavaScript identifiers. No spaces, no hyphens, no special characters. Use camelCase (eventName, not event-name or event name).

Real-World Examples

Example 1: Send to a Webhook Endpoint

This pattern is useful for teams with internal data pipelines, custom ingestion APIs, or tools like n8n, Zapier, or Make that accept webhook payloads.

navigator.sendBeacon is the preferred method for sending analytics data from the browser: it is fire-and-forget, does not block page unload, and works even when the page is closing. The data is sent as a POST request with Content-Type: text/plain.

var utils = window['optimizely'].get('utils');
var state = window['optimizely'].get('state');

var decision = state.getDecisionObject({ campaignId: campaignId });

var payload = JSON.stringify({
  experimentId: experimentId,
  experimentName: decision ? decision.experiment : null,
  variationId: variationId,
  variationName: decision ? decision.variation : null,
  campaignId: campaignId,
  isHoldback: isHoldback,
  timestamp: new Date().toISOString(),
  url: window.location.href,
  userId: window.optimizely_user_id || null
});

utils.waitUntil(function() {
  return typeof navigator !== 'undefined' && typeof navigator.sendBeacon === 'function';
}).then(function() {
  navigator.sendBeacon('https://hooks.example.com/optimizely-decisions', payload);
});

If your webhook endpoint requires a specific Content-Type, use fetch with a keepalive: true flag instead of sendBeacon. keepalive: true ensures the request completes even if the page unloads before it finishes.

Example 2: Tag a Session Replay Tool

Session replay tools (tools in the category of Hotjar, FullStory, LogRocket, and similar products) let you tag sessions with custom attributes. Tagging sessions with experiment and variation context lets you filter recordings to sessions in specific variations — invaluable for understanding why a variation performed the way it did.

The API differs across tools, but the pattern is the same: wait for the SDK to initialize, then call its tagging method with experiment context.

var utils = window['optimizely'].get('utils');
var state = window['optimizely'].get('state');

var decision = state.getDecisionObject({ campaignId: campaignId });
var experimentName = decision ? decision.experiment : 'exp_' + experimentId;
var variationName = decision ? decision.variation : 'var_' + variationId;

// Wait for the session replay SDK to load
utils.waitUntil(function() {
  return typeof window.__replay !== 'undefined' && typeof window.__replay.tag === 'function';
}).then(function() {
  // Tag the session with experiment context
  // Adapt this to your specific session replay tool's API
  window.__replay.tag({
    'Optimizely Experiment': experimentName,
    'Optimizely Variation': variationName,
    'Optimizely Campaign ID': campaignId,
    'Optimizely Is Holdback': isHoldback
  });
});

For tools that use a push-based API (e.g., window._replayer = window._replayer || []; window._replayer.push(...)) you can call the push directly without waiting:

var state = window['optimizely'].get('state');
var decision = state.getDecisionObject({ campaignId: campaignId });

window._replayer = window._replayer || [];
window._replayer.push(['experiment', {
  id: experimentId,
  name: decision ? decision.experiment : experimentId,
  variation: decision ? decision.variation : variationId
}]);

Example 3: Push to a Data Layer

Teams that route analytics through a tag manager (Google Tag Manager, Tealium, Adobe Launch) often use a window.dataLayer array as the central event bus. Pushing experiment decisions into the data layer makes them available to all tags configured in the tag manager, without requiring changes to the Optimizely integration each time you add a new downstream tool.

var state = window['optimizely'].get('state');
var decision = state.getDecisionObject({ campaignId: campaignId });

// Push decision into GTM data layer
window.dataLayer = window.dataLayer || [];
window.dataLayer.push({
  event: 'optimizely_decision',
  optimizely: {
    campaignId: campaignId,
    experimentId: experimentId,
    experimentName: decision ? decision.experiment : null,
    variationId: variationId,
    variationName: decision ? decision.variation : null,
    isHoldback: isHoldback
  }
});

This does not require waitUntil because window.dataLayer is initialized inline in the GTM snippet, which loads before Optimizely in a correctly configured setup. The fallback window.dataLayer = window.dataLayer || [] handles any edge cases where GTM has not yet initialized.

In Google Tag Manager, create a trigger on the custom event optimizely_decision and use data layer variables to access optimizely.experimentName and optimizely.variationName in your tag configurations.

Debugging Your Integration

Console Logging

Add console.log statements directly in the callback while developing. The simplest approach is to log the full decision context at the top of the callback:

console.log('[MyIntegration] Decision fired', {
  campaignId: campaignId,
  experimentId: experimentId,
  variationId: variationId,
  isHoldback: isHoldback,
  extension: extension
});

Remove or gate these logs with a trackingMode === 'debug' check before deploying to production. console.log calls in the Optimizely snippet run for every visitor and will appear in the browser console of any visitor who opens developer tools.

Optimizely Preview Mode

Preview mode in the Optimizely editor forces a bucketing decision into a specific variation without affecting production traffic. Use it to test your callback without running a live experiment. Open the experiment editor, click Preview, select a variation, and load your site. The callback fires immediately on page load.

Network Tab

Open browser developer tools and filter the Network tab for requests to your analytics endpoint. For navigator.sendBeacon calls, filter by Ping or search for your endpoint URL. Verify the request is sent, the status is 200, and the payload contains the expected fields.

Common Errors

"Invalid options specified" — This error appears in the browser console and means the track_layer_decision script failed to parse or execute. Common causes: JavaScript syntax errors in the callback string, unescaped quotes inside the JSON string, or a bare return statement at the top level.

Callback not firing — Verify the integration is enabled for the specific experiment (check Manage Campaign > Integrations), the visitor qualifies for the experiment (audience conditions are met), and the experiment is running. Use the Optimizely Debugger browser extension to inspect the active state.

Data not appearing in analytics — Check for JavaScript errors in the console. If using waitUntil, verify the predicate ever returns true. Add a timeout fallback if the analytics SDK may not load (for example, if blocked by an ad blocker). Network tab should show outbound requests.

Gotchas

No bare **return at the top level.** The callback is evaluated as a script, not a function body. return at the top level is a syntax error. Use if/else blocks to handle conditional logic instead.

All code is public. The Optimizely snippet is a JavaScript file served from Optimizely's CDN with no access control. Any visitor can read the full contents — including your track_layer_decision callbacks and any values in form_schema. Never include API secrets, private tokens, or credentials. Use write-only public keys (such as analytics ingest keys with no read access) if authentication is required.

getDecisionObject** returns null when Mask descriptive names is enabled.** This is a project-level privacy setting. If your project uses it, all state.getDecisionObject() calls return null. Always have a fallback to numeric IDs and never assume the return value is non-null.

Callbacks fire on every page load. The callback fires each time a bucketed visitor loads a page where the experiment is active — not just on the first visit. If your analytics tool counts unique experiment exposures by de-duplicating on session or user ID, this is fine. If it counts every event, implement deduplication in the callback using sessionStorage to track whether the event has already been sent in this session.

form_schema** field names must be valid JavaScript identifiers.** Field names become property names on the extension object. Names with spaces, hyphens, or special characters are not valid. Use camelCase (endpointUrl, not endpoint-url or endpoint url).

Multiple integrations fire independently. If several custom integrations are enabled for the same experiment, each fires its own callback. They do not share state and cannot communicate. Each callback sees the same campaignId, experimentId, variationId, and extension (where extension contains the fields specific to that integration's form_schema).

Troubleshooting

Callback Not Firing

Check that the integration is enabled for the experiment. Go to Manage Campaign > Integrations inside the experiment editor. The integration must be toggled on.
Check that the visitor qualifies for the experiment. Audience conditions, URL targeting, and traffic allocation must all pass. Use the Optimizely Debugger extension to see which experiments are active.
Check that the experiment is running. Paused, archived, or draft experiments do not fire callbacks.
Verify the Optimizely snippet is on the page. Open the Network tab and confirm the snippet loads before any other scripts that depend on it.

"Invalid options specified" Error

This error means the track_layer_decision value is not valid JavaScript. Common causes:

Unescaped quotes: The callback is a JSON string value. Single quotes inside the callback are fine, but double quotes must be escaped as \\". Alternatively, use single quotes throughout the callback code.
Bare **return statement**: Replace return; with an if block that wraps the code you want to skip.
Syntax error in the callback code: Copy the callback string out of the JSON, paste it into the browser console, and run it. The console will highlight the syntax error with a line number.

Data Not Reaching Analytics

SDK not loaded: Add a console.log immediately before the waitUntil call and inside the .then() callback to verify both points are reached.
waitUntil** predicate never resolves**: If the analytics SDK fails to load (network error, ad blocker, script error), the predicate never returns true and the .then() callback never executes. Add error monitoring or a timeout to detect this case.
Network request blocked: Ad blockers commonly block requests to analytics endpoints by domain pattern. Test in an incognito window with extensions disabled to rule this out.
Payload format rejected: If your endpoint returns a non-2xx status, the event is dropped. Inspect the response in the Network tab.

Wrong Experiment or Variation Names

If state.getDecisionObject() returns null, Mask descriptive names is enabled in your project settings. Optimizely strips human-readable names from the snippet at publish time when this setting is active. Your callback must handle null returns and fall back to numeric IDs. If you need human-readable names and your project uses this setting, maintain a separate ID-to-name mapping outside the callback (for example, a lookup table populated from your analytics tool's own metadata).

Related guides

Server-Side A/B Testing with Optimizely: A Practical Guide

David Sert — Sat, 18 Jul 2026 15:26:28 +0000

Most A/B testing happens in the browser: a script swaps a headline or button color after the page loads. That works for surface-level UI changes, but it cannot test the logic that runs before a page is ever rendered — a pricing algorithm, a search ranking model, a checkout flow, or a backend API response. Server-side A/B testing moves the experiment decision into your application code, where you control the full request lifecycle. This guide explains when to test server-side, how it differs from client-side testing, and how to implement it with Optimizely Feature Experimentation, including working SDK code for Node.js and Python.

What Server-Side A/B Testing Is

In a server-side A/B test, your application server decides which variation a user sees and renders the response accordingly. Instead of shipping the control experience and patching it in the browser, the server already knows the assignment by the time it builds the HTML, the JSON payload, or the rendered component.

The decision is deterministic: a given user ID is consistently bucketed into the same variation, so the experience stays stable across requests and devices. Your code branches on that assignment, serves the corresponding experience, and reports a conversion event when the user does something that matters — a purchase, a signup, a search that returns a click.

This is the model Optimizely calls Feature Experimentation. If you have used Optimizely before, you may know this product by its former name, Full Stack — the SDKs, datafile, and decision model are the same lineage, now under the Feature Experimentation name. Searchers still look for "Optimizely full stack," but the current product and documentation use Feature Experimentation.

When to Test Server-Side

Server-side testing is the right tool when the thing you are changing is not a cosmetic, post-render tweak. Reach for it in these situations:

Backend logic and algorithms. Recommendation engines, search ranking, fraud scoring, feed ordering, and routing logic all live on the server. The browser never sees the alternative implementations, only their output.
Pricing, offers, and business rules. Price tests, discount eligibility, and plan packaging should be decided server-side so the values are authoritative and cannot be inspected or tampered with in the client.
Eliminating flicker. Client-side tests momentarily render the control before the test script rewrites the DOM — the "flash of original content." Because a server-side test renders the correct variation from the first byte, there is no flicker.
Full-stack and omnichannel features. When the same experiment needs to span a web app, a mobile app, and an email pipeline, a server-side decision keyed on a shared user ID gives you one consistent assignment everywhere.
Performance-sensitive paths. Client-side experiment scripts add weight to the page and can delay rendering. A server-side decision adds no client-side JavaScript.

If the change is purely visual and lives entirely in the rendered page — copy, layout, imagery on a marketing page — a client-side tool is often faster to deploy and requires no engineering. Use the model that matches where the change actually lives.

Client-Side vs Server-Side Compared

Dimension	Client-side (Web Experimentation)	Server-side (Feature Experimentation)
Where the decision runs	Browser, after page load	Application server, before response
What it can change	Rendered DOM, styling, copy	Any code path: APIs, algorithms, pricing, UI
Flicker	Possible (flash of original content)	None
Who implements	Marketers, often no code	Engineers, in the codebase
Deployment	Visual editor, instant	Code release, behind feature flags
Best for	Marketing pages, visual tweaks	Backend logic, full-stack and omnichannel features

These are complementary, not competing. Many teams run client-side tests on marketing surfaces and server-side tests on product and backend behavior.

How Optimizely Feature Experimentation Works

Feature Experimentation runs experiments through SDKs you embed in your application (available for Node.js, Python, Java, Go, Ruby, PHP, C#, and more). Each experiment is wrapped in a feature flag. A flag can carry one or more rules — an A/B test rule that splits traffic between variations, or a delivery rule that rolls a feature out to an audience.

The SDK reads a datafile: a JSON snapshot of your project's flags, experiments, audiences, and traffic allocations for one environment. Because the SDK evaluates rules against a locally cached datafile, a decision involves no blocking network call — it resolves in microseconds. The SDK only makes network calls in the background to refresh the datafile and to send event data.

The runtime loop is the same in every language:

flowchart LR
    A[Incoming request] --> B[SDK: create user context]
    B --> C[decide flag]
    C --> D{Variation?}
    D -->|treatment| E[Serve treatment]
    D -->|control| F[Serve control]
    E --> G[track conversion event]
    F --> G
    G --> H[Optimizely results]

You create a user context, call decide on a flag, branch on the returned variation, serve the corresponding experience, and later track a conversion event. Optimizely's Stats Engine ties those events back to the variation and reports the results.

Setting Up the SDK

Install the SDK for your language. For Node.js:

npm install @optimizely/optimizely-sdk

For Python:

pip install optimizely-sdk

Each environment in your project (for example, development and production) has its own SDK key. The SDK uses that key to fetch the matching datafile from Optimizely's CDN. Initialize the client once at application startup and reuse it across requests — do not create a new instance per request.

Initializing in Node.js

For the JavaScript SDK v6 and later, you compose the client from a polling config manager and a batch event processor. Polling keeps the datafile current; batching reduces the number of network calls for event tracking.

import {
  createInstance,
  createPollingProjectConfigManager,
  createBatchEventProcessor,
} from '@optimizely/optimizely-sdk';

const pollingConfigManager = createPollingProjectConfigManager({
  sdkKey: process.env.OPTIMIZELY_SDK_KEY,
  autoUpdate: true,
  updateInterval: 60000, // refresh the datafile every 60 seconds
});

const optimizelyClient = createInstance({
  projectConfigManager: pollingConfigManager,
  eventProcessor: createBatchEventProcessor(),
});

await optimizelyClient.onReady();
// The client is ready: the datafile is loaded and decisions will resolve.

Always wait for onReady() before making decisions. Until the datafile is loaded, decide cannot evaluate rules and will fall back to the flag's default (off) state.

Initializing in Python

The Python SDK takes the SDK key directly and manages datafile polling internally:

from optimizely import optimizely

optimizely_client = optimizely.Optimizely(sdk_key="YOUR_SDK_KEY")

Instantiate this once (for example, as a module-level singleton or in your application factory) and share it across requests.

Making a Decision

A decision requires a user context — an object that pairs a stable user ID with optional attributes. The user ID is the key Optimizely hashes to bucket the user into a variation, so it must be consistent for the same person across requests.

In Node.js:

const attributes = { logged_in: true, plan: 'pro' };
const user = optimizelyClient.createUserContext('user123', attributes);

const decision = user.decide('product_sort');

if (decision.enabled) {
  const sortMethod = decision.variables['sort_method'];
  // Apply the variation's configuration, e.g. sort the catalog by sortMethod
}

if (decision.variationKey === 'treatment') {
  // Serve the treatment experience
} else {
  // Serve the control experience
}

In Python:

user = optimizely_client.create_user_context("user123", {"logged_in": True})
decision = user.decide("product_sort")

if decision.enabled:
    sort_method = decision.variables["sort_method"]
    # Apply the variation's configuration

if decision.variation_key == "treatment":
    pass  # Serve the treatment experience
else:
    pass  # Serve the control experience

The decide call returns a decision object exposing variationKey (the assigned variation), enabled (whether the flag is on for this user), variables (the flag's configuration values for that variation), and reasons (diagnostics when something goes wrong). Calling decide also sends a decision event recording that the user was exposed to the experiment — that exposure is what the results page measures conversions against.

A clean pattern is to drive behavior from flag variables rather than branching on variationKey. Reading sort_method from decision.variables means you can change the experience from the Optimizely UI without shipping new code.

Tracking Events and Metrics

Exposure alone is not a result. To measure impact, track the conversion events that represent value — purchases, signups, upgrades. Call trackEvent (Node) or track_event (Python) with an event key that matches an event you defined in the Optimizely app:

user.trackEvent('purchased');

To attach revenue or other numeric metrics, pass event tags. Optimizely reserves revenue (an integer in cents) and value (a float) for metric aggregation:

tags = {
    "revenue": 10000,  # $100.00, in cents
    "value": 100.00,
}
user.track_event("purchased", tags)

Track one event per real conversion, even when several experiments measure the same action — Optimizely attributes the conversion to every experiment the user was exposed to. The user ID on the tracking call must match the ID used for the decision, or the conversion will not be attributed correctly.

Targeting and Audiences

Audience targeting decides who is eligible for an experiment. You pass attributes when you create the user context, and Optimizely evaluates your audience conditions against them:

const user = optimizelyClient.createUserContext('user123', {
  country: 'US',
  plan: 'pro',
  app_version: '4.3.0',
});

Attributes can be strings, numbers, Booleans, or null. Define the matching audience conditions in the Optimizely app, then scope an experiment rule to that audience. One nuance worth knowing: if you pass an attribute value of the wrong type for a condition (a string where a Boolean is expected) or omit it entirely, that condition is silently skipped and the SDK logs a warning. Pass attributes with consistent types.

For application-version targeting, pass the version as a semantic-version string and use a version audience condition to target ranges.

Common Pitfalls and How to Avoid Them

Server-side experimentation is reliable once it is set up correctly, but a handful of mistakes recur in production implementations.

SDK initialization latency

The SDK must download the datafile before it can make decisions. If you create the client inside a request handler, that request blocks on a network fetch. Initialize the client once at startup, await readiness, and reuse the singleton. If a decision somehow runs before the datafile is ready, the SDK returns the flag's default state — always check enabled and have a sensible control fallback.

Datafile synchronization

The datafile is a cached snapshot, so there is a tradeoff between freshness and network traffic. Optimizely supports three sync strategies:

Pull (recommended) — the SDK polls for a new datafile at an interval you set. More frequent polling means changes propagate faster at the cost of more requests.
Push — a webhook fetches a new datafile the moment your project configuration changes, for near-instant updates.
Custom — fetch the datafile directly from the Optimizely CDN URL and manage caching yourself.

In a fleet of servers, instances poll independently, so a configuration change does not reach every instance at the same instant. For a large fleet, consider running Optimizely Agent — a standalone service that centralizes datafile management and exposes decisions over a REST API, so your application instances do not each maintain their own SDK and datafile.

Sticky bucketing and user IDs

By default the SDKs are stateless: bucketing is a deterministic hash of the user ID and experiment ID, so the same ID always lands in the same variation without any stored state. Two things can break that consistency:

Unstable user IDs. If you use an anonymous ID before login and the customer ID afterward, the user can flip variations mid-session. Create separate user contexts for the anonymous and logged-in journeys, fire events on each, and never mutate the ID on an existing context.
Reconfiguring a running experiment. Decreasing and then increasing traffic allocation, or other mid-flight changes, can rebucket users who have no persisted assignment. To pin assignments, implement a User Profile Service — a lookup/save pair backed by a store like Redis that persists each user's variation. This is the SDK's sticky-bucketing mechanism and is the recommended safeguard if you anticipate changing an experiment while it runs.

Server-side SDKs do not run in the browser

A common architectural mistake is calling a server SDK from client code. Keep server-side decisions on the server. If you need a decision in the browser, render it into the page or expose it through your own API — do not embed your server SDK key in client-side JavaScript.

Verifying Your Results

Before trusting an experiment, confirm the full loop works end to end:

Confirm exposure. Make a decision for a few test user IDs and check that decide returns the expected variations and that decision events appear in your Optimizely environment.
Confirm tracking. Trigger the conversion event and verify it lands on the Experiment Results page, attributed to the right experiment and variation.
Confirm bucketing stability. Re-run decisions for the same user IDs and check the assignment does not change between calls.
Check for sample ratio mismatch. If the observed split between variations diverges from the configured allocation, Optimizely's automatic SRM detection flags it — usually a sign of inconsistent user IDs or a logging gap.
Read results with the Stats Engine. Let the experiment accumulate enough conversions to reach statistical significance before acting on the numbers.

A correctly instrumented server-side test gives you something a client-side tool cannot: confidence that you measured a real change in backend behavior, with no flicker, no client weight, and a decision your application fully controls.