Building a Production-Ready A/B Testing Framework in Next.js

📋 Table of Contents

1. Introduction
2. Why Build Your Own A/B Testing Framework?
3. Architecture Overview
4. Core Implementation
5. Usage Patterns
6. Analytics Integration
7. Best Practices
8. Performance Considerations
9. Lessons Learned

---

🎯 Introduction

A/B testing is crucial for data-driven product decisions, but many third-party solutions come with tradeoffs:

• Performance overhead from external scripts
• Privacy concerns with data sharing
• Cost at scale
• Limited customization for complex use cases

This talk covers how we built a lightweight, type-safe A/B testing framework directly into our Next.js application, giving us full control over the testing lifecycle.

What We'll Cover

• Client-side variant assignment with persistent storage
• Type-safe test configuration
• Event tracking with GA4 integration
• Real-world implementation examples
• Performance optimization strategies

---

🤔 Why Build Your Own A/B Testing Framework?

The Business Case

Requirement	Third-Party	Custom Solution
Bundle Size	+50-200KB	+5KB
Privacy	Data shared	Full control
Customization	Limited	Unlimited
Cost at 1M users	$500-2000/mo	Infrastructure only
TypeScript Support	Varies	Native

Key Benefits

1. Zero external dependencies - Complete control over implementation
2. Type safety - Catch errors at compile time
3. Performance - Minimal runtime overhead
4. Privacy compliant - Data stays in your infrastructure
5. Flexibility - Customize for any use case

---

🏗️ Architecture Overview

High-Level Design

┌─────────────────────────────────────────┐
│         Next.js Application             │
├─────────────────────────────────────────┤
│  ┌─────────────────────────────────┐   │
│  │   ABTestProvider (Context)      │   │
│  │   - Variant Assignment          │   │
│  │   - Persistence Layer           │   │
│  │   - Event Tracking              │   │
│  └─────────────────────────────────┘   │
│              ↓                          │
│  ┌─────────────────────────────────┐   │
│  │      Components/Pages           │   │
│  │   - useABTest() hook            │   │
│  │   - getVariant<T>()             │   │
│  │   - trackEvent()                │   │
│  └─────────────────────────────────┘   │
└─────────────────────────────────────────┘
         ↓                      ↓
   localStorage          Google Analytics
                              OR
                       Custom Endpoint

Core Modules

lib/ab-testing/
├── types.ts          # TypeScript interfaces
├── context.tsx       # React context provider
├── utils.ts          # Variant assignment logic
├── analytics.ts      # Event tracking
└── index.ts          # Public API exports

---

💻 Core Implementation

1. Type Definitions

The foundation is strong TypeScript types:

// types.ts
export interface ABTestVariant<T = any> {
  id: string;
  name: string;
  weight: number;        // 0-100, probability of assignment
  value: T;              // Generic type for variant data
}

export interface ABTest<T = any> {
  id: string;
  name: string;
  description?: string;
  variants: ABTestVariant<T>[];
  enabled: boolean;
  startDate?: Date;
  endDate?: Date;
}

export interface ABTestAssignment {
  testId: string;
  variantId: string;
  assignedAt: number;
}

export interface ABTestEvent {
  testId: string;
  variantId: string;
  eventType: 'view' | 'click' | 'conversion' | 'custom';
  eventName?: string;
  metadata?: Record<string, any>;
  timestamp: number;
}

Key Design Decisions:

• ✅ Generic types (<T>) for type-safe variant values
• ✅ Weight-based assignment for flexible traffic splitting
• ✅ Rich event types for comprehensive tracking
• ✅ Optional metadata for custom analytics

---

2. Variant Assignment Algorithm

Weighted random selection with normalization:

// utils.ts
export function assignVariant<T>(test: ABTest<T>): ABTestVariant<T> | null {
  if (!test.enabled || test.variants.length === 0) return null;

  // Normalize weights to ensure they sum to 100
  const totalWeight = test.variants.reduce((sum, v) => sum + v.weight, 0);
  if (totalWeight === 0) return test.variants[0];

  const random = Math.random() * totalWeight;
  let cumulative = 0;

  for (const variant of test.variants) {
    cumulative += variant.weight;
    if (random <= cumulative) {
      return variant;
    }
  }

  return test.variants[test.variants.length - 1];
}

How It Works:

1. Generate random number between 0 and total weight
2. Iterate through variants, accumulating weights
3. Return first variant where random < cumulative weight

Example:

Variants: [A: 50, B: 30, C: 20]
Random: 65
Cumulative: 0 → 50 → 80 → 100
Result: Variant B (65 falls between 50-80)

---

3. Persistent Storage

Users get the same variant across sessions:

// utils.ts
export function storeAssignment(
  assignment: ABTestAssignment, 
  key: string
): void {
  if (typeof window === 'undefined') return;

  try {
    const stored = getStoredAssignments(key);
    const filtered = stored.filter(a => a.testId !== assignment.testId);
    filtered.push(assignment);
    localStorage.setItem(key, JSON.stringify(filtered));
  } catch (error) {
    console.error('[A/B Testing] Failed to store assignment:', error);
  }
}

export function getStoredAssignments(key: string): ABTestAssignment[] {
  if (typeof window === 'undefined') return [];

  try {
    const stored = localStorage.getItem(key);
    return stored ? JSON.parse(stored) : [];
  } catch (error) {
    console.error('[A/B Testing] Failed to retrieve assignments:', error);
    return [];
  }
}

Why localStorage?

• ✅ No server roundtrip needed
• ✅ Works offline
• ✅ Simple implementation
• ✅ Privacy-friendly (stays on device)

Alternative: User ID-based Assignment

For logged-in users, deterministic assignment:

export function assignVariantByUserId<T>(
  test: ABTest<T>,
  userId: string
): ABTestVariant<T> | null {
  const hash = hashString(userId + test.id);
  const totalWeight = test.variants.reduce((sum, v) => sum + v.weight, 0);
  const index = hash % totalWeight;

  let cumulative = 0;
  for (const variant of test.variants) {
    cumulative += variant.weight;
    if (index < cumulative) {
      return variant;
    }
  }

  return test.variants[test.variants.length - 1];
}

---

4. React Context Provider

Centralized state management:

// context.tsx
'use client';

import React, { createContext, useContext, useEffect, useState } from 'react';

interface ABTestContextValue {
  assignments: Map<string, string>;
  getVariant: <T>(testId: string) => T | null;
  trackEvent: (event: Omit<ABTestEvent, 'timestamp'>) => void;
  refreshAssignments: () => void;
}

const ABTestContext = createContext<ABTestContextValue | null>(null);

export function ABTestProvider({
  tests,
  children,
  persistenceKey = 'app_ab_tests',
  analyticsEndpoint,
  debug = false,
}: ABTestProviderProps) {
  const [assignments, setAssignments] = useState<Map<string, string>>(new Map());
  const [isInitialized, setIsInitialized] = useState(false);

  // Initialize assignments on mount
  useEffect(() => {
    const storedAssignments = getStoredAssignments(persistenceKey);
    const newAssignments = new Map<string, string>();

    tests.forEach(test => {
      if (!test.enabled) return;

      // Check if we have a stored assignment
      const stored = storedAssignments.find(a => a.testId === test.id);
      
      if (stored && test.variants.some(v => v.id === stored.variantId)) {
        newAssignments.set(test.id, stored.variantId);
      } else {
        // Assign new variant
        const variant = assignVariant(test);
        if (variant) {
          newAssignments.set(test.id, variant.id);
          storeAssignment(
            { testId: test.id, variantId: variant.id, assignedAt: Date.now() },
            persistenceKey
          );
        }
      }
    });

    setAssignments(newAssignments);
    setIsInitialized(true);

    if (debug) {
      console.log('[A/B Testing] Initialized:', Object.fromEntries(newAssignments));
    }
  }, [tests, persistenceKey, debug]);

  const getVariant = useCallback(
    <T,>(testId: string): T | null => {
      const variantId = assignments.get(testId);
      if (!variantId) return null;

      const test = tests.find(t => t.id === testId);
      const variant = test?.variants.find(v => v.id === variantId);
      return variant?.value ?? null;
    },
    [assignments, tests]
  );

  // Prevent flash of wrong variant
  if (!isInitialized) return null;

  return (
    <ABTestContext.Provider value={{ assignments, getVariant, ... }}>
      {children}
    </ABTestContext.Provider>
  );
}

export function useABTest() {
  const context = useContext(ABTestContext);
  if (!context) {
    throw new Error('useABTest must be used within ABTestProvider');
  }
  return context;
}

Key Features:

• ✅ Prevents flashing - Wait for initialization before render
• ✅ Memoized getters - Optimized performance
• ✅ Type-safe hooks - Full TypeScript support
• ✅ Debug mode - Console logging for development

---

5. Event Tracking

Multi-platform analytics support:

// analytics.ts
export function trackEvent(
  event: ABTestEvent,
  endpoint?: string,
  debug?: boolean
): void {
  if (debug) {
    console.log('[A/B Testing] Event tracked:', event);
  }

  // Send to custom endpoint
  if (endpoint) {
    sendToAnalytics(event, endpoint);
  }

  // Send to browser analytics
  sendToBrowserAnalytics(event);
}

function sendToBrowserAnalytics(event: ABTestEvent): void {
  // Google Analytics 4
  if (typeof window !== 'undefined' && 'gtag' in window) {
    (window as any).gtag('event', event.eventType, {
      event_category: 'ab_test',
      test_id: event.testId,
      variant_id: event.variantId,
      event_label: event.eventName,
      ...event.metadata,
    });
  }

  // Custom data layer
  if (typeof window !== 'undefined' && 'dataLayer' in window) {
    (window as any).dataLayer.push({
      event: 'ab_test_event',
      abTest: {
        testId: event.testId,
        variantId: event.variantId,
        eventType: event.eventType,
        eventName: event.eventName,
        metadata: event.metadata,
      },
    });
  }
}

// Convenience helpers
export function trackConversion(
  testId: string,
  variantId: string,
  conversionValue?: number
): void {
  trackEvent({
    testId,
    variantId,
    eventType: 'conversion',
    eventName: 'conversion',
    metadata: { value: conversionValue },
    timestamp: Date.now(),
  });
}

---

🔧 Usage Patterns

1. Define Your Tests

Create a centralized configuration:

// lib/ab-test-config.ts
import { ABTest } from './ab-testing';

export interface HeroVariant {
  headline: string;
  subheadline: string;
  ctaText: string;
  colorScheme: 'purple' | 'blue' | 'green';
}

export const abTests: ABTest[] = [
  {
    id: 'hero_headline_test',
    name: 'Hero Headline Test',
    description: 'Test different headline copy',
    enabled: true,
    variants: [
      {
        id: 'control',
        name: 'Control',
        weight: 50,
        value: {
          headline: 'Build Faster',
          subheadline: 'Ship with confidence',
          ctaText: 'Get Started',
          colorScheme: 'purple',
        } as HeroVariant,
      },
      {
        id: 'variant_a',
        name: 'Variant A - Action-Oriented',
        weight: 50,
        value: {
          headline: 'Ship Today',
          subheadline: 'Deploy in minutes',
          ctaText: 'Start Building',
          colorScheme: 'blue',
        } as HeroVariant,
      },
    ],
  },
];

2. Wrap Your App

// app/layout.tsx
import { ABTestProvider } from '@/lib/ab-testing';
import { abTests } from '@/lib/ab-test-config';

export default function RootLayout({ children }) {
  return (
    <html>
      <body>
        <ABTestProvider 
          tests={abTests}
          debug={process.env.NODE_ENV === 'development'}
        >
          {children}
        </ABTestProvider>
      </body>
    </html>
  );
}

3. Use in Components

// app/page.tsx
'use client';

import { useABTest } from '@/lib/ab-testing';
import { HeroVariant } from '@/lib/ab-test-config';

export default function HomePage() {
  const { getVariant, trackEvent } = useABTest();
  
  // Get variant with type safety
  const hero = getVariant<HeroVariant>('hero_headline_test');

  // Track page view
  useEffect(() => {
    if (hero) {
      trackEvent({
        testId: 'hero_headline_test',
        variantId: hero.colorScheme === 'purple' ? 'control' : 'variant_a',
        eventType: 'view',
        eventName: 'homepage_view',
      });
    }
  }, [hero, trackEvent]);

  // Track interactions
  const handleCTAClick = () => {
    trackEvent({
      testId: 'hero_headline_test',
      variantId: hero.colorScheme === 'purple' ? 'control' : 'variant_a',
      eventType: 'click',
      eventName: 'cta_click',
      metadata: { ctaText: hero.ctaText },
    });
  };

  return (
    <div>
      <h1>{hero?.headline}</h1>
      <p>{hero?.subheadline}</p>
      <button onClick={handleCTAClick}>
        {hero?.ctaText}
      </button>
    </div>
  );
}

---

📊 Analytics Integration

Google Analytics 4

Add to your layout:

// app/layout.tsx
import Script from 'next/script';

export default function RootLayout({ children }) {
  return (
    <html>
      <head>
        <Script src="https://www.googletagmanager.com/gtag/js?id=GA_MEASUREMENT_ID" />
        <Script id="google-analytics">
          {`
            window.dataLayer = window.dataLayer || [];
            function gtag(){dataLayer.push(arguments);}
            gtag('js', new Date());
            gtag('config', 'GA_MEASUREMENT_ID');
          `}
        </Script>
      </head>
      <body>
        <ABTestProvider tests={abTests}>
          {children}
        </ABTestProvider>
      </body>
    </html>
  );
}

Events automatically flow to GA4 with this structure:

{
  event_category: 'ab_test',
  test_id: 'hero_headline_test',
  variant_id: 'control',
  event_label: 'cta_click'
}

Custom Analytics Endpoint

Create an API route:

// app/api/analytics/route.ts
import { NextRequest, NextResponse } from 'next/server';

export async function POST(req: NextRequest) {
  const event = await req.json();
  
  // Store in database
  await db.abTestEvents.create({
    data: {
      testId: event.testId,
      variantId: event.variantId,
      eventType: event.eventType,
      eventName: event.eventName,
      metadata: event.metadata,
      timestamp: new Date(event.timestamp),
    },
  });
  
  return NextResponse.json({ success: true });
}

Configure provider:

<ABTestProvider 
  tests={abTests}
  analyticsEndpoint="/api/analytics"
/>

---

✅ Best Practices

1. Test One Thing at a Time

❌ Bad:

{
  id: 'homepage_redesign',
  variants: [
    { 
      value: { 
        headline: 'New Headline',
        layout: 'two-column',
        showTestimonials: true,
        ctaColor: 'blue'
      }
    }
  ]
}

✅ Good:

[
  { id: 'headline_test', variants: [...] },
  { id: 'layout_test', variants: [...] },
  { id: 'testimonials_test', variants: [...] },
  { id: 'cta_color_test', variants: [...] },
]

2. Set Clear Success Metrics

export interface TestMetrics {
  primary: 'conversion_rate' | 'click_through_rate' | 'revenue';
  secondary?: string[];
  minimumSampleSize: number;
  minimumDuration: number; // days
}

export const abTests: ABTest[] = [
  {
    id: 'cta_test',
    name: 'CTA Button Test',
    metrics: {
      primary: 'conversion_rate',
      secondary: ['time_on_page', 'bounce_rate'],
      minimumSampleSize: 1000,
      minimumDuration: 14,
    },
    variants: [...],
  },
];

3. Handle Edge Cases

// Always provide fallbacks
const hero = getVariant<HeroVariant>('hero_test');

return (
  <h1>{hero?.headline || 'Default Headline'}</h1>
);

// Check if test is enabled before tracking
if (hero) {
  trackEvent({...});
}

4. Clean Up Completed Tests

// Mark tests as completed
{
  id: 'old_hero_test',
  enabled: false,  // Disable after winner is chosen
  endDate: new Date('2025-01-15'),
  winner: 'variant_a',
  results: {
    control: { conversionRate: 0.12 },
    variant_a: { conversionRate: 0.15 },
  }
}

5. Admin Panel for Testing

// app/ab-admin/page.tsx
'use client';

import { useABTest } from '@/lib/ab-testing';
import { abTests } from '@/lib/ab-test-config';

export default function ABAdminPage() {
  const { assignments, refreshAssignments } = useABTest();

  return (
    <div>
      <h1>A/B Test Admin</h1>
      
      <h2>Current Assignments</h2>
      {abTests.map(test => (
        <div key={test.id}>
          <h3>{test.name}</h3>
          <p>Variant: {assignments.get(test.id)}</p>
        </div>
      ))}

      <button onClick={refreshAssignments}>
        Reset & Get New Variants
      </button>
    </div>
  );
}

---

⚡ Performance Considerations

1. Prevent Layout Shift

❌ Bad:

export default function Page() {
  const variant = getVariant('test');
  
  // Causes flash on first render
  return <div>{variant?.text}</div>;
}

✅ Good:

export function ABTestProvider({ children }) {
  const [isInitialized, setIsInitialized] = useState(false);
  
  // ... assignment logic
  
  // Wait for initialization
  if (!isInitialized) return null;
  
  return (
    <ABTestContext.Provider value={...}>
      {children}
    </ABTestContext.Provider>
  );
}

2. Minimize Bundle Size

Our framework adds only ~5KB (minified + gzipped):

types.ts:      ~1KB
context.tsx:   ~2KB
utils.ts:      ~1KB
analytics.ts:  ~1KB
Total:         ~5KB

Compare to alternatives:

• Optimizely: ~50KB+
• Google Optimize: ~70KB+
• VWO: ~100KB+

3. Optimize Re-renders

// Memoize variant getter
const getVariant = useCallback(
  <T,>(testId: string): T | null => {
    const variantId = assignments.get(testId);
    if (!variantId) return null;

    const test = tests.find(t => t.id === testId);
    const variant = test?.variants.find(v => v.id === variantId);
    return variant?.value ?? null;
  },
  [assignments, tests] // Only re-create when these change
);

4. SSR Considerations

A/B tests are client-side only to avoid:

• ❌ Hydration mismatches
• ❌ Inconsistent variant assignments
• ❌ Caching issues

// Always use 'use client' directive
'use client';

import { useABTest } from '@/lib/ab-testing';

---

🎓 Lessons Learned

What Worked Well

1. Type Safety - Caught numerous bugs at compile time
2. Centralized Configuration - Easy to manage all tests in one place
3. Lightweight - No external dependencies = fast load times
4. Flexible - Easy to adapt for new use cases

Challenges & Solutions

Challenge 1: Flash of Control Variant

Problem: Users briefly see control before variant loads.

Solution: Return null from provider until initialized:

if (!isInitialized) return null;

Challenge 2: Testing Locally

Problem: Hard to see different variants during development.

Solution: Admin panel + localStorage manipulation:

export function refreshAssignments() {
  localStorage.removeItem('app_ab_tests');
  window.location.reload();
}

Challenge 3: Statistical Significance

Problem: Need to know when to end a test.

Solution: Built helper utilities:

export function calculateSignificance(
  controlConversions: number,
  controlTotal: number,
  variantConversions: number,
  variantTotal: number
): { pValue: number; isSignificant: boolean } {
  // Z-test implementation
  // ...
}

Future Improvements

1. Server-side variant assignment for SEO-critical tests
2. Multi-variate testing (test combinations of features)
3. Automatic winner detection based on statistical significance
4. Visual editor for non-technical users
5. Integration with feature flags for gradual rollouts

---

📈 Real-World Results

Case Study: Hero Headline Test

Setup:

• 3 variants (Control, Variant A, Variant B)
• Ran for 14 days
• 50,000 visitors

Results:

Control:    12.3% conversion rate (baseline)
Variant A:  14.8% conversion rate (+20% lift) 🎉
Variant B:  11.9% conversion rate (-3% drop)

Winner: Variant A deployed to 100% of users

Impact:

• +20% conversion rate
• +$150K annual revenue (estimated)
• 0 cost for testing infrastructure

---

🚀 Getting Started

Quick Setup (5 minutes)

1. Copy the framework files:

lib/ab-testing/
├── types.ts
├── context.tsx
├── utils.ts
├── analytics.ts
└── index.ts

2. Create your test config:

// lib/ab-test-config.ts
export const abTests: ABTest[] = [
  {
    id: 'my_first_test',
    name: 'My First Test',
    enabled: true,
    variants: [
      { id: 'control', name: 'Control', weight: 50, value: { ... } },
      { id: 'variant_a', name: 'Variant A', weight: 50, value: { ... } },
    ],
  },
];

3. Wrap your app:

// app/layout.tsx
import { ABTestProvider } from '@/lib/ab-testing';
import { abTests } from '@/lib/ab-test-config';

export default function RootLayout({ children }) {
  return (
    <ABTestProvider tests={abTests}>
      {children}
    </ABTestProvider>
  );
}

4. Use in components:

const { getVariant, trackEvent } = useABTest();
const variant = getVariant<MyType>('my_first_test');

---

📚 Additional Resources

• Full Implementation: GitHub Repo
• Documentation: README.md
• Admin Panel: Demo
• Statistical Calculator: Tool

---

🙋 Q&A

Common Questions

Q: Can this work with server components?
A: Currently client-side only. Server-side A/B testing requires edge compute (Vercel Edge Functions, Cloudflare Workers).

Q: How do you handle bot traffic?
A: Filter in analytics layer. Most bots don't execute JavaScript, so they won't trigger events.

Q: What about GDPR compliance?
A: localStorage doesn't require consent in most jurisdictions (it's functional, not tracking). Consult your legal team.

Q: Can I test more than 2 variants?
A: Yes! Add as many variants as you want. Just ensure weights sum to reasonable values.

Q: How long should I run a test?
A: Minimum 1-2 weeks and at least 1000 conversions per variant for statistical significance.

---

💡 Key Takeaways

1. ✅ Build vs Buy - For many cases, a custom solution is simpler and faster
2. ✅ Type Safety - TypeScript makes A/B testing maintainable at scale
3. ✅ User Experience - Prevent flashing with careful initialization
4. ✅ Metrics Matter - Track the right events to make informed decisions
5. ✅ Keep It Simple - Start with basic tests, iterate based on needs

---

Questions?

Feel free to reach out or check the implementation in our codebase!

Happy Testing! 🧪