Testing in Production: Strategies and Safeguards

June 3, 2019

No staging environment perfectly replicates production. Traffic patterns differ, data volumes vary, and real users behave unpredictably. Testing in production acknowledges this reality and provides strategies for safe, effective production validation.

Why Test in Production

Staging Isn’t Production

Staging differs from production:

Bugs that pass staging fail in production.

Production-Only Scenarios

Some things can only be tested in production:

Shift-Right Testing

Complement pre-production testing with production validation:

Traditional: Dev → Test → Staging → Production
Shift-Right: Dev → Test → Staging → Production + Monitoring + Validation

Strategies

Feature Flags

Control feature exposure:

def checkout():
    if feature_flags.is_enabled("new_payment_flow", user=current_user):
        return new_payment_flow()
    return legacy_payment_flow()

Rollout stages:

0%: Feature off (deployed but inactive)
1%: Internal users only
5%: Beta users
25%: Progressive rollout
100%: Full rollout

Canary Releases

Route percentage of traffic to new version:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
spec:
  http:
  - route:
    - destination:
        host: api
        subset: v1
      weight: 95
    - destination:
        host: api
        subset: v2
      weight: 5

Compare canary metrics against baseline.

Dark Launching

Run new code path without affecting users:

def get_recommendations(user_id):
    # Always return current implementation
    result = current_recommendations(user_id)

    # Shadow call to new implementation
    if random.random() < 0.1:  # 10% sample
        try:
            new_result = new_recommendations(user_id)
            log_comparison(result, new_result)
        except Exception as e:
            log_shadow_error(e)

    return result

Validate correctness before switching.

Traffic Mirroring

Send copy of traffic to new version:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
spec:
  http:
  - route:
    - destination:
        host: api-v1
    mirror:
      host: api-v2
    mirrorPercentage:
      value: 10

New version handles requests but responses are discarded.

Synthetic Monitoring

Automated tests running continuously:

def synthetic_checkout_test():
    # Create test order with synthetic user
    response = api.create_order(
        user_id="synthetic-user-001",
        items=[TEST_PRODUCT],
        payment_method="test-card"
    )

    assert response.status_code == 200
    assert response.json()['status'] == 'confirmed'

    # Clean up
    api.cancel_order(response.json()['id'])

Detect issues before users do.

Chaos Engineering

Inject failures intentionally:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payment-latency
spec:
  action: delay
  mode: one
  selector:
    labelSelectors:
      app: payment
  delay:
    latency: 500ms
  duration: 5m

Validate system handles failures gracefully.

Safeguards

Observability

Can’t test in production without seeing what happens:

def process_order(order):
    with tracer.start_span("process_order") as span:
        span.set_attribute("order_id", order.id)
        span.set_attribute("variant", get_variant(order.user))

        result = do_process(order)

        metrics.record("order_processed",
            variant=get_variant(order.user),
            success=result.success,
            latency=result.latency)

        return result

Automated Rollback

Roll back automatically on degradation:

class CanaryAnalyzer:
    def should_rollback(self, canary_metrics, baseline_metrics):
        if canary_metrics.error_rate > baseline_metrics.error_rate * 2:
            return True
        if canary_metrics.p99_latency > baseline_metrics.p99_latency * 1.5:
            return True
        return False

Blast Radius Limits

Limit impact of failures:

# Maximum 5% of traffic to experiment
if get_experiment_traffic_percentage() > 5:
    disable_experiment()

# Automatic disable after threshold
if experiment_error_count > 100:
    disable_experiment()

Kill Switches

Instant feature disable:

@app.route('/admin/kill-switch/<feature>')
def kill_switch(feature):
    feature_flags.force_disable(feature)
    return f"Feature {feature} disabled"

Data Isolation

Protect real data:

def is_synthetic_user(user_id):
    return user_id.startswith("synthetic-") or user_id.startswith("test-")

def process_order(order):
    if is_synthetic_user(order.user_id):
        return process_test_order(order)
    return process_real_order(order)

Metrics for Production Testing

Compare Variants

# Error rate by variant
sum(rate(http_requests_total{status=~"5.."}[5m])) by (variant)
/
sum(rate(http_requests_total[5m])) by (variant)

# Latency by variant
histogram_quantile(0.99,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, variant)
)

Statistical Significance

Don’t make decisions on small samples:

def is_significant(control, experiment, confidence=0.95):
    from scipy import stats

    t_stat, p_value = stats.ttest_ind(control, experiment)
    return p_value < (1 - confidence)

Business Metrics

Technical metrics aren’t enough:

Conversion rate by variant
Revenue per user by variant
User engagement by variant

When Not to Test in Production

High-Risk Changes

Database schema migrations → Test thoroughly first
Security-sensitive code → Extra review and testing
Financial calculations → Extensive verification
Compliance-related → Full audit trail

When Rollback Is Hard

Data migrations → May not be reversible
External API changes → Partners depend on it
Contract changes → Legal implications

Key Takeaways

Production testing is a powerful tool when done safely. The safeguards are as important as the testing itself.