Retraining Strategies: Batch vs Incremental vs Hybrid

Full Batch Retrain

Three retraining strategies dominate production systems, each with distinct trade-offs. Full batch retrain starts from scratch on a sliding window (last 7 to 28 days), recomputing all weights. This is the safest and most reproducible approach, used by Netflix for nightly homepage personalization retrains. The cost is high compute (training clusters must handle peak load) and slow reaction (cannot deploy faster than training completes, typically hours to days for large models).

Incremental Updates

Incremental or online updates continue training from the previous model checkpoint, processing only new data. Meta uses this for embedding layers and calibrators, updating every 15 to 60 minutes to capture trending topics in feeds. The benefit is speed and low cost (only process deltas), but risks accumulate: catastrophic forgetting (model loses older patterns), bias drift (overweights recent data), and harder validation (no clean baseline).

Hybrid Strategies

Hybrid strategies combine both: Uber runs daily full retrains for core ETA models while updating nearline contextual features (traffic conditions, weather) every 5 minutes. This balances stability (full retrain prevents drift accumulation) with responsiveness (nearline features capture real-time conditions).

Cost Comparison

The pattern is periodic full retrain (daily to weekly) plus frequent small updates (minutes to hours) for fast-moving signals. Cost-wise, hybrid is the most expensive but delivers the best accuracy for high-value use cases like fraud detection or dynamic pricing.

Full Batch Retrain

• Last 21 days → Train from scratch
• Cost: High compute │ Cadence: Daily
• Examples: Netflix homepage, Airbnb pricing

Incremental Update

• New data → Warm start from checkpoint
• Cost: Low compute │ Cadence: 15–60 min
• Examples: Meta embeddings, calibrators

Hybrid Strategy

• Daily full + 5 min nearline features
• Cost: Highest │ Accuracy: Best
• Examples: Uber ETA, fraud detection

Key Takeaways

✓ Full batch retrain on 7 to 28 day sliding windows is safest and most reproducible but costs high compute and cannot react faster than training completes (hours to days for large models like Netflix homepage personalization)

✓ Incremental updates reduce cost by 10x and enable 15 to 60 minute cadences (Meta embedding updates) but risk catastrophic forgetting, bias drift toward recent data, and harder offline validation without clean baselines

✓ Hybrid strategies balance stability and responsiveness: Uber runs daily full retrains for core ETA models while updating nearline traffic and weather features every 5 minutes, costing 3x more but improving accuracy by 5 to 8 percent

✓ Warm start initialization from previous checkpoint speeds convergence by 30 to 50 percent for full retrains and preserves learned patterns, but requires careful learning rate tuning to avoid getting stuck in local minima

✓ Incremental updates work best for stateless components (embeddings, calibrators) or when paired with weekly full retrains to prevent drift accumulation and catastrophic forgetting over time

Interview Tips

1. Meta updates ad ranking embeddings incrementally every 15 minutes to capture trending topics, but runs full retrains daily to reset and prevent bias accumulation, maintaining AUC ROC within 0.5 percent of batch trained models
2. Netflix runs nightly full retrains for recommendation models processing hundreds of millions of interactions, using warm start from previous day to reduce training time from 8 hours to 5 hours