The baseline and a fair backtest
A backtest that looks good is usually a backtest with a leak. The model saw a future value through an aggregated feature, or the training and test periods overlapped, or a promotion flag was populated retrospectively in a way that would never be available at prediction time. Each of these produces flattering results that evaporate in production, and each is easy to introduce by accident. The discipline is to reconstruct exactly what would have been known at the moment of forecasting.
Rolling origin evaluation is the practical form of that. Fit on data up to a point, forecast forward over the horizon you actually need, step forward and repeat across the history, then compare against the naive baseline over the same windows. Metric selection matters too: mean absolute percentage error behaves badly around small and zero values, which is precisely where intermittent demand lives, so a scaled or weighted absolute measure is usually more informative.
The comparison has to happen at the grain and horizon that matter. A model accurate at national level in monthly buckets tells you nothing about whether it can support weekly replenishment at a single location. Accuracy generally improves as you aggregate, which is why aggregate figures are the ones that get presented in review meetings. We report at the grain the decision is actually taken, even when the number is less comfortable to show.
- A naive baseline computed first and used as the standard every model must beat
- Rolling origin backtesting that reconstructs only what was known at prediction time
- Feature availability checked, so nothing leaks information from the future
- Error metrics chosen for the series, avoiding percentage errors on small or zero values
- Accuracy reported at the grain and horizon of the real decision, not a flattering aggregate