Time-series cross-validation (rolling origin)

Deep dive

How it works

In: series of length n, horizon h, step, gap, min_train, window type, a fit-and-forecast function · Out: forecast errors per fold and per horizon step · State: none (folds are independent refits) · Cost: k folds, each one fit plus one predict

1. Fix the deployment setting: horizon h, how often you retrain (step), and gap (samples between train end and test start; at least the label look-ahead or the longest lag/rolling-window feature, else leakage).
2. Choose min_train (at least two seasons for seasonal models) and window type: expanding or sliding of length w.
3. For origin o = min_train, min_train + step, ... while o + gap + h <= n: train on [o-w, o) (or [0, o)), predict [o+gap, o+gap+h), store errors by horizon step.
4. Aggregate: mean over folds per horizon step (MAE, RMSE, MASE, pinball loss), plus the per-fold spread to see stability over time.
5. Do all preprocessing (scaling, Box-Cox, decomposition, feature selection) inside the fold on the training part only.
6. For hyperparameter selection, tune on rolling origins and report on a final untouched tail (nested rolling origin).

t:  0 ........ o0 ....... o1 ....... o2 ....... n
f0: [== train ==][gap][test h]
f1: [====== train ======][gap][test h]
f2: [========== train ==========][gap][test h]

Code

use std::ops::Range;

/// One fold: fit on `train`, predict `test`. Indices into the series.
#[derive(Debug, Clone, PartialEq)]
pub struct Fold { pub train: Range<usize>, pub test: Range<usize> }

/// Rolling-origin folds. `min_train`: size of the first training window;
/// `h`: forecast horizon (test length); `step`: how far the origin advances;
/// `gap`: samples dropped between train and test (leakage guard for features
/// that look `gap` steps back or labels that look ahead); `window`: Some(w) = sliding
/// window of the last w samples, None = expanding window.
pub fn rolling_origin(n: usize, min_train: usize, h: usize, step: usize,
                      gap: usize, window: Option<usize>) -> Vec<Fold> {
    let mut folds = Vec::new();
    let mut origin = min_train;                  // train = ..origin
    while origin + gap + h <= n {
        let start = window.map_or(0, |w| origin.saturating_sub(w));
        folds.push(Fold { train: start..origin, test: origin + gap..origin + gap + h });
        origin += step;
    }
    folds
}

/// Mean absolute error per fold and mean over folds. `forecast(train, h)` must
/// only look at `train`; it returns h predictions.
pub fn evaluate<F: Fn(&[f64], usize) -> Vec<f64>>(y: &[f64], folds: &[Fold], forecast: F) -> (Vec<f64>, f64) {
    let maes: Vec<f64> = folds.iter().map(|f| {
        // cut the slice so the model physically cannot see the future
        let pred = forecast(&y[f.train.clone()], f.test.end - f.train.end);
        let skip = f.test.start - f.train.end;   // predictions for the gap are discarded
        let truth = &y[f.test.clone()];
        pred[skip..].iter().zip(truth).map(|(p, t)| (p - t).abs()).sum::<f64>() / truth.len() as f64
    }).collect();
    let mean = maes.iter().sum::<f64>() / maes.len() as f64;
    (maes, mean)
}

fn naive(train: &[f64], h: usize) -> Vec<f64> { vec![*train.last().unwrap(); h] }
fn seasonal_naive(m: usize) -> impl Fn(&[f64], usize) -> Vec<f64> {
    move |train, h| (0..h).map(|k| train[train.len() - m + k % m]).collect()
}

fn main() {
    let m = 7;
    let mut x: u64 = 42;
    let mut noise = || { x = x.wrapping_mul(6364136223846793005).wrapping_add(1442695040888963407);
                         ((x >> 33) as f64 / (1u64 << 31) as f64 - 0.5) * 2.0 };
    let y: Vec<f64> = (0..200).map(|t| 50.0 + 0.1 * t as f64
        + 8.0 * (2.0 * std::f64::consts::PI * t as f64 / m as f64).sin() + noise()).collect();

    let folds = rolling_origin(y.len(), 100, 14, 14, 0, None);
    assert_eq!(folds.len(), 7);
    for f in &folds { assert!(f.train.end <= f.test.start); }               // no look-ahead
    for w in folds.windows(2) { assert_eq!(w[0].test.end, w[1].test.start); } // tests tile the tail
    let g = rolling_origin(y.len(), 100, 14, 14, 3, Some(60));
    assert!(g.iter().all(|f| f.test.start - f.train.end == 3 && f.train.len() <= 60));

    let (_, mae_n) = evaluate(&y, &folds, naive);
    let (_, mae_s) = evaluate(&y, &folds, seasonal_naive(m));
    println!("naive {mae_n:.2}  seasonal-naive {mae_s:.2}");
    assert!(mae_s < mae_n);
}

The forecaster receives only the training slice, so look-ahead is impossible by construction. Predictions for the gap are generated and discarded.

Complexity and constants

• Number of folds: floor((n - min_train - gap - h) / step) + 1; the example gives 7 folds for n=200, min_train=100, h=14, step=14.
• Time: k x (fit + predict). Expanding window makes fit cost grow with the origin (linear-time models: about k x n/2 on average); a sliding window makes it constant per fold.
• Memory: folds are index ranges (two usize pairs each); the data is never copied if the forecaster takes a slice.
• Statistical cost: overlapping tests (step < h) correlate fold errors, so the effective number of independent folds is closer to (n - min_train) / h than to k. Do not treat k fold errors as independent for confidence intervals (bootstrap with blocks).

Efficiency

• The bound is k model fits. Biggest lever: fewer folds. Raise step (to h) or evaluate a fixed number of evenly spaced origins; cost falls proportionally and the estimate usually changes little once folds cover several seasons.
• Refit less than you evaluate: fit every few origins and only update state in between (cheap for recursive models such as exponential smoothing, where one update is O(1)). This matches a production schedule that does not retrain per point.
• Warm-start each fold from the previous fold's parameters or weights; for neural models a few epochs of fine-tuning replace a full retrain.
• Cap training length with a sliding window: constant cost per fold, and old data often carries little weight anyway.
• Fold-level parallelism is free: folds are independent, so run them on separate cores; or share one fit across all horizons (one fit per origin, h predictions).
• Cheap screening: use few folds and a short window to discard bad candidates, and spend the full evaluation only on survivors (successive halving fits this directly).
• Expensive deep models (deep forecasting) make k fits the dominant cost, so a statistical baseline evaluated on the same folds is the cheap sanity check.

Implementation notes

• Return error per horizon step: errors[fold][step]; accuracy usually decays with horizon and a mean hides it.
• Use a scale-free metric across series (MASE with the in-sample seasonal-naive scale computed on the training slice, or sMAPE) when aggregating many series (evaluation metrics).
• Always include naive and seasonal-naive on the same folds; a model that does not beat them is not worth deploying.
• Many series: do the split once per series on a shared calendar so folds line up, then aggregate by origin date.
• Exogenous regressors must be the values known at forecast time (forecasted weather, not realized), or the backtest leaks.

Pitfalls

• Shuffled K-fold, or train/test split after random sampling of windows: the model is tested on neighbours of its training points and the error is optimistic. Bergmeir, Hyndman and Koo (2018) show ordinary K-fold can still be valid for purely autoregressive models with uncorrelated errors, but not in general, so default to rolling origin.
• Feature or label windows that cross the split (a 30-step rolling mean, a label defined over the next 5 steps): set gap to the longest such span (purged/embargoed splits in López de Prado's terms).
• Fitting scalers, decompositions or feature selection on the full series before splitting leaks future statistics; a decomposition over the test period also leaks (STL).
• Reporting only the best of many tried configurations: the minimum over many noisy fold means is biased low; confirm on an untouched tail.
• Too few folds or a single origin: the error estimate varies strongly with where the origin falls (holiday, regime shift).
• Non-stationarity: a long expanding window can hide a recent drift that a sliding window reveals; compare both.

Variants and alternatives

In production

Demand and capacity forecasting backtests; financial machine learning, where purged K-fold with an embargo removes leakage from overlapping labels (López de Prado, ch. 7); scikit-learn's TimeSeriesSplit for any time-ordered tabular model; model-selection loops in forecasting libraries (the Python and R packages below), and the standard protocol for forecasting competitions and benchmark papers that re-estimate models at successive origins.

Implementations

Sources

• Rob J. Hyndman, George Athanasopoulos, Forecasting: Principles and Practice, 3rd ed., OTexts 2021, section 5.10 (time series cross-validation). book
• Leonard J. Tashman, Out-of-sample tests of forecasting accuracy: an analysis and review, Int. J. Forecasting 16(4), 2000. paper
• Christoph Bergmeir, Jose M. Benitez, On the use of cross-validation for time series predictor evaluation, Information Sciences 191, 2012. paper
• Christoph Bergmeir, Rob J. Hyndman, Bonsoo Koo, A note on the validity of cross-validation for evaluating autoregressive time series prediction, Computational Statistics & Data Analysis 120, 2018. paper
• Vitor Cerqueira, Luis Torgo, Igor Mozetic, Evaluating time series forecasting models: an empirical study on performance estimation methods, Machine Learning 109, 2020. paper
• Jeff Racine, Consistent cross-validatory model-selection for dependent data: hv-block cross-validation, J. Econometrics 99(1), 2000. paper
• Marcos Lopez de Prado, Advances in Financial Machine Learning, Wiley 2018, ch. 7 (purged K-fold and embargo).