Module - Cross Validation

# Import to be able to import python package from src
import sys
sys.path.insert(0, '../src')

import numpy as np
import pandas as pd

import ontime as on
from ontime.module.processing.cross_validation import cross_validation, evaluate_cross_validation
from ontime.module.benchmarking.benchmark_metric import BenchmarkMetric
from darts.metrics import mae

---

Load data

For this notebook, we generate a simple synthetic time series so the notebook runs quickly and without external downloads.

index = pd.date_range(start='2020-01-01', periods=120, freq='D')
values = np.arange(len(index), dtype=float) + np.random.normal(scale=0.5, size=len(index))
df = pd.DataFrame({'consumption': values}, index=index)
ts = on.TimeSeries.from_dataframe(df)
ts.plot()

---

Why cross-validation for time series?

Unlike i.i.d. data, time series can’t be split randomly: shuffling would leak future information into the training set. ontime.module.processing.cross_validation implements the standard strategies used to evaluate forecasting models while respecting temporal order:

• expanding window (anchored, growing train set) — setting horizon=1 gives walk-forward validation, while horizon > 1 evaluates multi-step forecasts.

• sliding window (fixed-size, rolling train set).

• blocked cross-validation (non-overlapping contiguous blocks).

All strategies also support a gap parameter to leave a buffer between train and test (purged / embargoed cross-validation), which helps avoid leakage caused by autocorrelation.

Expanding window (walk-forward, horizon=1)

folds = cross_validation(
    ts, n_splits=5, strategy='expanding', initial_train_size=80, horizon=1
)
for train, test in folds:
    print(len(train), len(test))

80 1
81 1
82 1
83 1
84 1

Expanding window, multi-horizon (horizon > 1)

folds = cross_validation(
    ts, n_splits=5, strategy='expanding', initial_train_size=80, horizon=5
)
for train, test in folds:
    print(len(train), len(test))

80 5
85 5
90 5
95 5
100 5

Sliding window

folds = cross_validation(
    ts, n_splits=5, strategy='sliding', initial_train_size=40, horizon=5
)
for train, test in folds:
    print(len(train), len(test))

40 5
40 5
40 5
40 5
40 5

Blocked cross-validation

folds = cross_validation(ts, n_splits=5, strategy='blocked')
for train, test in folds:
    print(len(train), len(test))

20 20
20 20
20 20
20 20
20 20

Purged / embargoed cross-validation (gap)

folds = cross_validation(
    ts, n_splits=5, strategy='expanding', initial_train_size=80, horizon=1, gap=3
)
train, test = folds[0]
print(train.pd_dataframe().index[-1], '->', test.pd_dataframe().index[0])

2020-03-20 00:00:00 -> 2020-03-24 00:00:00

---

Evaluating a model across folds

evaluate_cross_validation fits (or refits) a model on the train set of each fold, predicts the test horizon, and aggregates a metric across folds.

from darts.models import NaiveDrift

model = on.Model(NaiveDrift())
metric = BenchmarkMetric(name='mae', metric_function=mae)

results = evaluate_cross_validation(
    model, ts, metrics=metric, n_splits=5, strategy='expanding',
    initial_train_size=80, horizon=5,
)
print(results['folds'])
print(results['mean'])
print(results['std'])

{'mae': [0.37643823540291804, 0.34522330711362487, 0.46193333443947326, 0.46227172007183415, 1.1057205920204525]}
{'mae': 0.5503174378096605}
{'mae': 0.28154361698443287}