Module - Benchmarking

Ontime provides a Benchmark class that can be used to run a number of prediction models on a number of datasets.

from ontime.module.benchmarking import BenchmarkDataset, BenchmarkMetric, BenchmarkModelConfig, Benchmark
import ontime as on

Initialization

A Benchmark instance can be initialized with a list of datasets, models and metrics to run through. When invoking run(), it will train (if needed) and test every dataset on every model, and compute every metric on the predicted data.

Preparing datasets

Datasets submitted to a Benchmark must be of type TimeSeries, wrapped into BenchmarkDataset. BenchmarkDataset allows to give datasets a name, give training and test splits, define the time series components to use as target, define normalization scaling to use, and define how data will be split to perform a rolling evaluation. While BenchmarkDataset contains information about training data, it is designed primarly to have a common test dataset, and therefore to allow an unified evaluation.

More specifically, BenchmarkDataset defines :

• test proportion to split the whole given dataset time series, e.g. 0.3 for using 30% of the whole series as test set :

• input, target, gap and stride length to define how test data will be split for creating test samples, as well as target columns to define which columns/components are used as target, i.e. the columns/components that has to be predicted

The dataset given to the wrapper can be instantiated or not, allowing to avoid potential memory issues. In the latter case, a processing function can be given so that time series can be pre-processed once loaded.

from ontime.module.datasets.dataset import Dataset
from darts.utils.missing_values import fill_missing_values # for filling missing values in the time series (for models that don't handle missing values)
from sklearn.preprocessing import StandardScaler

datasets = [
    # loaded time series, with standard normalization
    BenchmarkDataset(on.TimeSeries.from_darts(fill_missing_values(Dataset.TemperatureDataset.load())),
                     input_length=96,
                     target_length=24,
                     gap=0,
                     stride=96,
                     name="Daily temperature",
                     scaler_type=StandardScaler
    ),
    # unloaded time series, without normalization
    BenchmarkDataset(Dataset.ETTh1Dataset,
                     input_length=336,
                     gap=0,
                     stride=72,
                     target_length=72,
                     name = "ETTh1",
                     target_columns=["OT"],
                     processing_fn=lambda ts: ts[:10000])
]

Preparing models

Benchmark models must be given in a BenchmarkModelConfig class object, and their class must implement ontime AbstractModel interface. The BenchmarkModelConfig class contains attribute that allows to later instantiate the model with the desired configuration. It requires the following parameters :

• model_name : the name of the model, for result logging purpose,

• model_class : the class of the model that implements AbstractModel interface,

• zero_shot_only : either if the model must be evaluated in zero shot learning setting only.

  • if True, the model is not trained, and the evaluation is done on the test set. It is used for models that already has trained weights, available through checkpoints, or for some models from darts, where predictions are directly made using the fitted data as input (such as ARIMA),

  • if False, the model is trained on the entire given training set. Once trained, the model is evaluated using the learnt weights.

• is_univariate: whether the model is univariate (only handles univariate time series) or not.

• test_batch_size : function that takes a BenchmarkDataset and returns the test batch size for this model. Default to a function returning 32.

• static_model_params : the static parameters to give to the model class for instantiating it. This parameters can be defined when instantiating the BenchmarkModelConfig object.

• dynamic_model_params : the dynamic parameters to give to the model class for instantiating it. This parameters can only be known when the dataset on which the model is trained is known. Therefore, a callable object that take a BenchmarkDataset must be given.

• dynamic_fit_model_params : the additional dynamic parameters to give to the model fit method during the benchmark run. This parameters can only be known when the dataset on which the model is trained is known. Therefore, a callable object that take a BenchmarkDataset must be given.

• dynamic_predict_model_params : the additional dynamic parameters to give to the model predict method during the benchmark run.. This parameters can only be known when the dataset on which the model is trained is known. Therefore, a callable object that take a BenchmarkDataset must be given.

from ontime import Model
from darts.models import ExponentialSmoothing, TCNModel

# torch related parameters
pl_trainer_kwargs = {
    "accelerator": "cpu",
    "enable_progress_bar": False
    }

# dynamic parameters callback
input_length_param = lambda ds: ds.input_length
target_length_param = lambda ds: ds.target_length

model_configs = [
    BenchmarkModelConfig("ExponentialSmoothing", model_class=Model, is_univariate=True, zero_shot_only=True, test_batch_size=lambda ds: 10,
                         static_model_params={"wrapped_model" : ExponentialSmoothing()}),
    BenchmarkModelConfig("Temporal Convolutional Network", model_class=Model, zero_shot_only=False,
                         static_model_params={"wrapped_model":TCNModel, "n_epochs":2, "pl_trainer_kwargs":pl_trainer_kwargs},
                         dynamic_model_params={"input_chunk_length":input_length_param, "output_chunk_length":target_length_param})
]

Preparing metrics

Metrics must be given to the BenchmarkMetric constructor. If the function can’t be invoked as is in BenchmarkMetric’s implementation, a child class can be written and submitted.

import darts.metrics

metrics = [
   BenchmarkMetric(name="MAE", metric_function=darts.metrics.metrics.mae),
   BenchmarkMetric(name="sMAPE", metric_function=darts.metrics.metrics.smape),
   BenchmarkMetric(name="MASE", metric_function=darts.metrics.metrics.mase)
]

Creating and running a Benchmark

The benchmark is created trough the Benchmark class, to which datasets, model configurations and metrics must be provided.

In addition, proportions of the train set, as floats, can be provided to train and evaluate models in varied few shot learning settings. E.g. by setting few_shot_proportions to `[1.0, 0.8, 0.2, 0.0], one full-shot learning (100% of training data), two few-shot learning (80 and 20% of training data) and one zero-shot learning (no training data) training and evaluation will be performed.

Lastly, a string directory path can be passed in order to create a folder in which intermediate results will be saved. By default, the directory path is “/benchmark_results”.

benchmark = Benchmark(datasets=datasets,
                      model_configs=model_configs,
                      metrics=metrics,
                      few_shot_proportions=[0.4, 0.8, 1.0])

Datasets, models and metrics can also be added after instanciation, using respectively add_dataset(), add_model_config() and add_metric() methods.

benchmark.add_dataset(BenchmarkDataset(Dataset.ETTh2Dataset.load()[:5000], input_length=336, gap=0, stride=72, target_length=72, name = "ETTh2", target_columns=["OT"]))

Once the models and datasets have been added, the run() method will train instances of all the models on all the datasets individually and compute metrics. The logging level can be chosen to show less or more information about the benchmark execution in the console. A run name can be given in order to create the run directory where intermediate results will be saved. By default, it will creates a directory named “benchmark_time”. The save_all_predictions argument allows to save all the predictions made by the model, in case we would need them to compute other metrics afterwards, avoiding to launch a full benchmark again.

benchmark.run(logging_level="debug", run_name="benchmark_doc_example", save_all_predictions=True)

Visualizing results

The benchmark automatically stores measures and metrics computed during the run, available through class attributes.

Measures and metrics

To view the results, you can call get_report() and print the returned value

print(benchmark.get_report())

Dataset: Daily temperature

nb features                1
target column              ['Daily minimum temperatures']
training set size          2920
test set size              732
validation set proportion  0.2

Results:

+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+
| Model                          | Few-shot %   |    | Success   |    |   evaluation |   inference |   training |    |   MAE |   MASE |   sMAPE |
+================================+==============+====+===========+====+==============+=============+============+====+=======+========+=========+
| ExponentialSmoothing           | 0%           |    | ✓         |    |         0.22 |        0.03 |       0    |    | 2.279 |  1.194 |  22.914 |
+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+
| Temporal Convolutional Network | 40%          |    | ✓         |    |         0.1  |        0.02 |       1.21 |    | 4.537 |  2.32  |  48.27  |
+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+
| Temporal Convolutional Network | 80%          |    | ✓         |    |         0.07 |        0.03 |       2.55 |    | 2.323 |  1.208 |  23.354 |
+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+
| Temporal Convolutional Network | 100%         |    | ✓         |    |         0.08 |        0.03 |       3.4  |    | 2.139 |  1.11  |  21.712 |
+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+

Dataset: ETTh1

nb features                7
target column              ['OT']
training set size          7999
test set size              2001
validation set proportion  0.2

Results:

+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+
| Model                          | Few-shot %   |    | Success   |    |   evaluation |   inference |   training |    |   MAE |   MASE |   sMAPE |
+================================+==============+====+===========+====+==============+=============+============+====+=======+========+=========+
| ExponentialSmoothing           | 0%           |    | ✓         |    |         1.34 |        0.33 |       0    |    | 2.24  |  3.564 |  12.377 |
+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+
| Temporal Convolutional Network | 40%          |    | ✓         |    |         1.45 |        0.03 |       5.39 |    | 3.842 |  5.992 |  21.407 |
+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+
| Temporal Convolutional Network | 80%          |    | ✓         |    |         1.21 |        0.02 |      12.16 |    | 2.937 |  4.639 |  15.888 |
+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+
| Temporal Convolutional Network | 100%         |    | ✓         |    |         1.28 |        0.02 |      15.55 |    | 2.575 |  4.066 |  13.901 |
+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+

Dataset: ETTh2

nb features                7
target column              ['OT']
training set size          3999
test set size              1001
validation set proportion  0.2

Results:

+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+--------+--------+---------+
| Model                          | Few-shot %   |    | Success   |    |   evaluation |   inference |   training |    |    MAE |   MASE |   sMAPE |
+================================+==============+====+===========+====+==============+=============+============+====+========+========+=========+
| ExponentialSmoothing           | 0%           |    | ✓         |    |         0.69 |        0.38 |       0    |    | 29.405 | 50.202 | 120.533 |
+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+--------+--------+---------+
| Temporal Convolutional Network | 40%          |    | ✓         |    |         0.08 |        0.02 |       2.29 |    | 17.604 | 29.92  |  80.671 |
+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+--------+--------+---------+
| Temporal Convolutional Network | 80%          |    | ✓         |    |         0.08 |        0.02 |       5.79 |    | 21.682 | 36.808 |  90.388 |
+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+--------+--------+---------+
| Temporal Convolutional Network | 100%         |    | ✓         |    |         0.11 |        0.03 |       7.71 |    | 14.334 | 24.359 |  71.043 |
+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+--------+--------+---------+

You can also get results as dataframes by calling get_report_df(). The results are then returned as a dataframe with model names as columns, dataset names as main rows, and measure as sub rows.

df_1, df_2 = benchmark.get_report_dfs()
df_1

	Daily temperature	ETTh1	ETTh2
Characteristic
nb features	1	7	7
target column	[Daily minimum temperatures]	[OT]	[OT]
training set size	2920	7999	3999
test set size	732	2001	1001
validation set proportion	0.2	0.2	0.2

df_2

			Daily temperature	ETTh1	ETTh2
Model	Few-shot proportion	Metric/Time
ExponentialSmoothing	0.0%	training	0.000000	0.000000	0.000000
		evaluation	0.216740	1.336384	0.686351
		inference	0.025195	0.328040	0.375670
		MAE	2.279000	2.239636	29.404520
		sMAPE	22.913583	12.376649	120.533054
		MASE	1.194152	3.564028	50.201920
Temporal Convolutional Network	40.0%	training	1.211548	5.386545	2.287528
		evaluation	0.095372	1.452384	0.084936
		inference	0.023723	0.025015	0.022550
		MAE	4.537101	3.842499	17.603871
		sMAPE	48.269516	21.407110	80.670535
		MASE	2.319506	5.991686	29.920213
	80.0%	training	2.549360	12.163323	5.790170
		evaluation	0.066923	1.214023	0.083161
		inference	0.027643	0.023938	0.022326
		MAE	2.323460	2.937134	21.682443
		sMAPE	23.353536	15.888110	90.387898
		MASE	1.208343	4.639171	36.807820
	100.0%	training	3.395739	15.548726	7.707017
		evaluation	0.080246	1.277790	0.112053
		inference	0.031607	0.022760	0.026817
		MAE	2.138831	2.575154	14.333950
		sMAPE	21.712217	13.900607	71.043355
		MASE	1.109780	4.066479	24.359035

Plotting

By default (argument nb_predictions of benchmark.run() method), the benchmark will generate a prediction for one random input sample of each dataset with each model. The predictions, along input and target series, are stored in a dictionnary and can be retrieved by calling benchmark.get_predictions(). The predictions can be plotted using the Ontime plotting module.

predictions = benchmark.get_predictions()

input = predictions['inputs']['Daily temperature'][0].rename({'Daily minimum temperatures': 'input'})
target = predictions['targets']['Daily temperature'][0].rename({'Daily minimum temperatures': 'target'})
prediction = predictions['predictions']['Daily temperature']['Temporal Convolutional Network'][1.0][0].rename({'Daily minimum temperatures': 'prediction'})

(on.Plot()
    .add(on.marks.line, input)
    .add(on.marks.line, target)
    .add(on.marks.line, prediction, type='dashed')
    .properties(width=600, height=200)
    .show()
)

prediction = predictions['predictions']['Daily temperature']['ExponentialSmoothing'][0.0][0].rename({'Daily minimum temperatures': 'prediction'})

(on.Plot()
    .add(on.marks.line, input)
    .add(on.marks.line, target)
    .add(on.marks.line, prediction, type='dashed')
    .properties(width=600, height=200)
    .show()
)

input = predictions['inputs']['ETTh1'][0].univariate_component(0)[220:].rename({'HUFL': 'input'})
target = predictions['targets']['ETTh1'][0].univariate_component(0).rename({'HUFL': 'target'})
prediction = predictions['predictions']['ETTh1']['ExponentialSmoothing'][0.0][0].univariate_component(0).rename({'HUFL': 'prediction'})

(on.Plot()
    .add(on.marks.line, input)
    .add(on.marks.line, target)
    .add(on.marks.line, prediction, type='dashed')
    .properties(width=600, height=200)
    .show()
)

Loading back saved results

You can load the results and models predictions back in python if needed, using pickle and/or json libraries.

import pickle, json

# load saved data
with open("benchmark_results/benchmark_doc_example/predictions.pkl", "rb") as f:
    loaded_preds = pickle.load(f)

with open("benchmark_results/benchmark_doc_example/results.json", "r") as f:
    loaded_results = json.load(f)

with open("benchmark_results/benchmark_doc_example/all_predictions/Temporal Convolutional Network_ETTh1_0.4.pkl", "rb") as f:
    some_preds = pickle.load(f)

Benchmark.get_results_df(loaded_results)

			Daily temperature	ETTh1	ETTh2
Model	Few-shot proportion	Metric/Time
ExponentialSmoothing	0.0%	training	0.000000	0.000000	0.000000
		evaluation	0.216740	1.336384	0.686351
		inference	0.025195	0.328040	0.375670
		MAE	2.279000	2.239636	29.404520
		sMAPE	22.913583	12.376649	120.533054
		MASE	1.194152	3.564028	50.201920
Temporal Convolutional Network	40.0%	training	1.211548	5.386545	2.287528
		evaluation	0.095372	1.452384	0.084936
		inference	0.023723	0.025015	0.022550
		MAE	4.537101	3.842499	17.603871
		sMAPE	48.269516	21.407110	80.670535
		MASE	2.319506	5.991686	29.920213
	80.0%	training	2.549360	12.163323	5.790170
		evaluation	0.066923	1.214023	0.083161
		inference	0.027643	0.023938	0.022326
		MAE	2.323460	2.937134	21.682443
		sMAPE	23.353536	15.888110	90.387898
		MASE	1.208343	4.639171	36.807820
	100.0%	training	3.395739	15.548726	7.707017
		evaluation	0.080246	1.277790	0.112053
		inference	0.031607	0.022760	0.026817
		MAE	2.138831	2.575154	14.333950
		sMAPE	21.712217	13.900607	71.043355
		MASE	1.109780	4.066479	24.359035

some_preds[0].plot()