{ "cells": [ { "cell_type": "markdown", "id": "550c22a8-085f-45f1-9af5-35896f670515", "metadata": {}, "source": [ "# Module - Benchmarking\n", "Ontime provides a `Benchmark` class that can be used to run a number of prediction models on a number of datasets." ] }, { "cell_type": "code", "execution_count": 1, "id": "27e27d12-d338-47d3-8fd3-b31ff80ac57c", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:42:29.819630Z", "start_time": "2025-09-24T14:42:25.193779Z" } }, "outputs": [], "source": [ "from ontime.module.benchmarking import BenchmarkDataset, BenchmarkMetric, BenchmarkModelConfig, Benchmark\n", "import ontime as on" ] }, { "cell_type": "markdown", "id": "476a848a-77d7-4efc-840f-665a7cdc1857", "metadata": {}, "source": [ "## Initialization\n", "A `Benchmark` instance can be initialized with a list of datasets, models and metrics to run through. When invoking `run()`, it will train (if needed) and test every dataset on every model, and compute every metric on the predicted data.\n" ] }, { "cell_type": "markdown", "id": "f5641c4a", "metadata": {}, "source": [ "### Preparing datasets\n", "\n", "Datasets submitted to a `Benchmark` must be of type `TimeSeries`, wrapped into `BenchmarkDataset`. `BenchmarkDataset` allows to give datasets a name, give training and test splits, define the time series components to use as target, define normalization scaling to use, and define how data will be split to perform a rolling evaluation. While `BenchmarkDataset` contains information about training data, it is designed primarly to have a common test dataset, and therefore to allow an unified evaluation. \n", "\n", "More specifically, `BenchmarkDataset` defines :\n", "- test proportion to split the whole given dataset time series, e.g. 0.3 for using 30% of the whole series as test set : \n", "\"Times \n", "- input, target, gap and stride length to define how test data will be split for creating test samples, as well as target columns to define which columns/components are used as target, i.e. the columns/components that has to be predicted \n", "\"Times \"Times \n", "\n", "The dataset given to the wrapper can be instantiated or not, allowing to avoid potential memory issues. In the latter case, a processing function can be given so that time series can be pre-processed once loaded." ] }, { "cell_type": "code", "execution_count": 2, "id": "1177e73c", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:42:29.838252Z", "start_time": "2025-09-24T14:42:29.825650Z" } }, "outputs": [], "source": [ "from ontime.module.datasets.dataset import Dataset\n", "from darts.utils.missing_values import fill_missing_values # for filling missing values in the time series (for models that don't handle missing values)\n", "from sklearn.preprocessing import StandardScaler\n", "\n", "datasets = [\n", " # loaded time series, with standard normalization\n", " BenchmarkDataset(on.TimeSeries.from_darts(fill_missing_values(Dataset.TemperatureDataset.load())), \n", " input_length=96, \n", " target_length=24, \n", " gap=0, \n", " stride=96, \n", " name=\"Daily temperature\",\n", " scaler_type=StandardScaler\n", " ),\n", " # unloaded time series, without normalization\n", " BenchmarkDataset(Dataset.ETTh1Dataset, \n", " input_length=336, \n", " gap=0, \n", " stride=72, \n", " target_length=72, \n", " name = \"ETTh1\", \n", " target_columns=[\"OT\"], \n", " processing_fn=lambda ts: ts[:10000])\n", "]" ] }, { "cell_type": "markdown", "id": "26a88e291c85081", "metadata": {}, "source": [ "### Preparing models\n", "\n", "Benchmark models must be given in a `BenchmarkModelConfig` class object, and their class must implement ontime `AbstractModel` interface. The `BenchmarkModelConfig` class contains attribute that allows to later instantiate the model with the desired configuration.\n", "It requires the following parameters :\n", "- `model_name` : the name of the model, for result logging purpose,\n", "- `model_class` : the class of the model that implements `AbstractModel` interface,\n", "- `zero_shot_only` : either if the model must be evaluated in zero shot learning setting only.\n", " - if `True`, the model is not trained, and the evaluation is done on the test set. It is used for models that already has trained weights, available through checkpoints, or for some models from darts, where predictions are directly made using the fitted data as input (such as ARIMA),\n", " - if `False`, the model is trained on the entire given training set. Once trained, the model is evaluated using the learnt weights.\n", "- `is_univariate`: whether the model is univariate (only handles univariate time series) or not.\n", "- `test_batch_size` : function that takes a `BenchmarkDataset` and returns the test batch size for this model. Default to a function returning 32.\n", "- `static_model_params` : the static parameters to give to the model class for instantiating it. This parameters can be defined when instantiating the `BenchmarkModelConfig` object.\n", "- `dynamic_model_params` : the dynamic parameters to give to the model class for instantiating it. This parameters can only be known when the dataset on which the model is trained is known. Therefore, a callable object that take a `BenchmarkDataset` must be given.\n", "- `dynamic_fit_model_params` : the additional dynamic parameters to give to the model fit method during the benchmark run. This parameters can only be known when the dataset on which the model is trained is known. Therefore, a callable object that take a `BenchmarkDataset` must be given.\n", "- `dynamic_predict_model_params` : the additional dynamic parameters to give to the model predict method during the benchmark run.. This parameters can only be known when the dataset on which the model is trained is known. Therefore, a callable object that take a `BenchmarkDataset` must be given." ] }, { "cell_type": "code", "execution_count": 3, "id": "ad88a5ec8117a145", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:47:14.956175Z", "start_time": "2025-09-24T14:47:14.951800Z" } }, "outputs": [], "source": [ "from ontime import Model\n", "from darts.models import ExponentialSmoothing, TCNModel\n", "\n", "# torch related parameters\n", "pl_trainer_kwargs = {\n", " \"accelerator\": \"cpu\",\n", " \"enable_progress_bar\": False\n", " }\n", "\n", "# dynamic parameters callback\n", "input_length_param = lambda ds: ds.input_length\n", "target_length_param = lambda ds: ds.target_length\n", "\n", "model_configs = [\n", " BenchmarkModelConfig(\"ExponentialSmoothing\", model_class=Model, is_univariate=True, zero_shot_only=True, test_batch_size=lambda ds: 10,\n", " static_model_params={\"wrapped_model\" : ExponentialSmoothing()}),\n", " BenchmarkModelConfig(\"Temporal Convolutional Network\", model_class=Model, zero_shot_only=False,\n", " static_model_params={\"wrapped_model\":TCNModel, \"n_epochs\":2, \"pl_trainer_kwargs\":pl_trainer_kwargs},\n", " dynamic_model_params={\"input_chunk_length\":input_length_param, \"output_chunk_length\":target_length_param})\n", "]" ] }, { "cell_type": "markdown", "id": "fd484f03-399d-4e4e-9b7c-2462e6accbb9", "metadata": {}, "source": [ "### Preparing metrics\n", "Metrics must be given to the `BenchmarkMetric` constructor. If the function can't be invoked as is in `BenchmarkMetric`'s implementation, a child class can be written and submitted." ] }, { "cell_type": "code", "execution_count": 4, "id": "3bd6e05b-d86a-484a-bd28-7559b640c20f", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:47:16.118191Z", "start_time": "2025-09-24T14:47:16.116208Z" } }, "outputs": [], "source": [ "import darts.metrics\n", "\n", "metrics = [\n", " BenchmarkMetric(name=\"MAE\", metric_function=darts.metrics.metrics.mae),\n", " BenchmarkMetric(name=\"sMAPE\", metric_function=darts.metrics.metrics.smape),\n", " BenchmarkMetric(name=\"MASE\", metric_function=darts.metrics.metrics.mase)\n", "]" ] }, { "cell_type": "markdown", "id": "f1ce9193-2bd1-4859-b5c5-17a48cb4e0fa", "metadata": {}, "source": [ "## Creating and running a Benchmark\n", "\n", "The benchmark is created trough the `Benchmark` class, to which datasets, model configurations and metrics must be provided. \n", "In addition, proportions of the train set, as floats, can be provided to train and evaluate models in varied few shot learning settings. E.g. by setting `few_shot_proportions` to `[1.0, 0.8, 0.2, 0.0], one full-shot learning (100% of training data), two few-shot learning (80 and 20% of training data) and one zero-shot learning (no training data) training and evaluation will be performed.\n", "\"Times \n", "Lastly, a string directory path can be passed in order to create a folder in which intermediate results will be saved. By default, the directory path is \"/benchmark_results\"." ] }, { "cell_type": "code", "execution_count": 5, "id": "b99abb03-c3ee-4e19-87a7-c86e1b418a6b", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:47:16.720044Z", "start_time": "2025-09-24T14:47:16.717473Z" } }, "outputs": [], "source": [ "benchmark = Benchmark(datasets=datasets,\n", " model_configs=model_configs, \n", " metrics=metrics,\n", " few_shot_proportions=[0.4, 0.8, 1.0])" ] }, { "cell_type": "markdown", "id": "e7b7a83a-e878-4fbf-818a-3c70f73bcb05", "metadata": {}, "source": [ "Datasets, models and metrics can also be added after instanciation, using respectively `add_dataset()`, `add_model_config()` and `add_metric()` methods." ] }, { "cell_type": "code", "execution_count": 6, "id": "005d3bae-1fdb-4edd-91ba-0df918fae5fe", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:47:17.066278Z", "start_time": "2025-09-24T14:47:17.042134Z" } }, "outputs": [], "source": [ "benchmark.add_dataset(BenchmarkDataset(Dataset.ETTh2Dataset.load()[:5000], input_length=336, gap=0, stride=72, target_length=72, name = \"ETTh2\", target_columns=[\"OT\"]))" ] }, { "cell_type": "markdown", "id": "b86ec250-db58-4c10-b4e3-14c2b23bcf70", "metadata": {}, "source": [ "Once the models and datasets have been added, the `run()` method will train instances of all the models on all the datasets individually and compute metrics. The logging level can be chosen to show less or more information about the benchmark execution in the console. A run name can be given in order to create the run directory where intermediate results will be saved. By default, it will creates a directory named \"benchmark_*time*\". The `save_all_predictions` argument allows to save all the predictions made by the model, in case we would need them to compute other metrics afterwards, avoiding to launch a full benchmark again." ] }, { "cell_type": "code", "execution_count": null, "id": "e96d8811-efd0-4cb0-bf7c-a03364e05911", "metadata": {}, "outputs": [], "source": [ "benchmark.run(logging_level=\"debug\", run_name=\"benchmark_doc_example\", save_all_predictions=True)" ] }, { "cell_type": "markdown", "id": "04a22d7c", "metadata": {}, "source": [ "## Visualizing results\n", "\n", "The benchmark automatically stores measures and metrics computed during the run, available through class attributes." ] }, { "cell_type": "markdown", "id": "d4038313-7cf4-480d-8d3c-fd030e100da3", "metadata": {}, "source": [ "### Measures and metrics\n", "To view the results, you can call `get_report()` and print the returned value" ] }, { "cell_type": "code", "execution_count": 8, "id": "35b9b732-bd80-426f-a8bb-134f06bf1a3b", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:48:07.080447Z", "start_time": "2025-09-24T14:48:07.076143Z" }, "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Dataset: Daily temperature\n", "\n", "nb features 1\n", "target column ['Daily minimum temperatures']\n", "training set size 2920\n", "test set size 732\n", "validation set proportion 0.2\n", "\n", "Results:\n", "\n", "+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+\n", "| Model | Few-shot % | | Success | | evaluation | inference | training | | MAE | MASE | sMAPE |\n", "+================================+==============+====+===========+====+==============+=============+============+====+=======+========+=========+\n", "| ExponentialSmoothing | 0% | | ✓ | | 0.22 | 0.03 | 0 | | 2.279 | 1.194 | 22.914 |\n", "+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+\n", "| Temporal Convolutional Network | 40% | | ✓ | | 0.1 | 0.02 | 1.21 | | 4.537 | 2.32 | 48.27 |\n", "+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+\n", "| Temporal Convolutional Network | 80% | | ✓ | | 0.07 | 0.03 | 2.55 | | 2.323 | 1.208 | 23.354 |\n", "+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+\n", "| Temporal Convolutional Network | 100% | | ✓ | | 0.08 | 0.03 | 3.4 | | 2.139 | 1.11 | 21.712 |\n", "+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+\n", "\n", "Dataset: ETTh1\n", "\n", "nb features 7\n", "target column ['OT']\n", "training set size 7999\n", "test set size 2001\n", "validation set proportion 0.2\n", "\n", "Results:\n", "\n", "+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+\n", "| Model | Few-shot % | | Success | | evaluation | inference | training | | MAE | MASE | sMAPE |\n", "+================================+==============+====+===========+====+==============+=============+============+====+=======+========+=========+\n", "| ExponentialSmoothing | 0% | | ✓ | | 1.34 | 0.33 | 0 | | 2.24 | 3.564 | 12.377 |\n", "+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+\n", "| Temporal Convolutional Network | 40% | | ✓ | | 1.45 | 0.03 | 5.39 | | 3.842 | 5.992 | 21.407 |\n", "+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+\n", "| Temporal Convolutional Network | 80% | | ✓ | | 1.21 | 0.02 | 12.16 | | 2.937 | 4.639 | 15.888 |\n", "+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+\n", "| Temporal Convolutional Network | 100% | | ✓ | | 1.28 | 0.02 | 15.55 | | 2.575 | 4.066 | 13.901 |\n", "+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+-------+--------+---------+\n", "\n", "Dataset: ETTh2\n", "\n", "nb features 7\n", "target column ['OT']\n", "training set size 3999\n", "test set size 1001\n", "validation set proportion 0.2\n", "\n", "Results:\n", "\n", "+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+--------+--------+---------+\n", "| Model | Few-shot % | | Success | | evaluation | inference | training | | MAE | MASE | sMAPE |\n", "+================================+==============+====+===========+====+==============+=============+============+====+========+========+=========+\n", "| ExponentialSmoothing | 0% | | ✓ | | 0.69 | 0.38 | 0 | | 29.405 | 50.202 | 120.533 |\n", "+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+--------+--------+---------+\n", "| Temporal Convolutional Network | 40% | | ✓ | | 0.08 | 0.02 | 2.29 | | 17.604 | 29.92 | 80.671 |\n", "+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+--------+--------+---------+\n", "| Temporal Convolutional Network | 80% | | ✓ | | 0.08 | 0.02 | 5.79 | | 21.682 | 36.808 | 90.388 |\n", "+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+--------+--------+---------+\n", "| Temporal Convolutional Network | 100% | | ✓ | | 0.11 | 0.03 | 7.71 | | 14.334 | 24.359 | 71.043 |\n", "+--------------------------------+--------------+----+-----------+----+--------------+-------------+------------+----+--------+--------+---------+\n" ] } ], "source": [ "print(benchmark.get_report())" ] }, { "cell_type": "markdown", "id": "a0852085-874e-4583-9f91-1efcb486fbcb", "metadata": {}, "source": [ "You can also get results as dataframes by calling `get_report_df()`. The results are then returned as a dataframe with model names as columns, dataset names as main rows, and measure as sub rows." ] }, { "cell_type": "code", "execution_count": 9, "id": "687a0b79-ff02-4bde-aff7-6d6ccd5e7d40", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:48:29.372909Z", "start_time": "2025-09-24T14:48:29.353587Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Daily temperatureETTh1ETTh2
Characteristic
nb features177
target column[Daily minimum temperatures][OT][OT]
training set size292079993999
test set size73220011001
validation set proportion0.20.20.2
\n", "
" ], "text/plain": [ " Daily temperature ETTh1 ETTh2\n", "Characteristic \n", "nb features 1 7 7\n", "target column [Daily minimum temperatures] [OT] [OT]\n", "training set size 2920 7999 3999\n", "test set size 732 2001 1001\n", "validation set proportion 0.2 0.2 0.2" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_1, df_2 = benchmark.get_report_dfs()\n", "df_1" ] }, { "cell_type": "code", "execution_count": 10, "id": "a113a1a8-6988-4440-9afc-c4e9b5e6f898", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:48:29.711833Z", "start_time": "2025-09-24T14:48:29.705911Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Daily temperatureETTh1ETTh2
ModelFew-shot proportionMetric/Time
ExponentialSmoothing0.0%training0.0000000.0000000.000000
evaluation0.2167401.3363840.686351
inference0.0251950.3280400.375670
MAE2.2790002.23963629.404520
sMAPE22.91358312.376649120.533054
MASE1.1941523.56402850.201920
Temporal Convolutional Network40.0%training1.2115485.3865452.287528
evaluation0.0953721.4523840.084936
inference0.0237230.0250150.022550
MAE4.5371013.84249917.603871
sMAPE48.26951621.40711080.670535
MASE2.3195065.99168629.920213
80.0%training2.54936012.1633235.790170
evaluation0.0669231.2140230.083161
inference0.0276430.0239380.022326
MAE2.3234602.93713421.682443
sMAPE23.35353615.88811090.387898
MASE1.2083434.63917136.807820
100.0%training3.39573915.5487267.707017
evaluation0.0802461.2777900.112053
inference0.0316070.0227600.026817
MAE2.1388312.57515414.333950
sMAPE21.71221713.90060771.043355
MASE1.1097804.06647924.359035
\n", "
" ], "text/plain": [ " Daily temperature \\\n", "Model Few-shot proportion Metric/Time \n", "ExponentialSmoothing 0.0% training 0.000000 \n", " evaluation 0.216740 \n", " inference 0.025195 \n", " MAE 2.279000 \n", " sMAPE 22.913583 \n", " MASE 1.194152 \n", "Temporal Convolutional Network 40.0% training 1.211548 \n", " evaluation 0.095372 \n", " inference 0.023723 \n", " MAE 4.537101 \n", " sMAPE 48.269516 \n", " MASE 2.319506 \n", " 80.0% training 2.549360 \n", " evaluation 0.066923 \n", " inference 0.027643 \n", " MAE 2.323460 \n", " sMAPE 23.353536 \n", " MASE 1.208343 \n", " 100.0% training 3.395739 \n", " evaluation 0.080246 \n", " inference 0.031607 \n", " MAE 2.138831 \n", " sMAPE 21.712217 \n", " MASE 1.109780 \n", "\n", " ETTh1 \\\n", "Model Few-shot proportion Metric/Time \n", "ExponentialSmoothing 0.0% training 0.000000 \n", " evaluation 1.336384 \n", " inference 0.328040 \n", " MAE 2.239636 \n", " sMAPE 12.376649 \n", " MASE 3.564028 \n", "Temporal Convolutional Network 40.0% training 5.386545 \n", " evaluation 1.452384 \n", " inference 0.025015 \n", " MAE 3.842499 \n", " sMAPE 21.407110 \n", " MASE 5.991686 \n", " 80.0% training 12.163323 \n", " evaluation 1.214023 \n", " inference 0.023938 \n", " MAE 2.937134 \n", " sMAPE 15.888110 \n", " MASE 4.639171 \n", " 100.0% training 15.548726 \n", " evaluation 1.277790 \n", " inference 0.022760 \n", " MAE 2.575154 \n", " sMAPE 13.900607 \n", " MASE 4.066479 \n", "\n", " ETTh2 \n", "Model Few-shot proportion Metric/Time \n", "ExponentialSmoothing 0.0% training 0.000000 \n", " evaluation 0.686351 \n", " inference 0.375670 \n", " MAE 29.404520 \n", " sMAPE 120.533054 \n", " MASE 50.201920 \n", "Temporal Convolutional Network 40.0% training 2.287528 \n", " evaluation 0.084936 \n", " inference 0.022550 \n", " MAE 17.603871 \n", " sMAPE 80.670535 \n", " MASE 29.920213 \n", " 80.0% training 5.790170 \n", " evaluation 0.083161 \n", " inference 0.022326 \n", " MAE 21.682443 \n", " sMAPE 90.387898 \n", " MASE 36.807820 \n", " 100.0% training 7.707017 \n", " evaluation 0.112053 \n", " inference 0.026817 \n", " MAE 14.333950 \n", " sMAPE 71.043355 \n", " MASE 24.359035 " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df_2" ] }, { "cell_type": "markdown", "id": "35d12a40", "metadata": {}, "source": [ "### Plotting\n", "\n", "By default (argument `nb_predictions` of `benchmark.run()` method), the benchmark will generate a prediction for one random input sample of each dataset with each model. The predictions, along input and target series, are stored in a dictionnary and can be retrieved by calling `benchmark.get_predictions()`. The predictions can be plotted using the Ontime plotting module." ] }, { "cell_type": "code", "execution_count": 11, "id": "f077a621", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:48:30.600880Z", "start_time": "2025-09-24T14:48:30.599043Z" } }, "outputs": [], "source": [ "predictions = benchmark.get_predictions()" ] }, { "cell_type": "code", "execution_count": 12, "id": "9e46a81e", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:48:31.181459Z", "start_time": "2025-09-24T14:48:31.173027Z" } }, "outputs": [], "source": [ "input = predictions['inputs']['Daily temperature'][0].rename({'Daily minimum temperatures': 'input'})\n", "target = predictions['targets']['Daily temperature'][0].rename({'Daily minimum temperatures': 'target'})\n", "prediction = predictions['predictions']['Daily temperature']['Temporal Convolutional Network'][1.0][0].rename({'Daily minimum temperatures': 'prediction'})" ] }, { "cell_type": "code", "execution_count": 13, "id": "da7935d5", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:48:33.499517Z", "start_time": "2025-09-24T14:48:31.500580Z" } }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "
\n", "" ], "text/plain": [ "alt.LayerChart(...)" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(on.Plot()\n", " .add(on.marks.line, input)\n", " .add(on.marks.line, target)\n", " .add(on.marks.line, prediction, type='dashed')\n", " .properties(width=600, height=200)\n", " .show()\n", ")" ] }, { "cell_type": "code", "execution_count": 14, "id": "ffecb8fb", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:48:33.521209Z", "start_time": "2025-09-24T14:48:33.518402Z" } }, "outputs": [], "source": [ "prediction = predictions['predictions']['Daily temperature']['ExponentialSmoothing'][0.0][0].rename({'Daily minimum temperatures': 'prediction'})" ] }, { "cell_type": "code", "execution_count": 15, "id": "e6c1b476", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:48:33.556838Z", "start_time": "2025-09-24T14:48:33.523823Z" } }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "
\n", "" ], "text/plain": [ "alt.LayerChart(...)" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(on.Plot()\n", " .add(on.marks.line, input)\n", " .add(on.marks.line, target)\n", " .add(on.marks.line, prediction, type='dashed')\n", " .properties(width=600, height=200)\n", " .show()\n", ")" ] }, { "cell_type": "code", "execution_count": 16, "id": "4bc66495", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:48:33.566891Z", "start_time": "2025-09-24T14:48:33.560199Z" } }, "outputs": [], "source": [ "input = predictions['inputs']['ETTh1'][0].univariate_component(0)[220:].rename({'HUFL': 'input'})\n", "target = predictions['targets']['ETTh1'][0].univariate_component(0).rename({'HUFL': 'target'})\n", "prediction = predictions['predictions']['ETTh1']['ExponentialSmoothing'][0.0][0].univariate_component(0).rename({'HUFL': 'prediction'})" ] }, { "cell_type": "code", "execution_count": 17, "id": "6312ee42", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:48:33.608809Z", "start_time": "2025-09-24T14:48:33.571046Z" } }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "
\n", "" ], "text/plain": [ "alt.LayerChart(...)" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(on.Plot()\n", " .add(on.marks.line, input)\n", " .add(on.marks.line, target)\n", " .add(on.marks.line, prediction, type='dashed')\n", " .properties(width=600, height=200)\n", " .show()\n", ")" ] }, { "cell_type": "markdown", "id": "f81cfef3", "metadata": {}, "source": [ "### Loading back saved results\n", "\n", "You can load the results and models predictions back in python if needed, using pickle and/or json libraries." ] }, { "cell_type": "code", "execution_count": 18, "id": "9beb22a0", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:48:33.620120Z", "start_time": "2025-09-24T14:48:33.612824Z" } }, "outputs": [], "source": [ "import pickle, json\n", "\n", "# load saved data\n", "with open(\"benchmark_results/benchmark_doc_example/predictions.pkl\", \"rb\") as f:\n", " loaded_preds = pickle.load(f)\n", "\n", "with open(\"benchmark_results/benchmark_doc_example/results.json\", \"r\") as f:\n", " loaded_results = json.load(f)\n", "\n", "with open(\"benchmark_results/benchmark_doc_example/all_predictions/Temporal Convolutional Network_ETTh1_0.4.pkl\", \"rb\") as f:\n", " some_preds = pickle.load(f)" ] }, { "cell_type": "code", "execution_count": 19, "id": "6f8ff5f4", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:48:33.885729Z", "start_time": "2025-09-24T14:48:33.879989Z" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Daily temperatureETTh1ETTh2
ModelFew-shot proportionMetric/Time
ExponentialSmoothing0.0%training0.0000000.0000000.000000
evaluation0.2167401.3363840.686351
inference0.0251950.3280400.375670
MAE2.2790002.23963629.404520
sMAPE22.91358312.376649120.533054
MASE1.1941523.56402850.201920
Temporal Convolutional Network40.0%training1.2115485.3865452.287528
evaluation0.0953721.4523840.084936
inference0.0237230.0250150.022550
MAE4.5371013.84249917.603871
sMAPE48.26951621.40711080.670535
MASE2.3195065.99168629.920213
80.0%training2.54936012.1633235.790170
evaluation0.0669231.2140230.083161
inference0.0276430.0239380.022326
MAE2.3234602.93713421.682443
sMAPE23.35353615.88811090.387898
MASE1.2083434.63917136.807820
100.0%training3.39573915.5487267.707017
evaluation0.0802461.2777900.112053
inference0.0316070.0227600.026817
MAE2.1388312.57515414.333950
sMAPE21.71221713.90060771.043355
MASE1.1097804.06647924.359035
\n", "
" ], "text/plain": [ " Daily temperature \\\n", "Model Few-shot proportion Metric/Time \n", "ExponentialSmoothing 0.0% training 0.000000 \n", " evaluation 0.216740 \n", " inference 0.025195 \n", " MAE 2.279000 \n", " sMAPE 22.913583 \n", " MASE 1.194152 \n", "Temporal Convolutional Network 40.0% training 1.211548 \n", " evaluation 0.095372 \n", " inference 0.023723 \n", " MAE 4.537101 \n", " sMAPE 48.269516 \n", " MASE 2.319506 \n", " 80.0% training 2.549360 \n", " evaluation 0.066923 \n", " inference 0.027643 \n", " MAE 2.323460 \n", " sMAPE 23.353536 \n", " MASE 1.208343 \n", " 100.0% training 3.395739 \n", " evaluation 0.080246 \n", " inference 0.031607 \n", " MAE 2.138831 \n", " sMAPE 21.712217 \n", " MASE 1.109780 \n", "\n", " ETTh1 \\\n", "Model Few-shot proportion Metric/Time \n", "ExponentialSmoothing 0.0% training 0.000000 \n", " evaluation 1.336384 \n", " inference 0.328040 \n", " MAE 2.239636 \n", " sMAPE 12.376649 \n", " MASE 3.564028 \n", "Temporal Convolutional Network 40.0% training 5.386545 \n", " evaluation 1.452384 \n", " inference 0.025015 \n", " MAE 3.842499 \n", " sMAPE 21.407110 \n", " MASE 5.991686 \n", " 80.0% training 12.163323 \n", " evaluation 1.214023 \n", " inference 0.023938 \n", " MAE 2.937134 \n", " sMAPE 15.888110 \n", " MASE 4.639171 \n", " 100.0% training 15.548726 \n", " evaluation 1.277790 \n", " inference 0.022760 \n", " MAE 2.575154 \n", " sMAPE 13.900607 \n", " MASE 4.066479 \n", "\n", " ETTh2 \n", "Model Few-shot proportion Metric/Time \n", "ExponentialSmoothing 0.0% training 0.000000 \n", " evaluation 0.686351 \n", " inference 0.375670 \n", " MAE 29.404520 \n", " sMAPE 120.533054 \n", " MASE 50.201920 \n", "Temporal Convolutional Network 40.0% training 2.287528 \n", " evaluation 0.084936 \n", " inference 0.022550 \n", " MAE 17.603871 \n", " sMAPE 80.670535 \n", " MASE 29.920213 \n", " 80.0% training 5.790170 \n", " evaluation 0.083161 \n", " inference 0.022326 \n", " MAE 21.682443 \n", " sMAPE 90.387898 \n", " MASE 36.807820 \n", " 100.0% training 7.707017 \n", " evaluation 0.112053 \n", " inference 0.026817 \n", " MAE 14.333950 \n", " sMAPE 71.043355 \n", " MASE 24.359035 " ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "Benchmark.get_results_df(loaded_results)" ] }, { "cell_type": "code", "execution_count": 20, "id": "4b0d4c97", "metadata": { "ExecuteTime": { "end_time": "2025-09-24T14:48:34.388485Z", "start_time": "2025-09-24T14:48:34.366584Z" } }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "
\n", "" ], "text/plain": [ "alt.LayerChart(...)" ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "some_preds[0].plot()" ] } ], "metadata": { "kernelspec": { "display_name": "ontime", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.13" } }, "nbformat": 4, "nbformat_minor": 5 }