Model Comparison

The compare() function fits multiple models and evaluates their forecast accuracy against a test set.

---

Usage

from durbyn import SES, Holt, HoltWinters, AutoARIMA, compare

y_train = y[:120]
y_test = y[120:]

result = compare(
    models={
        "SES": SES(),
        "Holt": Holt(damped=True),
        "HW_mul": HoltWinters(seasonal="multiplicative"),
        "AutoARIMA": AutoARIMA(),
    },
    y_train=y_train,
    y_test=y_test,
    m=12,
    level=[80, 95],
)

---

Parameters

Parameter	Type	Default	Description
models	dict[str, BaseForecaster]	required	Dict of {name: model}
y_train	array-like	required	Training data
y_test	array-like	required	Test data for evaluation
m	int	1	Seasonal period
h	int \| None	None	Forecast horizon. Defaults to len(y_test)
level	list[int] \| None	None	Confidence levels

---

ComparisonResult

The returned ComparisonResult has:

Attributes

Attribute	Type	Description
model_names	list[str]	Names of successfully fitted models
forecasts	dict[str, ForecastResult]	Forecast per model
metrics	dict[str, dict[str, float]]	Accuracy metrics per model
y_train	np.ndarray	Training data
y_test	np.ndarray	Test data

Methods

.to_dataframe()

Returns a pandas.DataFrame with one row per model and metric columns (ME, RMSE, MAE, MPE, MAPE, ACF1, MASE).

df = result.to_dataframe()
print(df)
#             ME       RMSE       MAE       MPE      MAPE      ACF1      MASE
# SES       ...        ...       ...       ...       ...       ...       ...
# Holt      ...        ...       ...       ...       ...       ...       ...

.plot(ax=None)

Overlays all forecasts on the same axes with training and test data.

result.plot()

---

Error Handling

If a model fails to fit or forecast, a warning is issued and that model is skipped. The remaining models are still included in the result.

import warnings

with warnings.catch_warnings(record=True) as w:
    result = compare(models={"broken": SomeModel()}, y_train=y_train, y_test=y_test)
    for warning in w:
        print(warning.message)