User Guide

User Guide#

This guide introduces the main workflow for the forecast_evaluation package. It complements the API reference and follows the same usage patterns shown in the example notebook.

You can browse the full worked example in the example notebook.

The package is designed around a small number of core tasks:

  • loading forecast and outturn data

  • filtering data to a consistent evaluation sample

  • visualising forecasts, errors, and revisions

  • computing accuracy, bias, and efficiency diagnostics

  • comparing models with benchmark forecasts

  • exploring results in an interactive dashboard

Quick Start#

import forecast_evaluation as fe
import pandas as pd

You can either load the built-in Forecast Evaluation Report data, or create a dataset from your own forecasts and outturns.

Load the built-in dataset#

data = fe.ForecastData(load_fer=True)

This is the quickest way to start exploring the package and reproducing the examples.

Create a dataset from your own data#

forecast_data = fe.ForecastData(
    forecasts_data=forecasts_dataframe,
    outturns_data=outturns_dataframe,
)

The central object is forecast_evaluation.data.ForecastData. It stores raw forecasts, outturns, transformed series, and the main evaluation table used by the analysis and plotting functions.

Data Requirements#

Forecasts and outturns are provided as pandas DataFrames.

Forecasts must include the standard identification columns together with a value:

date, vintage_date, variable, source, frequency, forecast_horizon, value

Outturns use the same date, vintage_date, variable, frequency, and value columns, but do not require source or forecast_horizon.

An example forecast table looks like this:

           date vintage_date variable source frequency  forecast_horizon  value
0    2015-03-31   2015-03-31      gdp   BVAR         Q                 0    101
1    2015-06-30   2015-03-31      gdp   BVAR         Q                 1    102
2    2015-09-30   2015-03-31      gdp   BVAR         Q                 2    103

forecast_horizon is the information horizon supplied by the forecaster. It is based on the forecast target date (date) and the last target observation used for estimation: forecast_horizon = forecast_target_date - date_last_target_obs_used_for_estimation - 1, measured in periods at the stated frequency. Therefore, horizon 0 is the first period after the last observation used for estimation; it is not inferred from vintage_date. Negative horizons, including -1, may be supplied for forecast/outturn consistency checks, but are removed before forecasts are stored for evaluation.

The two horizons#

Results tables and plots use a different horizon from the one you supply, so it is worth keeping the two apart:

horizon

The calendar distance from the forecast vintage to the target date, date - vintage_date, in periods. This is the dimension every analysis groups, filters and reports on, and the one TestResult.filter(horizon=...) selects. It is 0 for a same-period forecast, 1 for one period ahead, and -1 for a backcast. Because it depends only on when a forecast was made, it is comparable across sources that conditioned on different amounts of data.

forecast_horizon

The information horizon you supply, as described above. It is not a reporting dimension: its only role is to set the lag length of the HAC standard errors in the regression-based tests, because it is what bounds the serial correlation in forecast errors.

The two coincide when forecasts are published as soon as the previous period’s outturn is released, and differ when there is a publication lag. Where the lag changes over a sample, one calendar horizon pools observations made at several information horizons; the lag length is then taken from the largest information horizon in the group and reported in the hac_maxlags column of the results. See Forecast Evaluation Methodology for what pooling implies for interpretation.

The package supports different forecast metrics such as levels, pop (period-on-period), and yoy (year-on-year). When required, transformations between these representations are computed internally when enough outturn history is available.

Working With ForecastData#

Load data and filter the sample#

The example notebook starts by loading FER data and filtering the variables used in subsequent analysis:

data = fe.ForecastData(load_fer=True)
data.filter(variables=["gdpkp", "cpisa", "aweagg"])

The forecast_evaluation.data.ForecastData.filter() method can restrict the sample by:

  • forecast dates via start_date and end_date

  • vintage dates via start_vintage and end_vintage

  • variables, metrics, sources, and frequencies

  • a custom filtering function through custom_filter

For example, you can exclude the COVID period from the analysis using the built-in covid_filter helper:

data_covid_filtered = data.copy()
data_covid_filtered.filter(custom_filter=fe.covid_filter)

If you need to reset to the original unfiltered data, use:

data.clear_filter()

Inspect the stored tables#

Useful accessors on a ForecastData object are:

  • data.df for the main evaluation table

  • data.forecasts for the transformed forecast table

  • data.outturns for the transformed outturn table

  • data.id_columns for the identification columns used to distinguish models

The summary() method prints a compact overview of the loaded variables, date range, vintages, and horizons.

Adding forecasts and labels#

You can extend an existing dataset with additional forecasts.

The notebook demonstrates adding a new label column that is treated as part of the forecast identifier:

sample_forecasts = fe.create_sample_forecasts()
sample_forecasts["extra label"] = "Model family A"

data_example_extra_columns = fe.ForecastData(load_fer=True)
data_example_extra_columns.add_forecasts(
    sample_forecasts,
    extra_ids=["extra label"],
)

This is useful when you want to separate forecasts by model family, conditioning assumption, scenario, or other metadata beyond source.

Nowcasting#

forecast_evaluation.data.NowcastData extends ForecastData for evaluating forecasts that are released multiple times within a single target period, e.g. weekly nowcast vintages of a quarterly variable. It uses integer-period forecast horizons (h=-1 backcast, h=0 nowcast, h=1 one-period-ahead) so that many intra-period vintages sharing the same horizon can be pooled for accuracy statistics.

Key differences from ForecastData#

  • first_forecast_horizon defaults to -1 so that backcasts are retained by default.

  • default_k defaults to 0, i.e. the first outturn release, rather than the fully revised outturn.

  • The load_fer and outturn_vintages constructor arguments used by ForecastData are not exposed.

Creating a NowcastData instance#

The package ships sample nowcast data with weekly forecast vintages and realistic outturn publication lags and revisions:

nowcast_outturns = fe.create_sample_nowcast_outturns()
nowcast_forecasts = fe.create_sample_nowcast_forecasts()

nowcast_data = fe.NowcastData(
    outturns_data=nowcast_outturns,
    forecasts_data=nowcast_forecasts,
)

Plotting nowcast vintages#

forecast_evaluation.plot_nowcasts() shows how forecasts for a single target quarter evolved over successive vintages, with one line per forecast identity (source plus any extra_ids) and the selected outturn maturity drawn as a horizontal reference. Pass k=0 to reference the first outturn release, which is usually the relevant comparison for nowcasts:

fe.plot_nowcasts(
    data=nowcast_data,
    variable="gdp",
    target_date="2024-06-30",
    frequency="Q",
    metric="levels",
    k=0,
)

Intra-period accuracy and bias#

forecast_evaluation.compute_intra_period_accuracy() and forecast_evaluation.compute_intra_period_bias() group forecast errors by a within-period time axis rather than by calendar vintage or forecast horizon, which suits densely-spaced intra-period vintages. Their plotting counterparts, forecast_evaluation.plot_intra_period_accuracy() and forecast_evaluation.plot_intra_period_bias(), share the same arguments.

The axis argument selects the time axis:

  • axis="period_end" (default) measures days from the forecast vintage to the end of the target period.

  • axis="publication" measures days from the forecast vintage to the release of the selected outturn vintage.

If k is left as None it resolves to data.default_k (0 for NowcastData).

intra_period_accuracy = fe.compute_intra_period_accuracy(
    data=nowcast_data,
    variable="gdp",
    metric="levels",
    frequency="Q",
    statistic="rmse",
    axis="period_end",
)

fe.plot_intra_period_accuracy(
    data=nowcast_data,
    variable="gdp",
    metric="levels",
    frequency="Q",
    statistic="rmse",
    axis="period_end",
)

intra_period_bias = fe.compute_intra_period_bias(
    data=nowcast_data,
    variable="gdp",
    metric="levels",
    frequency="Q",
    axis="publication",
)

fe.plot_intra_period_bias(
    data=nowcast_data,
    variable="gdp",
    metric="levels",
    frequency="Q",
    axis="publication",
)

Unsupported ForecastData functionality#

Some ForecastData methods and analyses do not apply to intra-period nowcast vintages and are unsupported on NowcastData. The following raise NotImplementedError:

  • FER data loading and filtering (add_fer_data, add_fer_outturns, add_fer_forecasts, filter_fer)

  • pseudo-vintage creation (create_pseudo_vintages)

  • benchmark generation (add_benchmarks)

The following raise ValueError:

  • efficiency, Blanchard-Leigh, revision-predictability, and revision-error-correlation analyses

Visualisation Workflow#

The package includes plotting functions for vintages, forecast errors, outturns, revisions, and rolling diagnostics. The examples below mirror the notebook.

Recent forecast errors against their historical distribution#

dates_to_highlight = pd.date_range(
    start="2022-01-01",
    end="2024-12-31",
    freq="QE",
)

fe.plot_forecast_error_density(
    data=data,
    horizon=4,
    variable="cpisa",
    metric="yoy",
    frequency="Q",
    source="mpr",
    k=12,
    highlight_dates=dates_to_highlight,
)

Vintage plots#

fe.plot_vintage(
    data=data,
    variable="cpisa",
    forecast_source=["mpr", "compass conditional", "bvar conditional"],
    frequency="Q",
    vintage_date="2020-03-31",
    metric="yoy",
)

Hedgehog charts#

fe.plot_hedgehog(
    data=data,
    variable="cpisa",
    forecast_source="mpr",
    metric="yoy",
    frequency="Q",
    k=12,
    convert_to_percentage=True,
)

Forecast errors over time#

fe.plot_errors_across_time(
    data_covid_filtered,
    variable="gdpkp",
    metric="yoy",
    ma_window=4,
    error="raw",
    sources=["mpr", "baseline ar(p) model"],
    k=12,
    horizons=[0, 4],
)

Forecast errors by vintage or horizon#

fe.plot_forecast_errors(
    data=data,
    variable="cpisa",
    metric="yoy",
    frequency="Q",
    source="mpr",
    vintage_date_forecast="2022-03-31",
    k=12,
    convert_to_percentage=True,
)

fe.plot_forecast_errors_by_horizon(
    data=data,
    variable="cpisa",
    metric="yoy",
    frequency="Q",
    source="mpr",
    k=12,
    convert_to_percentage=True,
)

Outturns and revisions#

fe.plot_outturns(
    data=data,
    variable="gdpkp",
    metric="yoy",
    frequency="Q",
    k=[0, 12],
    fill_k=True,
    convert_to_percentage=True,
)

fe.plot_outturn_revisions(
    data=data,
    variable="gdpkp",
    metric="yoy",
    frequency="Q",
    k=[4, 12],
    ma_window=4,
    fill_k=True,
    convert_to_percentage=True,
)

Forecast Evaluation Methods#

Most analytical functions return a forecast_evaluation.tests.results.TestResult object. These results can be converted to a DataFrame and, in many cases, plotted directly with .plot().

Accuracy statistics#

Use forecast_evaluation.compute_accuracy_statistics() to calculate statistics such as RMSE, mean absolute error, root median square error, and observation counts.

accuracy_results = fe.compute_accuracy_statistics(data=data, k=12)

accuracy_results.plot(
    variable="cpisa",
    metric="yoy",
    frequency="Q",
    statistic="rmse",
    convert_to_percentage=True,
)

Comparing to a benchmark#

You can compare model performance relative to a benchmark using summary functions and companion plots.

accuracy_comparison = fe.compare_to_benchmark(
    df=accuracy_results,
    benchmark_model="baseline ar(p) model",
    statistic="rmse",
)

fe.plot_compare_to_benchmark(
    df=accuracy_results,
    variable="cpisa",
    metric="yoy",
    frequency="Q",
    benchmark_model="baseline ar(p) model",
    statistic="rmse",
)

To create a compact table for selected horizons:

comparison_table = fe.create_comparison_table(
    df=accuracy_results.to_df(),
    variable="cpisa",
    metric="yoy",
    frequency="Q",
    benchmark_model="baseline ar(p) model",
    statistic="rmse",
    horizons=[0, 1, 2, 4, 8, 12],
)

Relative accuracy tests#

The package includes Diebold-Mariano testing and rolling-window extensions.

diebold_mariano_results = fe.diebold_mariano_table(
    data=data,
    benchmark_model="mpr",
)

For rolling analysis, first create a focused dataset and then pass the test function to forecast_evaluation.rolling_analysis().

forecast_data_dm_rolling = data.copy()
forecast_data_dm_rolling.filter(
    variables=["gdpkp"],
    metrics=["yoy"],
    sources=["mpr", "baseline random walk model"],
)

rolling_dm = fe.rolling_analysis(
    data=forecast_data_dm_rolling,
    window_size=40,
    analysis_func=fe.diebold_mariano_table,
    analysis_args={"benchmark_model": "mpr"},
)

rolling_dm.plot(variable="gdpkp", horizons=[0, 4])

Fluctuation tests#

Fluctuation tests provide a multiple-window diagnostic that is robust to repeated rolling analysis.

rolling_dm_fluctuation = fe.fluctuation_tests(
    data=forecast_data_dm_rolling,
    window_size=40,
    test_func=fe.diebold_mariano_table,
    test_args={"benchmark_model": "mpr"},
)

rolling_dm_fluctuation.plot(variable="gdpkp", horizons=[0, 4])

Bias and efficiency analysis#

The notebook also demonstrates the main econometric diagnostics provided by the package.

bias_results = fe.bias_analysis(data=data, source="mpr", k=12, verbose=False)
bias_results.plot(variable="aweagg", source="mpr", metric="yoy", frequency="Q")

rolling_bias = fe.rolling_analysis(
    data=data_gdp,
    window_size=40,
    analysis_func=fe.bias_analysis,
    analysis_args={"k": 12},
)

bl_results = fe.blanchard_leigh_horizon_analysis(
    data=data,
    source="mpr",
    outcome_variable="cpisa",
    outcome_metric="yoy",
    instrument_variable="gdpkp",
    instrument_metric="yoy",
)

weak_efficiency_results = fe.weak_efficiency_analysis(
    data=data,
    source="mpr",
    k=12,
    verbose=False,
)

Revisions analysis#

Revisions can be analysed directly through dedicated tests and plots.

revisions_correlation_results = fe.revisions_errors_correlation_analysis(
    data=data,
    source="mpr",
    k=12,
)

revisions_predictable_results = fe.revision_predictability_analysis(
    data=data,
    frequency="Q",
    n_revisions=5,
)

fe.plot_average_revision_by_period(
    data=data,
    source="mpr",
    variable="gdpkp",
    metric="yoy",
    frequency="Q",
)

Adding Benchmark Forecasts#

You can augment a ForecastData object with simple benchmark models using forecast_evaluation.data.ForecastData.add_benchmarks().

Supported benchmark families are AR (autoregressive) and random_walk.

data.add_benchmarks(metric="pop", models=["AR", "random_walk"])

Optional arguments let you restrict the benchmark generation to selected variables or frequencies, control the number of forecast periods, and supply an estimation start date.

Density Forecasts#

For probabilistic forecasts with quantiles, use forecast_evaluation.data.DensityForecastData, which extends ForecastData.

Density forecast input must include a quantile column with values between 0 and 1.

density_df = fe.create_sample_density_forecasts()
density_data = fe.DensityForecastData(forecasts_data=density_df)

You can also add density forecasts to an existing object:

density_data = fe.DensityForecastData()
density_data.add_density_forecasts(density_df)

Density forecast objects retain the standard forecast and outturn workflow while also exposing a density_forecasts table for quantile-level analysis.

Dashboard#

The package includes an interactive dashboard for exploring forecasts, errors, and analysis outputs.

Run it from a ForecastData object:

data.run_dashboard()

When working inside a notebook, you can embed the dashboard in the notebook output:

data.run_dashboard(from_jupyter=True)

Further Reading#

For a worked example covering the main plotting and testing functions, see the example notebook in the repository. For function signatures and parameter-level details, refer to the API reference.