Data Loading

class forecast_evaluation.data.DensityForecastData(outturns_data: DataFrame | None = None, forecasts_data: DataFrame | None = None, load_fer: bool | None = False, extra_ids: list[str] | None = None, compute_levels: bool = True)

Bases: ForecastData

Class for density forecasts with quantile information.

Extends the ForecastData class to handle density forecasts. DensityForecastData objects include a density_forecasts attribute that contains forecasts with quantile information (basically an extra column indicating quantiles). It can still handle point forecasts with the forecasts attribute.

Parameters:

• outturns_data (pd.DataFrame, optional) – DataFrame containing outturn (actual) data.

• forecasts_data (pd.DataFrame, optional) – DataFrame containing point forecast records.

• density_forecasts_data (pd.DataFrame, optional) – DataFrame containing density forecast records. Must include ‘quantile’ column.

• point_estimator (str, optional) – Estimator for point forecasts; can be ‘median’, ‘mean’ or ‘mode’. Default is ‘median’.

• load_fer (bool, optional) – Whether to load FER (Forecast Evaluation Report) data. Default is False.

• extra_ids (list of str, optional) – Additional identification columns beyond ‘source’ and ‘quantile’.

Examples

>>> import pandas as pd
>>> from forecast_evaluation.data import DensityForecastData
>>>
>>> # Create sample density forecasts
>>> df = pd.DataFrame({
...     'date': pd.date_range('2023-01-01', periods=4, freq='QE'),
...     'vintage_date': pd.Timestamp('2023-01-01'),
...     'variable': 'gdp',
...     'frequency': 'Q',
...     'forecast_horizon': [1, 2, 3, 4],
...     'source': 'model_1',
...     'quantile': 0.5,
...     'value': [100, 101, 102, 103]
... })
>>>
>>> density_data = DensityForecastData(forecasts_data=df)
>>> median = density_data.get_median_forecast()

__init__(outturns_data: DataFrame | None = None, forecasts_data: DataFrame | None = None, load_fer: bool | None = False, extra_ids: list[str] | None = None, compute_levels: bool = True)

Initialise DensityForecastData.

Initialises the density forecast data object. If forecasts_data is provided, it will be validated and added. The ‘quantile’ column is automatically included as an identification column.

add_density_forecasts(df: DataFrame, extra_ids: list[str] | None = None) → None

Validate and add density forecasts with quantile column.

Parameters:

• df (pd.DataFrame) – DataFrame containing density forecast records. Must include ‘quantile’ column with values between 0 and 1.

• extra_ids (list of str, optional) – Additional identification columns beyond ‘source’ and ‘quantile’.

Raises:

ValueError – If ‘quantile’ column is missing from the DataFrame.

Examples

>>> density_data = DensityForecastData()
>>> df = pd.DataFrame({
...     'date': ['2023-01-01'],
...     'vintage_date': ['2023-01-01'],
...     'variable': ['gdp'],
...     'frequency': ['Q'],
...     'forecast_horizon': [1],
...     'source': ['model_1'],
...     'quantile': [0.5],
...     'value': [100]
... })
>>> density_data.add_density_forecasts(df)

clear_filter() → None

Reset both parent forecasts and density forecasts to include all original data.

property density_forecasts: DataFrame

Get density forecasts with quantile information.

filter(start_date: str | None = None, end_date: str | None = None, start_vintage: str | None = None, end_vintage: str | None = None, variables: list[str] | None = None, metrics: list[str] | None = None, sources: list[str] | None = None, frequencies: list[str] | None = None, custom_filter: Callable[[DataFrame], DataFrame] | None = None, filter_point_forecasts: bool | None = True, filter_density_forecasts: bool | None = True)

Filter the forecasts and main tables to only include data within specified date and vintage ranges, and optionally by variables, metrics, sources, or a custom filter.

Parameters:

• start_date (str, optional) – Start date to filter forecasts (inclusive). Format ‘YYYY-MM-DD’. Default is None in which case the analysis starts with the earliest date.

• end_date (str, optional) – End date to filter forecasts (inclusive). Format ‘YYYY-MM-DD’. Default is None in which case the analysis ends with the latest date.

• start_vintage (str, optional) – Start vintage date to filter forecasts (inclusive). Format ‘YYYY-MM-DD’. Default is None in which case the analysis starts with the earliest vintage.

• end_vintage (str, optional) – End vintage date to filter forecasts (inclusive). Format ‘YYYY-MM-DD’. Default is None in which case the analysis ends with the latest vintage.

• variables (Optional[Union[str, list[str]]] = None) – List of variable identifiers to filter. Default is None (no filtering).

• metrics (Optional[list[str]] = None) – List of metric identifiers to filter. Default is None (no filtering).

• sources (Optional[Union[str, list[str]]] = None) – List of source identifiers to filter. Default is None (no filtering).

• frequencies (Optional[Union[str, list[str]]] = None) – List of frequency identifiers to filter. Default is None (no filtering).

• custom_filter (Callable[[pd.DataFrame], pd.DataFrame], optional) – A custom filtering function that takes a DataFrame as input and returns a filtered DataFrame. Default is None. Custom filters should use ‘vintage_date_forecast’ as the column name.

• filter_point_forecasts (bool, optional) – Whether to apply the filter to point forecasts. Default is True.

• filter_density_forecasts (bool, optional) – Whether to apply the filter to density forecasts. Default is True.

merge(other: ForecastData, compute_levels: bool = True) → None

Merge another ForecastData or DensityForecastData instance into this one.

Parameters:

• other (ForecastData or DensityForecastData) – Another ForecastData or DensityForecastData instance to merge with this one.

• compute_levels (bool, optional) – Whether to automatically transform non-levels forecasts from other to levels if outturns data is available. When True, forecasts in ‘pop’ and ‘yoy’ metrics will be converted to levels using the available outturns data. Useful if you add ‘pop’ and want to analyse ‘yoy’ forecasts and vice versa. If the transformation fails for specific groups (e.g., due to insufficient historical data), those groups will be skipped with a warning message. Default is True.

plot_density_vintage(variable: str, vintage_date: str | Timestamp, quantiles: list[float] | None = [0.16, 0.5, 0.84], forecast_source: list[str] | None = None, outturn_start_date: str | Timestamp | None = None, frequency: Literal['Q', 'M'] | None = None, metric: Literal['levels', 'pop', 'yoy'] = 'levels', return_plot: bool = False, **kwargs) → tuple | None

Plot forecast density plots.

Parameters:

**kwargs – Additional keyword arguments passed to the plotting function.

Notes

This method creates density plots for the density forecasts.

sample_from_density(n_samples: int = 10000, random_state: int | None = None) → DataFrame

Generate samples from the empirical distribution defined by quantiles.

Uses inverse transform sampling to draw samples from the distribution defined by the quantile forecasts.

Parameters:

• n_samples (int, optional) – Number of samples to generate per forecast group. Default is 10000.

• random_state (int, optional) – Random seed for reproducibility.

Returns:

DataFrame with sampled values. Columns: [id_columns…, ‘sample_id’, ‘value’]

Return type:

pd.DataFrame

Notes

Works well when you have many quantiles (50+). With few quantiles, consider using parametric fitting instead.

Examples

>>> samples = density_data.sample_from_density(n_samples=10000, random_state=42)
>>> mean = samples.groupby(['date', 'variable'])['value'].mean()

to_point_forecast(method: str = 'median') → ForecastData

Convert density forecasts to point forecasts.

Parameters:

method (str, optional) – Method to extract point forecast: - ‘median’: Use 0.5 quantile (default, most robust) - ‘mean’: Average via sampling from distribution - specific quantile: e.g., ‘0.5’, ‘0.75’

Returns:

Point forecast data object.

Return type:

ForecastData

Examples

>>> # Convert using median
>>> point_data = density_data.to_point_forecast('median')
>>>
>>> # Convert using mean via sampling
>>> point_data = density_data.to_point_forecast('mean')
>>>
>>> # Convert using specific quantile
>>> point_data = density_data.to_point_forecast('0.75')

class forecast_evaluation.data.ForecastData(outturns_data: DataFrame | None = None, forecasts_data: DataFrame | None = None, load_fer: bool | None = False, *, extra_ids: list[str] | None = None, metric: Literal['levels', 'pop', 'yoy'] = 'levels', compute_levels: bool = True, data_check: bool = True, first_forecast_horizon: int | dict[str, int] | None = None, outturn_vintages: bool = True)

Bases: PlottingMixin

Class for validation and extending forecast data.

The main method is .add_forecasts() which validates the input data and compute relevant dataframes. underscore indicates that the object only meant to be used internally.

Notes

Each ForecastData instance should only contain forecasts of a single frequency (e.g., all quarterly or all monthly). To work with multiple frequencies, create separate ForecastData instances for each frequency.

__init__(outturns_data: DataFrame | None = None, forecasts_data: DataFrame | None = None, load_fer: bool | None = False, *, extra_ids: list[str] | None = None, metric: Literal['levels', 'pop', 'yoy'] = 'levels', compute_levels: bool = True, data_check: bool = True, first_forecast_horizon: int | dict[str, int] | None = None, outturn_vintages: bool = True)

Initialise with user data, FER data or null.

Parameters:

• outturns_data (pd.DataFrame, optional) – DataFrame containing outturn records to add on initialisation. Default is None.

• forecasts_data (pd.DataFrame, optional) – DataFrame containing forecast records to add on initialisation. Default is None.

• load_fer (bool, optional) – Whether to load FER outturns and forecast data on initialisation. Default is False.

• extra_ids (Optional[list[str]], optional) – List of extra label columns (in addition to ‘source’) present in the forecasts data. Default is None.

• metric (str, optional) – Metric to assign to the forecasts if ‘metric’ column is not present or contains null values. Default is ‘levels’. Options: ‘levels’, ‘pop’, ‘yoy’.

• compute_levels (bool, optional) – Whether to automatically transform non-levels forecasts to levels if outturns data is available. When True, forecasts in ‘pop’ and ‘yoy’ metrics will be converted to levels using the available outturns data. Useful if you add ‘pop’ and want to analyse ‘yoy’ forecasts and vice versa. If the transformation fails for specific groups (e.g., due to insufficient historical data), those groups will be skipped with a warning message. Default is True.

• data_check (bool, optional) – Whether to run data checks when adding forecasts. See add_forecasts() for details. Default is True.

• outturn_vintages (bool, optional) – Whether the outturn data contains vintage information (multiple releases of the same data point over time). When False, the data is assumed to contain a single final outturn per date, and columns vintage_date and forecast_horizon are not required in the outturn data. The k and latest_vintage columns in the main table will be set to sentinel values and filter_k will be a no-op. Features that depend on outturn revisions (e.g., plot_outturn_revisions, create_outturn_revisions) will raise an error. Default is True.

• first_forecast_horizon (int, dict[str, int], or None, optional) – The minimum forecast horizon to retain in processed forecasts. Pass an int to apply the same threshold to all variables, or a dict mapping variable names to per-variable thresholds (variables not in the dict default to 0). Set to a negative value (e.g., -1, -2) to include backcasts, i.e., forecasts for periods that have already ended but whose data has not yet been released. When None (default), the threshold for each variable is max(0, min(forecast_horizon)) — i.e. the smallest non-negative horizon present in that variable’s forecasts.

add_benchmarks(models: list[str] | str = ['AR', 'random_walk'], variables: str | Iterable[str] | None = None, metric: Literal['levels', 'diff', 'pop', 'yoy'] = 'levels', frequency: Literal['Q', 'M'] | Iterable[Literal['Q', 'M']] | None = None, forecast_periods: int = 13, *, max_lag: Literal[1, 2] = 2, estimation_start_date: Timestamp | None = None, show_progress: bool = False) → None

Add benchmark model forecasts to the ForecastData instance.

Builds the requested benchmark models from the available outturn data and appends their forecasts to the instance via add_forecasts(). The instance is modified in place.

Parameters:

• models (list of str or str, optional) – Benchmark model(s) to add. Valid options are "AR" (autoregressive AR(p) model) and "random_walk". Default is both (["AR", "random_walk"]).

• variables (str, iterable of str, or None, optional) – Variable(s) to build benchmarks for. If None (default), benchmarks are built for all variables present in the outturns.

• metric (str, optional) – Metric to build the benchmarks for. Options: "levels", "diff", "pop", "yoy". Default is "levels".

• frequency (str, iterable of str, or None, optional) – Frequency (or frequencies) to build benchmarks for ("Q" for quarterly, "M" for monthly). If None (default), frequencies are inferred from the outturns.

• forecast_periods (int, optional) – Number of periods to forecast ahead. Default is 13.

• max_lag (int, optional) – Maximum number of lags (AR order) to consider when selecting the AR(p) model via BIC. Must be 1 or 2. Only applies to the "AR" model. Default is 2.

• estimation_start_date (pd.Timestamp, optional) – The date from which to start including data for AR(p) model estimation. If None (default), all available data is used. Only applies to the "AR" model.

• show_progress (bool, optional) – Whether to show progress bars while building the benchmarks. Default is False.

Returns:

The method modifies the ForecastData instance in place.

Return type:

None

Raises:

ValueError – If any model in models is not a recognised benchmark model.

add_fer_data() → None

Load and add FER outturns and forecast data to existing records.

add_fer_forecasts() → None

Load and add FER forecast data to existing records.

add_fer_outturns() → None

Load and add FER outturn data to existing records.

add_forecasts(df: DataFrame, *, extra_ids: list[str] | None = None, metric: Literal['levels', 'pop', 'yoy']='levels', compute_levels: bool = True, data_check: bool = True, first_forecast_horizon: int | dict[str, int] | None=<object object>) → None

Validate new forecasts, transform forecasts and outturns and compute main table and revisions.

Parameters:

• df (pd.DataFrame) – DataFrame containing new forecast records to add.

• extra_ids (list of str, optional) – List of extra label/identification columns (in addition to ‘source’) present in the forecasts data. Default is None.

• metric (str, optional) – Metric to assign to the forecasts if ‘metric’ column is not present or contains null values. Default is ‘levels’. Options: ‘levels’, ‘pop’, ‘yoy’.

• compute_levels (bool, optional) – Whether to automatically transform non-levels forecasts to levels if outturns data is available. When True, forecasts in ‘pop’ and ‘yoy’ metrics will be converted to levels using the available outturns data. Useful if you add ‘pop’ and want to analyse ‘yoy’ forecasts and vice versa. If the transformation fails for specific groups (e.g., due to insufficient historical data), those groups will be skipped with a warning message. Default is True.

• data_check (bool, optional) –

  Whether to run data checks comparing forecast values to outturns per (source, variable, metric, frequency) group. When True:

  • Horizon -1 check (primary): if forecast_horizon == -1 rows exist, each is compared to the outturn from the same vintage at the same date. Warns if the mean absolute deviation exceeds 0.5 std of the outturn series.

  • IQR ratio check (fallback): over all (date, vintage_date) pairs that overlap, warns if the forecast IQR differs from the outturn IQR by >5x.

  Detects common user errors: wrong metric column, scaling mistakes (e.g. pct*100 instead of pct), or non-real-time vintages.

  Warnings only; never raises errors. Set to False to disable. Default is True.

• first_forecast_horizon (int, dict[str, int], or None, optional) – The minimum forecast horizon to retain. Overrides the instance-level attribute when provided. Pass an int to apply the same threshold to all variables, or a dict mapping variable names to per-variable thresholds (variables not in the dict default to 0). Set to None to auto-compute as max(0, min(forecast_horizon)) per variable. When not provided, uses the existing instance attribute.

Notes

Outturns must be added before forecasts (call add_outturns first). All forecasts added to a ForecastData instance must have the same frequency. To work with forecasts of different frequencies, create separate ForecastData instances for each frequency. When compute_levels is True, sufficient historical outturn data is required for transformation, especially for ‘yoy’ metrics which need data from one year prior.

add_outturns(df: DataFrame, *, metric: Literal['levels', 'pop', 'yoy'] = 'levels') → None

Validate new outturns and add them to the outturns dataset

Parameters:

• df (pd.DataFrame) – DataFrame containing new outturn records to add.

• metric (str, optional) – Metric to assign to the outturns if ‘metric’ column is not present or contains null values. Default is ‘levels’. Options: ‘levels’, ‘pop’, ‘yoy’.

clear_filter() → None

Reset the forecasts, main and revisions tables to include all original data.

copy() → ForecastData

Return a deep copy of the ForecastData object.

create_pseudo_vintages(fill_to: str, vintage_frequency: Literal['M', 'Q'] = 'Q', publication_lags: dict[str, int] | None = None) → None

Create pseudo vintages for outturns.

Starts from the earliest available vintage in the data and fills backward to fill_to.

This method computes the publication lag from existing data and creates a full vintage structure where each vintage contains all data available at that point in time. A vintage at date X contains all data up to (X - publication_lag).

Parameters:

• fill_to (str) – The earliest vintage date to create (i.e. how far back to fill). Format ‘YYYY-MM-DD’. Vintages are generated from this date up to the earliest existing vintage in the data.

• vintage_frequency (str, optional) – Frequency at which to create vintages. Default is ‘Q’ (quarterly). Options: ‘M’ (monthly), ‘Q’ (quarterly).

• publication_lags (dict[str, int] or None, optional) – A dictionary mapping variable names to their publication lag (in units of vintage_frequency). If None (default), the lag is computed from existing data.

Notes

• Computes publication lag per variable from existing data (max_vintage - max_date)

• Expands the dataset by creating multiple vintage records for each data point

• Each vintage V includes all data points D where (D + lag) <= V

• Requires outturns to already have vintage_date values to compute the lag

property df: DataFrame

Get the main DataFrame.

filter(start_date: str | None = None, end_date: str | None = None, start_vintage: str | None = None, end_vintage: str | None = None, variables: str | list[str] | None = None, metrics: list[str] | None = None, sources: str | list[str] | None = None, frequencies: str | list[str] | None = None, custom_filter: Callable[[DataFrame], DataFrame] | None = None) → None

Filter the forecasts and main tables to only include data within specified date and vintage ranges, and optionally by variables, metrics, sources, or a custom filter.

Parameters:

• start_date (str, optional) – Start date to filter forecasts (inclusive). Format ‘YYYY-MM-DD’. Default is None in which case the analysis starts with the earliest date.

• end_date (str, optional) – End date to filter forecasts (inclusive). Format ‘YYYY-MM-DD’. Default is None in which case the analysis ends with the latest date.

• start_vintage (str, optional) – Start vintage date to filter forecasts (inclusive). Format ‘YYYY-MM-DD’. Default is None in which case the analysis starts with the earliest vintage.

• end_vintage (str, optional) – End vintage date to filter forecasts (inclusive). Format ‘YYYY-MM-DD’. Default is None in which case the analysis ends with the latest vintage.

• variables (str or list of str, optional) – List of variable identifiers to filter. Default is None (no filtering).

• metrics (list of str, optional) – List of metric identifiers to filter. Default is None (no filtering).

• sources (str or list of str, optional) – List of source identifiers to filter. Default is None (no filtering).

• frequencies (str or list of str, optional) – List of frequency identifiers to filter. Default is None (no filtering).

• custom_filter (Callable[[pd.DataFrame], pd.DataFrame], optional) – A custom filtering function that takes a DataFrame as input and returns a filtered DataFrame. Default is None. Custom filters should use ‘vintage_date_forecast’ as the column name.

filter_fer() → None

Filter the main dataset to only include specific variable-metric and model combinations

property forecast_required_columns: list[str]

Get the required columns list to help the user.

property forecasts: DataFrame

Get forecasts.

property id_columns: list[str] | None

Get identification / labelling columns.

merge(other: ForecastData, compute_levels: bool = True) → None

Merge another ForecastData instance into this one.

Parameters:

• other (ForecastData) – Another ForecastData instance to merge with this one.

• compute_levels (bool, optional) – Whether to automatically transform non-levels forecasts from other to levels if outturns data is available. When True, forecasts in ‘pop’ and ‘yoy’ metrics will be converted to levels using the available outturns data. Useful if you add ‘pop’ and want to analyse ‘yoy’ forecasts and vice versa. If the transformation fails for specific groups (e.g., due to insufficient historical data), those groups will be skipped with a warning message. Default is True.

Returns:

The method modifies this instance in place.

Return type:

None

property outturn_required_columns: list[str]

Get the required columns list to help the user.

property outturn_vintages: bool

Whether the outturn data contains vintage information.

property outturns: DataFrame

Get outturns.

run_dashboard(from_jupyter: bool = False, host: str = '127.0.0.1', port: int = 8000) → None

Run the Shiny dashboard with the current data.

Parameters:

• from_jupyter (bool, optional) – Whether to run the dashboard within a Jupyter notebook. Default is False.

• host (str, optional) – Host address for the dashboard server. Default is “127.0.0.1”.

• port (int, optional) – Port number for the dashboard server. Default is 8000.

summary() → None

Print a summary of the forecast and outturns datasets.

For each dataset, displays: - Number of variables - List of variables with their properties - For each variable: frequency, date range, and first vintage date

forecast_evaluation.data.create_sample_forecasts() → DataFrame

Create sample forecasts DataFrame for testing and examples.

forecast_evaluation.data.create_sample_outturns() → DataFrame

Create sample outturns DataFrame for testing and examples.