Data Loading

class forecast_evaluation.data.DensityForecastData(outturns_data: DataFrame | None = None, forecasts_data: DataFrame | None = None, load_fer: bool | None = False, extra_ids: list[str] | None = None, compute_levels: bool = True)

Bases: ForecastData

Class for density forecasts with quantile information.

Extends the ForecastData class to handle density forecasts. DensityForecastData objects include a density_forecasts attribute that contains forecasts with quantile information (basically an extra column indicating quantiles). It can still handle point forecasts with the forecasts attribute.

Parameters:

• outturns_data (pd.DataFrame, optional) – DataFrame containing outturn (actual) data.

• forecasts_data (pd.DataFrame, optional) – DataFrame containing point forecast records.

• density_forecasts_data (pd.DataFrame, optional) – DataFrame containing density forecast records. Must include ‘quantile’ column.

• point_estimator (str, optional) – Estimator for point forecasts; can be ‘median’, ‘mean’ or ‘mode’. Default is ‘median’.

• load_fer (bool, optional) – Whether to load FER (Forecast Evaluation Report) data. Default is False.

• extra_ids (list of str, optional) – Additional identification columns beyond ‘source’ and ‘quantile’.

Examples

>>> import pandas as pd
>>> from forecast_evaluation.data import DensityForecastData
>>>
>>> # Create sample density forecasts
>>> df = pd.DataFrame({
...     'date': pd.date_range('2023-01-01', periods=4, freq='QE'),
...     'vintage_date': pd.Timestamp('2023-01-01'),
...     'variable': 'gdp',
...     'frequency': 'Q',
...     'forecast_horizon': [1, 2, 3, 4],
...     'source': 'model_1',
...     'quantile': 0.5,
...     'value': [100, 101, 102, 103]
... })
>>>
>>> density_data = DensityForecastData(forecasts_data=df)
>>> median = density_data.get_median_forecast()

__init__(outturns_data: DataFrame | None = None, forecasts_data: DataFrame | None = None, load_fer: bool | None = False, extra_ids: list[str] | None = None, compute_levels: bool = True)

Initialise DensityForecastData.

Initialises the density forecast data object. If forecasts_data is provided, it will be validated and added. The ‘quantile’ column is automatically included as an identification column.

add_density_forecasts(df: DataFrame, extra_ids: list[str] | None = None) → None

Validate and add density forecasts with quantile column.

Parameters:

• df (pd.DataFrame) – DataFrame containing density forecast records. Must include ‘quantile’ column with values between 0 and 1.

• extra_ids (list of str, optional) – Additional identification columns beyond ‘source’ and ‘quantile’.

Raises:

ValueError – If ‘quantile’ column is missing from the DataFrame.

Examples

>>> density_data = DensityForecastData()
>>> df = pd.DataFrame({
...     'date': ['2023-01-01'],
...     'vintage_date': ['2023-01-01'],
...     'variable': ['gdp'],
...     'frequency': ['Q'],
...     'forecast_horizon': [1],
...     'source': ['model_1'],
...     'quantile': [0.5],
...     'value': [100]
... })
>>> density_data.add_density_forecasts(df)

clear_filter() → None

Reset both parent forecasts and density forecasts to include all original data.

property density_forecasts: DataFrame

Get density forecasts with quantile information.

filter(start_date: str | None = None, end_date: str | None = None, start_vintage: str | None = None, end_vintage: str | None = None, variables: list[str] | None = None, metrics: list[str] | None = None, sources: list[str] | None = None, frequencies: list[str] | None = None, custom_filter: Callable[[DataFrame], DataFrame] | None = None, filter_point_forecasts: bool | None = True, filter_density_forecasts: bool | None = True)

Filter the forecasts and main tables to only include data within specified date and vintage ranges, and optionally by variables, metrics, sources, or a custom filter.

Parameters:

• start_date (str, optional) – Start date to filter forecasts (inclusive). Format ‘YYYY-MM-DD’. Default is None in which case the analysis starts with the earliest date.

• end_date (str, optional) – End date to filter forecasts (inclusive). Format ‘YYYY-MM-DD’. Default is None in which case the analysis ends with the latest date.

• start_vintage (str, optional) – Start vintage date to filter forecasts (inclusive). Format ‘YYYY-MM-DD’. Default is None in which case the analysis starts with the earliest vintage.

• end_vintage (str, optional) – End vintage date to filter forecasts (inclusive). Format ‘YYYY-MM-DD’. Default is None in which case the analysis ends with the latest vintage.

• variables (Optional[Union[str, list[str]]] = None) – List of variable identifiers to filter. Default is None (no filtering).

• metrics (Optional[list[str]] = None) – List of metric identifiers to filter. Default is None (no filtering).

• sources (Optional[Union[str, list[str]]] = None) – List of source identifiers to filter. Default is None (no filtering).

• frequencies (Optional[Union[str, list[str]]] = None) – List of frequency identifiers to filter. Default is None (no filtering).

• custom_filter (Callable[[pd.DataFrame], pd.DataFrame], optional) – A custom filtering function that takes a DataFrame as input and returns a filtered DataFrame. Default is None. Custom filters should use ‘vintage_date_forecast’ as the column name.

• filter_point_forecasts (bool, optional) – Whether to apply the filter to point forecasts. Default is True.

• filter_density_forecasts (bool, optional) – Whether to apply the filter to density forecasts. Default is True.

merge(other: ForecastData, compute_levels: bool = True) → ForecastData

Merge another ForecastData or DensityForecastData instance into this one.

Parameters:

• other (ForecastData or DensityForecastData) – Another ForecastData or DensityForecastData instance to merge with this one.

• compute_levels (bool, optional) – Whether to automatically transform non-levels forecasts from other to levels if outturns data is available. When True, forecasts in ‘pop’ and ‘yoy’ metrics will be converted to levels using the available outturns data. Useful if you add ‘pop’ and want to analyse ‘yoy’ forecasts and vice versa. If the transformation fails for specific groups (e.g., due to insufficient historical data), those groups will be skipped with a warning message. Default is True.

Returns:

Updated DensityForecastData instance containing merged data from both instances.

Return type:

DensityForecastData

plot_density_vintage(variable: str, vintage_date: str | Timestamp, quantiles: list[float] | None = [0.16, 0.5, 0.84], forecast_source: list[str] = None, outturn_start_date: str | Timestamp = None, frequency: Literal['Q', 'M'] = 'Q', metric: Literal['levels', 'pop', 'yoy'] = 'levels', return_plot: bool = False, **kwargs) → None

Plot forecast density plots.

Parameters:

**kwargs – Additional keyword arguments passed to the plotting function.

Notes

This method creates density plots for the density forecasts.

sample_from_density(n_samples: int = 10000, random_state: int | None = None) → DataFrame

Generate samples from the empirical distribution defined by quantiles.

Uses inverse transform sampling to draw samples from the distribution defined by the quantile forecasts.

Parameters:

• n_samples (int, optional) – Number of samples to generate per forecast group. Default is 10000.

• random_state (int, optional) – Random seed for reproducibility.

Returns:

DataFrame with sampled values. Columns: [id_columns…, ‘sample_id’, ‘value’]

Return type:

pd.DataFrame

Notes

Works well when you have many quantiles (50+). With few quantiles, consider using parametric fitting instead.

Examples

>>> samples = density_data.sample_from_density(n_samples=10000, random_state=42)
>>> mean = samples.groupby(['date', 'variable'])['value'].mean()

to_point_forecast(method: str = 'median') → ForecastData

Convert density forecasts to point forecasts.

Parameters:

method (str, optional) – Method to extract point forecast: - ‘median’: Use 0.5 quantile (default, most robust) - ‘mean’: Average via sampling from distribution - specific quantile: e.g., ‘0.5’, ‘0.75’

Returns:

Point forecast data object.

Return type:

ForecastData

Examples

>>> # Convert using median
>>> point_data = density_data.to_point_forecast('median')
>>>
>>> # Convert using mean via sampling
>>> point_data = density_data.to_point_forecast('mean')
>>>
>>> # Convert using specific quantile
>>> point_data = density_data.to_point_forecast('0.75')

class forecast_evaluation.data.ForecastData(outturns_data: DataFrame | None = None, forecasts_data: DataFrame | None = None, load_fer: bool | None = False, *, extra_ids: list[str] | None = None, metric: Literal['levels', 'pop', 'yoy'] = 'levels', compute_levels: bool = True, data_check: bool = True)

Bases: object

Class for validation and extending forecast data.

The main method is .add_forecasts() which validates the input data and compute relevant dataframes. underscore indicates that the object only meant to be used internally.

Notes

Each ForecastData instance should only contain forecasts of a single frequency (e.g., all quarterly or all monthly). To work with multiple frequencies, create separate ForecastData instances for each frequency.

__init__(outturns_data: DataFrame | None = None, forecasts_data: DataFrame | None = None, load_fer: bool | None = False, *, extra_ids: list[str] | None = None, metric: Literal['levels', 'pop', 'yoy'] = 'levels', compute_levels: bool = True, data_check: bool = True)

Initialise with user data, FER data or null.

Parameters:

• outturns_data (pd.DataFrame, optional) – DataFrame containing outturn records to add on initialisation. Default is None.

• forecasts_data (pd.DataFrame, optional) – DataFrame containing forecast records to add on initialisation. Default is None.

• load_fer (bool, optional) – Whether to load FER outturns and forecast data on initialisation. Default is False.

• extra_ids (Optional[list[str]], optional) – List of extra label columns (in addition to ‘source’) present in the forecasts data. Default is None.

• metric (str, optional) – Metric to assign to the forecasts if ‘metric’ column is not present or contains null values. Default is ‘levels’. Options: ‘levels’, ‘pop’, ‘yoy’.

• compute_levels (bool, optional) – Whether to automatically transform non-levels forecasts to levels if outturns data is available. When True, forecasts in ‘pop’ and ‘yoy’ metrics will be converted to levels using the available outturns data. Useful if you add ‘pop’ and want to analyse ‘yoy’ forecasts and vice versa. If the transformation fails for specific groups (e.g., due to insufficient historical data), those groups will be skipped with a warning message. Default is True.

• data_check (bool, optional) – Whether to run data checks when adding forecasts. See add_forecasts() for details. Default is True.

add_benchmarks(models: list[str] | str = ['AR', 'random_walk'], variables: str | Iterable[str] | None = None, metric: Literal['levels', 'diff', 'pop', 'yoy'] = 'levels', frequency: Literal['Q', 'M'] | Iterable[Literal['Q', 'M']] | None = None, forecast_periods: int = 13, *, estimation_start_date: Timestamp = None, show_progress: bool = False)

Add benchmark models to the ForecastData instance.

add_fer_data() → None

Load and add FER outturns and forecast data to existing records.

add_fer_forecasts() → None

Load and add FER forecast data to existing records.

add_fer_outturns() → None

Load and add FER outturn data to existing records.

add_forecasts(df: DataFrame, *, extra_ids: list[str] | None = None, metric: Literal['levels', 'pop', 'yoy'] = 'levels', compute_levels: bool = True, data_check: bool = True) → None

Validate new forecasts, transform forecasts and outturns and compute main table and revisions.

Parameters:

• df (pd.DataFrame) – DataFrame containing new forecast records to add.

• extra_ids (list of str, optional) – List of extra label/identification columns (in addition to ‘source’) present in the forecasts data. Default is None.

• metric (str, optional) – Metric to assign to the forecasts if ‘metric’ column is not present or contains null values. Default is ‘levels’. Options: ‘levels’, ‘pop’, ‘yoy’.

• compute_levels (bool, optional) – Whether to automatically transform non-levels forecasts to levels if outturns data is available. When True, forecasts in ‘pop’ and ‘yoy’ metrics will be converted to levels using the available outturns data. Useful if you add ‘pop’ and want to analyse ‘yoy’ forecasts and vice versa. If the transformation fails for specific groups (e.g., due to insufficient historical data), those groups will be skipped with a warning message. Default is True.

• data_check (bool, optional) – Whether to run data checks on the forecasts against outturns. When True, two checks are performed per (source, variable, metric, frequency) group:

• data_check –

  Whether to run data checks comparing forecast values to outturns per (source, variable, metric, frequency, vintage_date) group. When True:

  • Horizon -1 check: if forecast_horizon == -1 rows exist, warns if mean deviation from outturns exceeds 0.5 standard deviations.

  • Scale/mean check (fallback): over overlapping dates, warns if forecast mean differs from outturns by >2 std, or spread ratio >5× (larger/smaller).

  Detects common user errors: wrong metric column, scaling mistakes (e.g. pct*100 instead of pct), or non-real-time vintages.

  Warnings only; never raises errors. Set to False to disable. Default is True.

Notes

Outturns must be added before forecasts (call add_outturns first). All forecasts added to a ForecastData instance must have the same frequency. To work with forecasts of different frequencies, create separate ForecastData instances for each frequency. When compute_levels is True, sufficient historical outturn data is required for transformation, especially for ‘yoy’ metrics which need data from one year prior.

add_outturns(df: DataFrame, *, metric: Literal['levels', 'pop', 'yoy'] = 'levels') → None

Validate new outturns and add them to the outturns dataset

Parameters:

• df (pd.DataFrame) – DataFrame containing new outturn records to add.

• metric (str, optional) – Metric to assign to the outturns if ‘metric’ column is not present or contains null values. Default is ‘levels’. Options: ‘levels’, ‘pop’, ‘yoy’.

clear_filter() → None

Reset the forecasts, main and revisions tables to include all original data.

copy()

Return a deep copy of the ForecastData object.

create_pseudo_vintages(fill_to: str, vintage_frequency: Literal['M', 'Q'] = 'Q') → None

Create pseudo vintages for outturns.

Starts from the earliest available vintage in the data and fills backward to fill_to.

This method computes the publication lag from existing data and creates a full vintage structure where each vintage contains all data available at that point in time. A vintage at date X contains all data up to (X - publication_lag).

Parameters:

• fill_to (str) – The earliest vintage date to create (i.e. how far back to fill). Format ‘YYYY-MM-DD’. Vintages are generated from this date up to the earliest existing vintage in the data.

• vintage_frequency (str, optional) – Frequency at which to create vintages. Default is ‘Q’ (quarterly). Options: ‘M’ (monthly), ‘Q’ (quarterly).

Notes

• Computes publication lag per variable from existing data (max_vintage - max_date)

• Expands the dataset by creating multiple vintage records for each data point

• Each vintage V includes all data points D where (D + lag) <= V

• Requires outturns to already have vintage_date values to compute the lag

property df: DataFrame

Get the main DataFrame.

filter(start_date: str | None = None, end_date: str | None = None, start_vintage: str | None = None, end_vintage: str | None = None, variables: str | list[str] | None = None, metrics: list[str] | None = None, sources: str | list[str] | None = None, frequencies: str | list[str] | None = None, custom_filter: Callable[[DataFrame], DataFrame] | None = None)

Filter the forecasts and main tables to only include data within specified date and vintage ranges, and optionally by variables, metrics, sources, or a custom filter.

Parameters:

• start_date (str, optional) – Start date to filter forecasts (inclusive). Format ‘YYYY-MM-DD’. Default is None in which case the analysis starts with the earliest date.

• end_date (str, optional) – End date to filter forecasts (inclusive). Format ‘YYYY-MM-DD’. Default is None in which case the analysis ends with the latest date.

• start_vintage (str, optional) – Start vintage date to filter forecasts (inclusive). Format ‘YYYY-MM-DD’. Default is None in which case the analysis starts with the earliest vintage.

• end_vintage (str, optional) – End vintage date to filter forecasts (inclusive). Format ‘YYYY-MM-DD’. Default is None in which case the analysis ends with the latest vintage.

• variables (Optional[Union[str, list[str]]] = None) – List of variable identifiers to filter. Default is None (no filtering).

• metrics (Optional[list[str]] = None) – List of metric identifiers to filter. Default is None (no filtering).

• sources (Optional[Union[str, list[str]]] = None) – List of source identifiers to filter. Default is None (no filtering).

• frequencies (Optional[Union[str, list[str]]] = None) – List of frequency identifiers to filter. Default is None (no filtering).

• custom_filter (Callable[[pd.DataFrame], pd.DataFrame], optional) – A custom filtering function that takes a DataFrame as input and returns a filtered DataFrame. Default is None. Custom filters should use ‘vintage_date_forecast’ as the column name.

filter_fer() → None

Filter the main dataset to only include specific variable-metric and model combinations

property forecast_required_columns: list[str]

Get the required columns list to help the user.

property forecasts: DataFrame

Get forecasts.

property id_columns: list[str]

Get identification / labelling columns.

merge(other: ForecastData, compute_levels: bool = True) → ForecastData

Merge another ForecastData instance into this one.

Parameters:

• other (ForecastData) – Another ForecastData instance to merge with this one.

• compute_levels (bool, optional) – Whether to automatically transform non-levels forecasts from other to levels if outturns data is available. When True, forecasts in ‘pop’ and ‘yoy’ metrics will be converted to levels using the available outturns data. Useful if you add ‘pop’ and want to analyse ‘yoy’ forecasts and vice versa. If the transformation fails for specific groups (e.g., due to insufficient historical data), those groups will be skipped with a warning message. Default is True.

Returns:

Updated ForecastData instance containing merged data from both instances.

Return type:

ForecastData

property outturn_required_columns: list[str]

Get the required columns list to help the user.

property outturns: DataFrame

Get outturns.

run_dashboard(from_jupyter: bool = False, host='127.0.0.1', port=8000)

Run the Shiny dashboard with the current data.

Parameters:

• from_jupyter (bool, optional) – Whether to run the dashboard within a Jupyter notebook. Default is False.

• host (str, optional) – Host address for the dashboard server. Default is “127.0.0.1”.

• port (int, optional) – Port number for the dashboard server. Default is 8000.

summary()

Print a summary of the forecast and outturns datasets.

For each dataset, displays: - Number of variables - List of variables with their properties - For each variable: frequency, date range, and first vintage date

forecast_evaluation.data.create_sample_forecasts() → DataFrame

Create sample forecasts DataFrame for testing and examples.

forecast_evaluation.data.create_sample_outturns() → DataFrame

Create sample outturns DataFrame for testing and examples.