scores

Description

The scores configuration function is designed to facilitate comprehensive evaluation of the forecasting model's performance. It enables the calculation of various metrics at custom resolutions during the scoring steps of a pipeline. This function is crucial for assessing the accuracy and effectiveness of the forecasted outcomes, allowing users to fine-tune the model based on empirical data.

By specifying particular metrics and their respective resolutions, users can tailor the evaluation to reflect the specific needs and goals of their forecasting tasks. Whether it's determining the root mean square error (RMSE), mean absolute error (MAE), or any other statistical measure, the score function provides the flexibility to apply these metrics where they are most needed, ensuring a thorough analysis of the model's performance across different segments or periods.

This targeted approach to scoring not only enhances the understanding of model behavior but also supports continuous improvement by highlighting areas where the model may be optimized further.

Options

Scores can have multiple entries, one for each custom resolution that you want to evaluate.

For each custom resolution, you can define three parameters:

Parameter	Usage	Type
resolution	This parameter determines the specific grouping or segmentation of the data for which the score will be calculated. Each unique value within the chosen resolution serves as a basis for grouping the data.	List of columns
aggregation	Before metrics are calculated, the data is grouped by each unique value within the specified resolution. This aggregation ensures that the scoring metrics are reflective of performance within each distinct group, providing detailed insight into model behavior across different data segments.	List of columns
filters	This parameter allows for the exclusion of certain data rows from the scoring calculations. Rows are excluded based on the specified conditions; typically, rows where a designated column's value is False are not included in the metric calculations. This filtering is	List of boolean columns

Steps impacted

score

Example

scores:
        custom_resolutions:
                per_shop_type:
                        resolution:
                        - day
                        aggregation:
                        - shop_type
                        filters:
                        - to_score
        
                per_item_category:
                        resolution:
                        - day
                        aggregation:
                        - item_category
                        filters:
                        - to_score