Design an honest forecast accuracy evaluation
Set up a train/test evaluation with scale-appropriate error measures instead of judging models by in-sample fit.
A forecast evaluation protocol with a leakage-free split, benchmarks, justified error measures, and a single deciding metric.
Role
forecasting analyst
Task
Design the accuracy evaluation for forecasting {{series_name}} at horizon {{horizon}}. Specify the training and test split, the candidate methods including at least one naive benchmark, and the error measures to report, choosing among absolute, squared, percentage, and scaled measures with a justification tied to the properties of {{series_name}}. State which measure decides the comparison and why in-sample fit will not be used.
Context
The series has {{seasonality}} seasonality, {{observation_count}} observations, and values that {{zero_behaviour}}, which constrains the usable error measures.
Inputs
- Series description: {{series_name}}, {{observation_count}} observations, {{seasonality}} seasonality
- Forecast horizon {{horizon}} and how forecasts will be used
- Whether the series can be zero or negative: {{zero_behaviour}}
- Candidate forecasting methods under consideration
Constraints
- Keep the test set out of every stage of model development and say so explicitly
- Include a naive or seasonal-naive benchmark in the comparison
- Justify or exclude percentage error measures based on {{zero_behaviour}}
- Distinguish residuals from out-of-sample forecast errors in the reporting plan
Output Format
An evaluation protocol with split definition, method list, a measures table (measure, formula in words, why it fits or fails here), and the deciding metric.
Quality Criteria
- Split protocol prevents test-set leakage
- Benchmark method included
- Measure choice argued from series properties, not habit
- Residual/forecast-error distinction stated