Set up rolling-origin cross-validation with residual diagnostics
Compare forecasting methods over many origins and confirm the winner leaves no exploitable structure in its residuals.
A rolling-origin cross-validation specification plus a residual diagnostic checklist tied to forecasting consequences.
Role
time series modeller
Task
Specify a rolling-origin cross-validation for {{series_name}}: the initial training window, the step size, horizon {{horizon}}, and how errors are aggregated across origins. Then define the residual diagnostics for the selected model, autocorrelation inspection, mean-zero and constant-variance checks, distributional check, and a portmanteau test with its lag choice for {{seasonality}} data, and state what each failure would imply for the forecasts.
Context
A single holdout comparison produced a near-tie between {{method_a}} and {{method_b}}, and the team wants a more stable comparison plus evidence the winner is not leaving signal behind.
Inputs
- Series description and length for {{series_name}}
- Methods to compare: {{method_a}} and {{method_b}}
- Horizon {{horizon}} and seasonal period for {{seasonality}}
- Operational constraint on retraining frequency
Constraints
- Define the origin schedule explicitly, including the number of evaluation points
- Aggregate errors across origins with the chosen measure and report dispersion, not just the mean
- Set the portmanteau lag count according to whether the data are seasonal
- For each residual assumption, state the forecasting consequence of violating it
Output Format
A cross-validation specification, an aggregation plan, and a residual diagnostic checklist mapping each check to its consequence.
Quality Criteria
- Origin schedule fully specified and reproducible
- Error dispersion across origins reported
- Lag choice justified by seasonality
- Each residual failure mapped to a concrete forecasting consequence