Score and decompose a set of probability forecasts
Separate calibration failure from lack of discrimination when judging a probabilistic forecaster.
A probability-forecast scoring report with a three-way decomposition, per-bin calibration table, and a weakness diagnosis.
Role
forecast evaluation analyst
Task
Evaluate {{forecast_count}} probability forecasts of {{event_type}} against their outcomes. Compute the mean squared error of the probabilities, then decompose it into a calibration term, a discrimination term, and the irreducible term set by the base rate. Bin the forecasts, tabulate predicted versus observed frequency per bin, and diagnose whether the forecaster is miscalibrated, indecisive, or both.
Context
The forecaster is {{forecaster_name}} and stakeholders currently judge them by counting how often the most likely outcome happened.
Inputs
- Forecast/outcome pairs: {{forecast_count}} records
- Event definition and resolution criteria for {{event_type}}
- Observed base rate over the period
- Bin scheme or the number of bins to use
Constraints
- Report the overall score and all three decomposition components
- Show the per-bin table of mean prediction versus observed frequency
- Interpret deviations as overconfidence or underconfidence with direction stated
- Do not treat a low overall score as proof of good calibration
Output Format
A scoring summary, a decomposition table, a per-bin calibration table, and a diagnosis paragraph naming the dominant weakness.
Quality Criteria
- All three components reported with the base rate stated
- Per-bin table lets a reader recompute calibration
- Over/underconfidence direction identified
- Diagnosis distinguishes calibration from discrimination