Benchmark a probabilistic model against a base-rate forecaster
Show whether a probabilistic model beats always predicting the base rate before it is trusted in production.
A model-versus-base-rate skill comparison with an imbalance explanation and a justified shipping threshold.
Role
machine learning evaluation lead
Task
Compare the probability forecasts from {{model_name}} on {{event_type}} against a reference forecaster that always predicts the observed base rate of {{base_rate}}. Compute the skill of the model relative to that reference, explain why the raw score is misleading when the outcome is rare, and state the skill threshold below which the model should not ship.
Context
The dataset is imbalanced, the raw score already looks low, and a stakeholder has read it as strong performance.
Inputs
- Model predictions and outcomes for {{event_type}}
- Observed base rate: {{base_rate}}
- Class balance and evaluation period
- Deployment decision criteria
Constraints
- Score the base-rate reference explicitly and show both scores
- Express skill relative to the reference and interpret zero and negative values
- Explain the effect of the base rate on the irreducible component
- Recommend a shipping threshold and justify it against the decision being supported
Output Format
A two-row score comparison, the skill calculation, an explanation of the base-rate effect, and a shipping recommendation with threshold.
Quality Criteria
- Reference forecaster scored, not assumed
- Skill interpretation covers zero and negative cases
- Imbalance effect explained quantitatively
- Threshold tied to the downstream decision