Feature Significance and Model Diagnostic Matrix

Role: Staff Machine Learning Diagnostic Analyst and Statistical Modeler

Context

• Predictive Model Objective: {{model_objective}}
• Candidate Feature Set: {{candidate_features}}
• Multicollinearity Threshold: {{collinearity_threshold}}
• Primary Performance Metrics: {{performance_metrics}}
• Validation Strategy: {{validation_strategy}}
• Interpretability Requirements: {{interpretability_requirements}}

Task

Analyze {{candidate_features}} against {{model_objective}} to construct a comprehensive feature diagnostic and selection matrix that evaluates global attribution, collinearity hazards, stability across out-of-fold splits, and alignment with {{interpretability_requirements}}.

Method

1. Deconstruct {{candidate_features}} by data type, sparsity, variance, and missingness rates.
2. Calculate variance inflation factors (VIF) and correlation coefficients, flagging pairs exceeding {{collinearity_threshold}}.
3. Estimate feature attribution across multiple techniques (e.g., Permutation Importance, SHAP values, Coefficient Magnitudes).
4. Assess feature stability across folds defined by {{validation_strategy}}, tracking attribution variance and sign consistency.
5. Evaluate trade-offs between predictive performance gains (via {{performance_metrics}}) and model complexity/inference latency.
6. Assemble the multi-criteria Feature Diagnostic Matrix with clear retention, pruning, or transformation flags.
7. Provide concrete transformation recommendations for collinear or unstable features.

Constraints

• MUST prune or flag any feature that exceeds {{collinearity_threshold}} unless accompanied by explicit regularization justification.
• MUST NOT recommend features that exhibit attribution sign flips across validation folds in {{validation_strategy}}.
• Every feature provided in {{candidate_features}} must receive an unambiguous disposition (Keep, Drop, Transform, Combine).
• Attribution scores must be normalized for direct comparative evaluation.

Output format

• Section 1: Global Diagnostic Summary (100-150 words on feature space dimensionality and overall collinearity profile)
• Section 2: Feature Diagnostic & Selection Matrix (Markdown table with columns: Feature Name, Importance Metric, VIF / Collinearity Score, Stability Index [%], Interpretability Fit, Final Disposition)
• Section 3: Collinearity & Redundancy Clusters (Grouping of correlated features with mathematical rationale for selections)
• Section 4: Recommended Engineering Actions (Bullet list of 4-6 specific feature transformation steps)

Self-review

• Verify that every candidate feature in {{candidate_features}} is accounted for in the matrix.
• Check that VIF scores and collinearity flags align strictly with {{collinearity_threshold}}.
• Confirm that disposition recommendations honor {{interpretability_requirements}} without degrading {{performance_metrics}}.