Model performance metrics, leakage and data alerts

What the exam expects

• Interpret AUC (binary and multiclass) and R2 (numeric) on a model's Performance overview and say whether the model is ready to deploy.
• Read the evaluation statistics behind a recommendation: accuracy, MCC, the threshold trade-off, precision and recall, and numeric error.
• Recognise data leakage from a suspiciously strong metric or predictor, and name the variable to remove.
• Propose more or better explanatory data when predictors are weak, including historical values the source doesn't keep.
• Turn each data alert into an action: proxy variable, multicollinearity, outliers, imbalanced distribution and leakage alerts.
• Know where alerts are reviewed and what happens when you change a setting or ignore one.

Key terms: AUC (area under the curve, the rate of correct classification; 0.5 is random guessing); R2 (a regression model's ability to explain variation in the outcome, from zero to one); data leakage (training data that contains the information you are trying to predict); proxy variable (a variable highly correlated with a sensitive variable); multicollinearity (two or more highly correlated variables with a duplicate impact on the outcome).

Key facts

Check yourself