Machine Learning Evaluation

Definition

The process of assessing a model’s output quality by comparing predictions to actual results using statistical benchmarks.

Purpose

To ensure the trained model performs accurately, generalizes well, and avoids overfitting or underfitting.

Classification Metrics

Accuracy: Measures the share of total instances where the model's output aligns with actual outcomes.

Example: 90 out of 100 emails correctly identified as spam or not.

Precision: Indicates how many of the flagged positives were genuinely correct according to ground truth.

Example: Of 30 predicted spam emails, 25 were actually spam.

Recall: Correct positive predictions among all actual positives.

Example: Model found 25 of 28 total spam emails.

F1-Score: Represents a balanced blend of precision and recall, emphasizing consistency in positive prediction performance.

Example: Useful when spam vs. non-spam is imbalanced.

Regression Metrics

Mean Absolute Error (MAE): Average of absolute prediction errors.

Example: Predicting house prices with an average $3,000 difference.

Mean Squared Error (MSE): Average squared error to penalize large mistakes.

Example: Predicting house prices where one estimate is off by $100,000 will greatly raise the Mean Squared Error due to squaring the large difference.

Root Mean Squared Error (RMSE): Square root of MSE; more interpretable.

Example: Easier to compare in same units as original prediction

Evaluation Techniques

Train/Test Split: Separates data to evaluate unseen predictions.

Cross-Validation: Rotates data through different test folds for reliable performance.

Visualization Tools

Confusion Matrix: Displays true vs. predicted classifications.

Example: Helps understand types of misclassification (e.g., spam marked as not-spam).

ROC-AUC Curve: Measures classifier quality across thresholds.

Example: Shows how well the model separates classes.