Unit 3: Evaluating Models & Confusion Matrix
Complete study notes, Train-Test split methods, Overfitting intuition, Confusion Matrix (TP, TN, FP, FN), and step-by-step calculation formulas for Accuracy, Precision, Recall, and F1-Score with all official CBSE case studies.
Importance of Model Evaluation & Train-Test Split
In the AI project cycle, after Problem Scoping, Data Acquisition, Data Exploration, and Modelling, Model Evaluation is like issuing a school report card to your algorithm. It provides constructive feedback to identify strengths, weaknesses, and suitability before deploying the AI in real life.
The Train-Test Split Method
The dataset is divided into two distinct subsets:
- Training Dataset (Typically 70% to 80%): Used by the algorithm to adjust weights and learn patterns.
- Testing Dataset (Typically 20% to 30%): Completely unseen data fed to the model to assess its generalisation capability.
Why Not Test on Training Data? (Overfitting)
If an AI model is evaluated on the exact same data it was trained on, it simply memorizes all data points without understanding the underlying logic. This phenomenon is called Overfitting. The model achieves 100% training accuracy but fails completely when deployed in the real world.
Accuracy vs Error: Formulas & Numerical Calculations
Absolute Error
|Actual − Predicted|
Quantifies difference without sign
Error Rate
Error / Actual
Relative error as a fraction
Accuracy %
(1 − Error Rate) × 100%
Proportion of correct estimation
Handbook Activity 1: House Price Prediction Accuracy Table (Page 66)
| Predicted | Actual | Abs Error | Error Rate | Accuracy | Accuracy % |
|---|---|---|---|---|---|
| 391k | 402k | 11k | 11/402 = 0.027 | 1 - 0.027 = 0.973 | 97.3% |
| 453k | 488k | 35k | 35/488 = 0.072 | 1 - 0.072 = 0.928 | 92.8% |
| 125k | 97k | 28k | 28/97 = 0.289 | 1 - 0.289 = 0.711 | 71.1% |
| 871k | 907k | 36k | 36/907 = 0.040 | 1 - 0.040 = 0.960 | 96.0% |
| 322k | 425k | 103k | 103/425 = 0.242 | 1 - 0.242 = 0.758 | 75.8% |
| Overall Model Mean Accuracy: | 86.6% | ||||
The Confusion Matrix & Classification Metrics
A Confusion Matrix is a 2×2 contingency table presenting the actual ground-truth values against the predicted outcomes generated by a classification model:
Classification Accuracy
Accuracy = (TP + TN) / (TP + TN + FP + FN)
Ratio of total correct predictions out of all predictions made.
Precision (When FP is costly)
Precision = TP / (TP + FP)
Measures accuracy of positive predictions. Crucial for Rocket Launch weather & Email Spam detection.
Recall / Sensitivity (When FN is dangerous)
Recall = TP / (TP + FN)
Measures proportion of actual positives correctly caught. Vital for Cancer / COVID-19 diagnosis & Fraud detection.
F1-Score (Harmonic Mean)
F1 = 2 × (Precision × Recall) / (Precision + Recall)
Harmonizes precision and recall for unbalanced datasets where both false positives and false negatives matter.
The 90% Accuracy Faulty Model Paradox
Imagine testing 1000 students where 900 pass (Yes) and 100 fail (No). A completely broken “lazy” model predicts “Yes” for all 1000 students without looking at any data!
$TP = 900, FN = 0, FP = 100, TN = 0$
$Accuracy = (900 + 0) / (900 + 0 + 100 + 0) = 900 / 1000 = 90\%$
Despite doing zero learning, the model claims 90% accuracy! But its Recall on failing students is 0%!
Test Yourself: Official MCQs & Answer Key
Directly from Pages 81–83 of the Facilitator Handbook with official answers from Page 181. Click an option to test your knowledge!
Q1. In a medical test for a rare disease, out of 1000 people tested, 50 actually have the disease while 950 do not. The test correctly identifies 40 out of the 50 people with the disease as positive, but it also wrongly identifies 30 of the healthy individuals as positive. What is the accuracy of the test?
Q2. A student solved 90 out of 100 questions correctly in a multiple-choice exam. What is the error rate of the student's answers?
Q3. In a spam email detection system, out of 1000 emails received, 300 are spam. The system correctly identifies 240 spam emails as spam, but it also marks 60 legitimate emails as spam. What is the precision of the system?
Q4. In a binary classification problem, a model predicts 70 instances as positive out of which 50 are actually positive. What is the precision/recall of the model?
Q5. In a sentiment analysis task, a model correctly predicts 120 positive sentiments out of 200 positive instances. However, it also incorrectly predicts 40 negative sentiments as positive. What is the F1 score of the model?
Q6. A medical diagnostic test is designed to detect a certain disease. Out of 1000 people tested, 100 have the disease, and the test identifies 90 of them correctly. However, it also wrongly identifies 50 healthy people as having the disease. What is the precision of the test?
Q7. A teacher's marks prediction system predicts the marks of a student as 75, but the actual marks obtained by the student are 80. What is the absolute error in the prediction?
Q8. The goal when evaluating an AI model is to:
Q9. A high F1 score generally suggests:
Q10. How is the relationship between model performance and accuracy described?
Complete Numerical Case Studies (Pages 84–85)
Step-by-step Confusion Matrix calculations for Accuracy, Precision, Recall, and F1-Score.
Case Study 1: Spam Email Detection System (1000 Emails)
Data: TP = 150 (spam correctly caught), FP = 50 (ham mislabeled as spam), TN = 750 (ham correctly caught), FN = 50 (spam missed).
Case Study 2: Credit Loan Default Prediction (1000 Applicants)
Data: TP = 90 (defaulters caught), FP = 40 (non-defaulters blocked), TN = 820 (good loans), FN = 50 (defaulters missed).
Case Study 3: Fraudulent Transaction Classifier (1000 Transactions)
Data: TP = 80 (fraud caught), FP = 30 (legit flagged), TN = 850 (legit clear), FN = 40 (fraud missed).
Case Study 4: Disease Diagnosis Diagnostic Test (1000 Patients)
Data: TP = 120 (sick detected), FP = 20 (healthy false alarm), TN = 800 (healthy confirmed), FN = 60 (sick missed).
Case Study 5: Inventory Stockout Prediction (1000 Products)
Data: TP = 100 (out-of-stock predicted), FP = 50 (in-stock predicted out), TN = 800 (in-stock), FN = 50 (out-of-stock missed).
Unit 3 Key Formulas & Decision Rules
Frequently Asked Questions
Book a Free Demo Class
Score 100% in Class 10 AI evaluation numericals! Master Confusion Matrices, Precision, Recall, and F1 calculations with expert teachers.