I’ve mixed up the different machine learning metrics while explaining them on more than one occasion. I found that quickly drawing a confusion matrix really helps until the concept sticks. Here’s a short conversation between a mother and her 12-year-old son, where she explains machine learning metrics in simple terms.
Raghav was rigorously preparing for a test that was less than a week away. His mother was making sure he was studying everything as per the syllabus, and decided to test him after he studied each chapter.
Today, he went over the entire chapter in a hurry because of how long and boring the subject was, and went to his mother to be quizzed. She gave him a set of 20 true/false questions. He started answering them and saw that most of the answers were true; for the ones he wasn’t sure about, he guessed True, since most of the earlier answers had been True. When his mother cross-questioned him on the questions he got wrong, he confessed that he’d guessed because most answers were true.
Amma: “Raghav, you can’t guess answers during exams. Does this mean you didn’t actually study the chapter well enough?”
Raghav: “Ma, but it’s so difficult, the chapter is so long. I didn’t learn it properly, so I guessed some.”
Amma: “Okay, here’s the deal, take a small break. I’ll first tell you something about how machines learn and perform when tested, you might find this interesting.”
Raghav: “Sure, tell me! I love computer science, this should be fun.”
Amma: “Imagine you have 100 bank transactions and you feed them into a machine. You want the machine to learn from this data so that it can correctly tell a genuine transaction from a fraudulent one in the future.”
Amma: “In these 100 rows, you already know that only 5 are fraudulent; the rest are genuine.”
Amma: “Say the machine has learnt from this, and when we test it, its accuracy is 95%. Would you say the model is classifying the transactions correctly?”
Raghav: “Obviously! It’s giving you an accuracy of 95%. Doesn’t that mean it’s performing well?”
Amma: “Well, not obviously. Here’s what happens when the classes are imbalanced…”
Raghav: “What’s an imbalanced class?”
Amma: “I’m just getting there. Of these 100 rows, how many are fraudulent and how many aren’t? You know ratios and proportions, tell me.”
Raghav: “Umm, it’s 5:95.”
Amma: “Exactly. Do you see that the two classes, fraudulent and not fraudulent, are imbalanced?”
Raghav: “Yes.”
Amma: “When that happens and we don’t teach the machine to pay close attention to this imbalance, it learns poorly. When it starts predicting on new bank transactions, chances are it’s just guessing the majority answer, ‘genuine’, instead of learning patterns from the data.”
Raghav: “Ohh, I see what you’re getting at. It performs badly because it’s guessing the most common answer, like I did, and not actually learning.”
Amma: “Something like that.”
Raghav: “But wait, if the accuracy is really good, how do you prove that the machine is guessing?”
Amma: “Before I get there, remember that Positive means fraudulent transactions and Negative means non-fraudulent ones.”
Confusion Matrix by Author
Raghav: “Got it.”
Amma: “There are a few other measures that tell us this. Precision tells us, ‘Of all the Positive, or fraudulent, predictions the machine made, how many were actually Positive?’ And Recall tells us, ‘Of all the fraudulent transactions, how many did the model catch?’”
Amma: “So back in our example, we had 100 transactions, of which 95 were non-fraudulent. The model mostly kept guessing ‘genuine’ on new transaction data and missed every chance to catch a fraudulent case. Had it learnt how important it is to catch even those 5, it would perform a lot better on new data.”
Raghav: “Right, so accuracy wasn’t enough, because we were only looking at how often the model was right and not at what it got wrong. And Recall is the one that tells us about the missed chances.”
Amma: “True. And Precision tells us how reliable the fraud predictions are: it guards against false alarms, when a model predicts that a case is fraudulent when it’s actually genuine.”
Amma: “There’s just one more measure, called Specificity. It tells us, ‘Of all the negative, that is non-fraudulent, cases, how many did the machine identify correctly?’”
Raghav: “Amazing! It’s interesting how these machines make human-like mistakes.”
Amma: “Some ideas in machine learning were inspired by how our brains learn, so yes, some of it looks very similar to how humans think and learn.”
Raghav: “So cool! Will you teach me more about machine learning?”
Amma: “Yes, but first you’ll have to do well on your test. And don’t even think about guessing your way through it: really sit down, learn the concepts, practise, and test yourself every now and then.”
Raghav: “Yes, Amma, I’ll do that.”
In your work, which mistake costs more: a missed positive or a false alarm?
This was Day 1 of my 30 Days of Data Science writing challenge. Writing an ML concept as a conversation was a fun experiment. Would you like to see more posts in this style?
Top comments (0)