Take a quick glance at this meme. You've seen something like this before, haven't you? AI agreeing with something that is obviously and badly wrong. “That's correct, it was toxic.” In a tone of confidence similar to that used to tell you the capital of France. It's funny; the confidence is so far removed from correctness, and it's like that's the whole point.
The thing is, though. Funny when it's a food that's poisoned. But is it still funny when it's your bloodwork
?
That kind of tone that's certain, slick, relaxed, is already residing in actual medical gear now. Not hypothetically. Right now.
To start with, the first problem is so easy: "AI can sound right when it's wrong, and in healthcare, that's wrong when it's right. Take the time to think the other way around. You are a doctor who has just been on a twelve-hour shift, or a patient who just wants someone, anyone, to tell him or her what's going on. An AI provides you with an answer; it's clean, it's confident, and it's without hesitation. Do you really plan to interrogate it? Honestly evaluate yourself. Last time you double-checked an AI, rather than taking its word for it because it sounded certain?Last time you checked a point an AI had made, rather than just believing it because it seemed like it had a big idea of itself?
Now imagine yourself as that student taking the exam. Don't read about it; be it. A person in the "scrubs" is telling you what "the system" has marked, and you're nodding your head, right? What else would you have to do? Demand a second opinion from the algorithm? When it comes to your own body, and a decision is at stake, that confidence you laughed at 30 seconds prior doesn't feel funny anymore.
Which leads directly into the second problem, the heavier one: When it goes wrong, who owns that?
State that an algorithm causes a negative consequence. Who's responsible? The doctor who believed in a tool that they were told to believe in? The hospital where the software was bought and implemented in all departments? The builder of it, away from the real examination room? In reality, it's often the response of "a little bit of all these people, so no one really. Now suppose you are that person giving the explanation to a member of your family. Say out loud to someone who just lost a person they loved: "Well, the algorithm suggested it". But when no one seems to have committed a mistake, is it still an accident? Or could it be just the way we've settled for doing it these days, one tool at a time, not even voting on it?
That's really all there is to it, right? The meme's success was that the stakes were fake and easily verifiable: poison or not, you would know within the hour, either way. Medicine doesn't give you a safety net of this kind. When you do not have an immediate, evident answer as to whether the confident answer was correct or not, you must play it by ear as to how much trust to place in the tone of voice emanating from a machine.
So, that's where I'll leave you. The next time you hear someone, whether a doctor or a computer, state with absolute certainty that he/she knows what is happening, ask yourself what he or she is basing that certainty on. The reason for this is that you're right: it was a joke about a snack, and it was poisonous. But somewhere out there, right now, a similarly worded sentence is being repeated about somebody's diagnosis, but without the coolness, and it isn't being laughed out loud in the room.
Top comments (0)