Can Symptom Checkers be bad for your health?

The number of symptom checkers designed for patient use has mushroomed over the last few years.

Patients use these tools to try and identify health symptoms, so the choice of symptom checker is very important. The problem is that most tools of this kind look similar, so how do you know which one to trust with your life?

To answer this question, you need to understand how these tools are built, because there are some key differences under the hood. 

What is a symptom checker?

Symptom checkers are designed to help people without medical experience understand the potential causes of their symptoms and develop a clearer idea of what to do next. However, there are about 10,000 known diseases or potential causes and an almost infinite number and combination of symptoms.  Add to this the enormous variety of ways in which patients can present and you’ll begin to understand why such a complex task can only be handled by sophisticated software. 

Even though today’s symptom checkers look similar on the surface, they’re underpinned by two very different technologies – we really are talking chalk and cheese!

The two types of symptom checker

Modern symptom checkers are developed with either:

- rules; or

- machine learning.

Nearly all patient symptom checkers are based on classic rules-based technology. This means that they work from a set of rules that are created and pre-defined by the software developer. 

For rules-based symptom checkers, the key rule that needs to be created is the relationship of a symptom to a disease; in other words, the likelihood of a particular symptom being the cause of that disease. The problem is that there is hardly any published evidence of the probability a symptom has of causing a specific disease. Consequently, developers have to assemble a group of clinicians to provide their opinion instead. Inevitably there’ll be disagreements, resulting in compromises. As a result, the probabilities eventually defined by the developer will often be far from accurate.  And if the patient has multiple symptoms, those defined probabilities are multiplied and the potential for error grows bigger and bigger. 

Rules-based symptom checkers aim to offer a single diagnosis, or a short list ranked by clinical probability which is probably just what the patient wants. But when you take into consideration the potential for diagnostic error, this isn’t really credible and becomes very dangerous. Imagine a patient with pain down her left arm; it could be muscular strain or tennis elbow. But what if it was a precursor to a heart attack? 

For just one disease model, there could be 1,500 or more probabilities which need to be agreed by the clinicians and defined by the developer. COVID-19 is an example where scientists estimate well over 100 different symptoms (and counting). Add to this details like age, location and gender, and the number of variables are enormous. This makes it very labour intensive for the developer to create a model for a disease or symptom - let alone keep it up to date and is why most symptom checkers will cover just a few hundred common diseases and symptoms - just 5-10% of the number of known diseases.

For a job this complex, machine learning is the only feasible option. The key difference with machine learning is that the developer doesn’t define the rules that govern the system but instead trains the system to learn about the ways in which diseases present. The computer then calculates, defines and maintains the relationship between symptoms and diseases using the constantly growing knowledge from its training. 

This is true artificial intelligence and works in a similar way to a doctor learning medicine. A doctor learns how diseases present and, over many years, builds up a memory bank of diseases that he or she has experienced or read about. When the doctor takes a patient’s history and examines them, he or she then has a set of signs and symptoms which can be matched against his or her memory bank of diseases. 

This works very well 90% of the time but relies on the doctor’s memory, the ability to recall it during a patient interview, and the ability to remain objective. Most doctors deal in a universe of 150-200 diseases; no doctor can be expected to know and recall the typical and atypical presentations of 10,000 diseases.   

The most important element of a machine learning symptom checker is how it has been trained. If it has been given poor quality or limited data, then it’ll be less accurate in the way it calculates the relationships between symptoms and diseases. The only symptom checker that currently uses machine learning is the Isabel Symptom Checker which, over the past 20 years, has been trained on 6,000 diseases, using evidence-based published information about how diseases present. 

How can you tell which technology a symptom checker uses?

It’s easy to tell if a symptom checker is rules-based because it always starts by asking you something like, “what’s your most important symptom?”. This seemingly innocuous question is really quite dangerous as it can completely change your presentation. If you had fever, a headache, a sore throat, shortness of breath, diarrhoea and vomiting, for example, how would you know which of these was the “most important”? 

The reason for this question is that the rules-based system needs to get you onto one of its pre-designed pathways so that it can start asking you questions about your other symptoms. At this point you’ll really need to be patient as there can be 30 to 50 questions that follow (we’ve even heard of patients being asked more than 100 questions!). 

Another way to reveal the technology behind the tool is to hit it with three symptoms, but enter them in a different order each time. You’ll be amazed – and concerned – by how different the results can be.

Some tools include triage functionality that’s designed to give you advice on where to go if you need care. The main purpose of triage is to help you decide whether what you have is serious enough to warrant a trip to the emergency department or more suited to a general doctor’s appointment. The problem with most rules-based systems is that the advice is normally tied to a particular diagnosis. This means that you essentially need to pick a diagnosis (i.e., diagnose yourself!) before receiving the advice.  

Think back to the example of pain in the left arm. Had you chosen ‘arm strain’, you’d have stayed at home, instead of going to the emergency department with what could potentially be a heart attack.

Finally, a machine learning system like Isabel is much more straightforward and easier to use; you can enter all of your symptoms in one go and in a conversational style. This shouldn’t take more than a minute and Isabel will even give you triage advice after asking you seven key questions. This means that the triage advice is based on your overall clinical picture rather than on one diagnosis you were forced to select from a list. 

Choosing the right symptom checker isn’t easy, but a quick look at the technology behind the tool should be enough to tell you whether or not you’re going to receive reliable advice. 

Remember - Isabel is the only tool which is also used by doctors, so if it’s good enough for them, patients can definitely trust it, too!