Virtual Triage: Do more questions lead to better patient outcomes?

One of the common misconceptions related to virtual triage / symptom checker tools is that the more questions the system asks, the more accurate the list of possible conditions / triage level recommendations will be and so, overall, the conclusion is that it is more thorough and, therefore, a better tool. If you are looking to provide your patients with a symptom checker/triage tool or are a patient looking to use one, this may intuitively make sense, but you would be very wrong. For virtual triage tools, more really is less.

Most symptom checker tools operate using decision tree / rule-based logic and often ask 30-50 questions and need several minutes to use. As a hospital system you might think that the patient has plenty of time and it’s an important step so why wouldn’t they be prepared to put up with up to 50 questions if the final advice was going to be more accurate.

That assumption misses some key facts. First, the patient or parent is going to be feeling unwell and likely worried when they are using the application. While in this state you are really asking them for much more than just 5 minutes of their time - you are really burdening them with 30 to 50 separate decisions. You are asking them to decide what their most important symptom or chief complaint is, how bad their headache is, where it is, when it happens, how long have they had it etc., etc., just at the time when they feel least like making endless decisions. In many cases the result also forces them to select a specific condition they think they might have in order to get care advice! This can be especially burdensome for parents seeking care for children.

Think how you feel when you phone a company, for example, and are asked to choose from 6 options, then when you choose one there are another 5 options and when you choose one and finally think you may be getting somewhere you are presented with yet another 6 options. You start to curse the company, shouting that you just want to speak to a human being! And this is how you feel when you are healthy, not worried and feeling terrible.

To some patients this might be bearable so long as they believe the result at the end of this process will be better advice. Sadly, that faith will be misplaced.

Decision tree or rules-based symptom checkers are based on the false premise that they can mimic a good real doctor by first assessing what the patient’s most important symptom is then by teasing out the other symptoms with questions. However, practicing doctors don’t mechanically follow a script but instead take numerous mental short cuts by focusing and jumping to the key issues. All their questions are relevant.

They are also able to cope with the almost infinite ways that patients can describe their symptoms and are not restricted to only understanding the limited number of symptoms a system has been programmed with. Due to the complexity of building the models, decision tree symptom checkers typically cover only a few hundred common symptoms and diseases. Bear in mind, there are over 10,000 diseases and an infinite way in which patients present and describe their symptoms.

Sadly, computer systems attempting to mimic a human process very rarely work as we always underestimate what we do as humans. The best systems are those that are intended as tools to help the human become smarter.

The Isabel symptom checker / virtual triage tool was always designed as a tool to help with matching clinical features to diseases, initially for doctors to broaden their differential diagnosis and later to help patients get information about possible conditions and to get to the correct care venue with the appropriate level of urgency.

An important part of the diagnostic process is deciding which diseases should initially be considered and this normally relies on the human doctor matching the set of clinical features obtained from taking the patient’s history and physical examination to the diseases they carry in their memory from previous experience and learning. The obvious limitation is that no human can ever know the various ways that 10,000 diseases can present or let alone recall them in a 10-minute consultation. Computers excel at sorting through vast amounts of information in milliseconds so we designed Isabel to provide a short list back to the doctor or patient which they can then use to help them research further.

Most patients can describe what their problem is without being asked about lots of things they don’t have! And yet the 30 to 50 question decision tree symptom checker is built on the assumption that the patient most likely can’t do this.

At Isabel, our philosophy is to play to the strengths of humans and use computers for elements of the process that humans are not so good at but where computers excel.

In a recent study published in the BMJ Quality and Safety Journal, researchers at the prestigious McMaster University in Canada looked at how Isabel Professional (DDx Generator) changed the performance of doctors looking at a range of cases that were previously used to test older generation decision tree based systems that took a long time to use. 

“The system (Isabel) is designed around entering a small number of symptoms in free-text without the need for symptom qualifiers, pertinent negatives, medical, social or family history background, physical signs, lab values or investigations. Generating differential diagnoses in this way dramatically reduces the time required to several minutes, making it feasible at point-of-care for both physicians and patients.”

Although the researchers found that the improvement in physician performance was similar with Isabel and the previous generation systems, the crucial finding was that this improvement was achieved with far less effort.

“Nevertheless, the critical difference between Isabel and the previous systems is ease of use. In the present study, average time spent with Isabel ranged from 1.5 to 3 min, whereas average times for previous systems were much longer, ranging from 22 to 240 minutes. This is not simply due to a more streamlined interface; unlike prior systems, Isabel uses a minimal amount of (primarily) historical data and yet achieves similar accuracy as previous systems that required far more information. An important implication of this efficiency is that it becomes feasible to integrate Clinical Electronic Decision Support into one’s processing of a case in real time.” 

This meant that the use of tools to help the physician with clinical reasoning was transformed by Isabel from a totally impractical pipe dream to a practical reality.

The same effect will be seen with symptom checker / virtual triage tools for patients. The drop off rates for patients using decision tree based tools that ask 30 to 50 questions is extremely high and with poor accuracy for those that do manage to complete the process. In contrast, users of the Isabel symptom checker report finding it very quick and easy to use with just 4 questions to answer before seeing possible conditions and then another 7 standard questions to get advice on where to find care. Health institutions that use Isabel with a chatbot find the patient can be triaged in just 45 seconds and have them in a queue for scheduling an appointment at the appropriate venue of care for their symptom presentation.

As you saw in the BMJ study, this difference in time and burden on the user will determine whether patients use the resource or not. A high level of due diligence to confirm the clinical accuracy of any virtual triage / symptom checker tool you evaluate is the foundation for success. As this is a key component of any Digital Front Door, patient’s trust and the health system’s reputation are on the line. Choose wisely and all your good work will be rewarded!