Get Technical: Decision Trees

Don’t want a ‘black box’ algorithm? Decision trees may be a simple and effective solution

Let’s start with the context. Civic Tech often involves a broad set of stakeholders from various fields – examples include volunteers, social workers, government agencies and non-profit or community leaders. While these stakeholders may be keen to explore new technology solutions, many may have limited direct experience with technology. Furthermore, accountability and transparency are often critical for such users.

In such a context, complex algorithms, while potentially more sophisticated and effective, may not be the appropriate tool to deploy. They can be alienating to stakeholders and difficult to explain to third-parties. This erodes trust, which eventually limits the capacity for solutions to gain adoption. In such situations, we explore a simple approach known as “Fast and Frugal Trees” that can be both a effective and explainable approach compared to more complex methods.

Real-Life Examples

In early April, Singapore’s National University Health System (NUHS), National Centre for Infectious Disease (NCID) and the Ministry of Health Office for Healthcare Transformation (MOHT) launched COVID-19 Symptom Checker as an online screening tool for members of the public to run a quick self-check and get some basic advice (see Figure 1).

A screening tool like this would not rely on sophisticated machine learning algorithms, but instead would use basic questions to determine whether the user is fine, needs to see a doctor at the clinic, or should go to the emergency department right away. Tools like these are known as “Fast-and-Frugal Heuristics”.

Fast-and-frugal heuristics are simple, task-specific decision strategies that help a decision maker to solve judgment and decision tasks quickly. In applied settings, such as aviation, medicine, and finance, individuals have to often make decisions under varying degrees of uncertainty. In such situations, fast-and-frugal heuristics can help to guide decision making.

Fast-and-frugal trees (FFTs) are a type of fast-and-frugal heuristic. Not unlike decision trees in machine learning, FFTs (see Figure 2) are a specific type of classification tree

• With sequentially ordered cues,
• Where every cue or question has two branches and one branch is an exit point and
• Where the final cue in the sequence will have two exit points to ensure that a decision is always made

Simple trees can be surprisingly effective. One of the classic examples of FFTs was developed by two medical researchers (Green & Mehr) in 1997, where their tree helped heart physicians assess the risk level of patients they were seeing for a particular heart condition. This would allow physicians to direct immediate medical attention to the riskiest patients while avoiding low-risk patients consuming intensive care resources unnecessarily. They discovered that 3 cues were all it took to make such a decision, based on the historical records of patients.

Similar to the Green and Mehr FFT, the Covid-19 Symptom Checker likely uses just a few cues (e.g. age, chronic diseases) to determine whether an individual is in a vulnerable group or not. Cues with the highest sensitivity or specificity are typically used first, and some cues may be used in combination.

How to create such trees: judgement or data?

While FFTs can be constructed just based on domain knowledge, they are often more useful when there is actual data. In February, a World Health Organisation report on a study conducted with China mentioned the percentage occurrence of various symptoms in COVID-19 patients (see Figure 3). Information like these are especially useful for the construction of FFTs. If you’re interested in experimenting with FFTs, cognitive scientist Francisco Quiroga from Chile has developed an FFT Builder that runs on Python. You may also find out more about Francisco and his work from his blog.

Can such simple techniques really be so effective?

Computer scientists often raise concerns over the reliability of FFTs compared to more complex machine learning classification trees. However, studies have shown that the predictive accuracy of FFTs is actually comparable to decision trees generated by machine learning, especially when dealing with cases that are out of the norm. As the construction of FFTs use the data in a general fashion, they turn out to be less prone to over-fitting.

For Civic Tech, FFTs have another important characteristic: they are meant to be used by people. Even if a more complex machine learning decision tree is more accurate by a few percentage points, its structure may not be easy for people to follow and use on the fly. A mini-study that I conducted (see blog article above) suggests that FFTs have the advantage of being understood and used more accurately than a slightly more complex decision tree. If that is true, FFTs will be especially useful for settings where quick but informed decisions need to be made, such as in medical emergencies.

I’m interested: how to I learn more?

If you would like to read more articles about science, social science and data science that I’ve written, please feel free to check out my blog here. You can also reach out and contact me at yeesiang@better.sg.

___