Using AI to support NHS Resolution with negligence claims prediction

Exploring how AI might help the NHS to understand and identify risk, preventing harm and saving valuable resources.

NHS trusts undertake an enormous number of activities each year (around 240 million in the year 2018 to 2019 according to the Kings Fund). The vast majority of people receive safe care, however NHS Resolution received over 15,000 claims for compensation last year on behalf of the NHS in England, which includes hospital trusts and GPs. Although many (almost half the claims in 2018 to 2019) get settled without damages, NHS Resolution figures show that claims can cost the health and care system up to £2.6 billion a year.

In early 2021, the NHS AI Lab Skunkworks team started a rapid feasibility study with NHS Resolution to investigate whether it is possible to use machine learning AI to predict the number of claims a trust is likely to receive and learn what drives them in order to improve safety for patients.

Overview

NHS Resolution provides expertise to the NHS on resolving concerns and disputes fairly, shares learning to enable organisation-wide improvement and to preserve valuable resources for patient care.

The organisation holds a wealth of historic data around claims, giving insight and valuable data around the causes and impacts of harm.

Over a 10-week period, the project sought to:

• improve patient safety by reducing the time lag between incidents and the detection of claims, giving the opportunity for prevention
• create a risk profile for individual trusts by analysing a variety of NHS data
• better understand future costs to the NHS by predicting the volume of incidents that lead to a claim
• help reduce new claims arising by understanding the factors that drive the likelihood of a claim

The NHS AI Lab Skunkworks team worked with NHS Resolution and Accelerated Capability Environment (ACE) suppliers to provide a rapid delivery plan to:

How the AI machine learning was developed

The project aimed to prove the value of machine learning in determining insights from the available data. Automated machine learning was used to run repeated processes on the available data in order to select the AI models that uncovered the most relevant information.

NHS Resolution provided multiple sources of data from the healthcare system spanning two years. This included claims, incidents, workforce, maternity, staff survey and publicly available population data sources.

The team applied automated machine learning methods to rapidly test multiple approaches. This testing:

• looked for correlations, or similarities, between the datasets and investigated effects associated with time lag
• used different modelling approaches including an XGBoost model and a Bayesian Hierarchical Model (BHM) to reach an end result that could both provide accurate predictions and actionable explanations
• identified the “decision tree” model as the best performing and most cost-effective to train because of its ability to cope with varied data quality (data variables and missing values)
• included a focus on “model explainability” - the need to understand how the model is making predictions so that we can explain what drives negligence claims

Constraints

There were a number of constraints that impacted the project including a lack of completeness and consistency across datasets. The data understandably showed a strong association between the number of claims and the size of the trust population, and it was necessary to eliminate the “size effect” so as not to hide other effects present in the data.

It was also not possible to try to predict the rate of claims per specialty (for instance the rate of claims specific to maternity), only by trust, because the data was not complete or granular enough to do so.

Data security

The data used for this testing process was pseudonymised by replacing any personal identifiable information with artificial data (pseudonyms). It was responsibly managed in accordance with General Data Protection Regulations (GDPR).

The project made use of ACE’s PodDev environment for the data. Using this bespoke environment meant that sensitive clinical data could be included securely and destroyed at the end of the project in a way that met the requirements of the NHS Resolution data controller.

Impacts and outcomes

The completion of the “proof of concept” feasibility study led to some significant successes. Although claims prediction was not perfected, the results significantly outperformed baseline models when predicting rates of claims, by trust and by month.

Whilst useful for hospital trusts, predicting the rate of claims is not yet impactful in improving patient safety without further development of this project.

The project has provided an informative view of the data landscape across the wider health service, and its relevance to understanding medical negligence claims.

As a result of the work, it is possible to gain the following insights into what drives rates of claims, from wider health service data:

• The datasets for incidents, specialty, referral-to-treatment and hospital activities have the largest potential impact on the predicted rate of claims.
• The presence of specialties in a trust tends to have a correlation with the predicted rate of claims.
• The number of incidents, and specifically the percentage of incidents resulting in severe harm or death of the patient, also has a correlation with the predicted rate of claims.
• Longer waiting times also appear to correlate with the predicted rate of claims, although this varies with specialty.

The results are an indication that prediction of claims is possible but that large volumes of quality data are required to take the challenge further.

Niamh McKenna, Chief Information Officer, NHS Resolution

Giuseppe Sollazzo, Head of Skunkworks, NHS AI Lab, NHSX

What next?

The resulting code base and methodology is a starting point for further machine learning exercises. The proof of concept has demonstrated how machine learning could increase NHS Resolution’s understanding of medical negligence claims.

In order to demonstrate a reduction in patient harm, more work will need to be done to select and engineer new features.

Ideally, improving the consistency of the claim reporting methodology across trusts would significantly improve the predictive power of the negligence claims data.

Future improvements to both the data and the model would provide a more accurate forecasting model and more insightful explanations.

NHS AI Lab Skunkworks

This project is supported by the NHS AI Lab Skunkworks, which exists within the NHS AI Lab to support the health and care community to rapidly progress ideas from the conceptual stage to a proof of concept.

The NHS AI Lab is working with the Home Office programme: Accelerated Capability Environment (ACE) to develop some of its skunkworks projects, providing access to a large pool of talented and experienced suppliers who pitch their own vision for the project.

Accelerated Capability Environment (ACE) is part of the Homeland Security Group within the Home Office. It provides access to more than 250 organisations from across industry, academia and the third sector who collaborate to bring the right blend of capabilities to a given challenge. Most of these are small and medium-sized enterprises (SMEs) offering cutting-edge specialist expertise.

ACE is designed to bring innovation at pace, accelerating the process from defining a problem to developing a solution and delivering practical impact to just 10 to 12 weeks.

Further information

• Read about the Skunkworks’ database search project: Data Lens
• Find out more about the NHS AI Lab Skunkworks.
• Join our NHS AI Virtual Hub to hear about opportunities to get your AI project taken up by the Skunkworks team
• Get in touch with the Skunkworks team at aiskunkworks@nhsx.nhs.uk.