New opportunity to test AI driven COVID-19 models on the largest database of COVID-19 chest images in the UK
Developers of COVID-19 AI technology can now apply to have their model’s performance tested on a national COVID-19 chest X-ray data set.
At the NHS AI Lab, we started collecting chest images in May 2020 in order to support the NHS response to the coronavirus pandemic. The X-ray, CT and MRI images are being used to study and understand the COVID-19 illness to improve the care for patients hospitalised with severe infection. Since the pandemic began, AI technologies have been in development that have the potential to identify the virus more quickly, start treatments earlier and reduce the likelihood that people will need intensive care.
The National COVID-19 Chest Imaging Database is the largest database of its kind in the UK, with more than 20 NHS trusts contributing to it
Now, almost a year on, we’ve got over 40,000 images from people with and without coronavirus. The National COVID-19 Chest Imaging Database (NCCID) is the largest database of its kind in the UK, with more than 20 NHS trusts contributing to it.
As the database continues to grow, we want to make the best use of such a great resource. This is why we are now inviting research institutions and technology companies that have been developing AI models using COVID-19 chest X-ray data to apply to have their model’s performance assessed using NCCID data.
Why is this testing so important?
How an AI technology performs is inherently linked to the characteristics of the data it was trained on. This is why testing the performance of the AI on a representative data set that reflects the UK population is so important. We want to make sure that when an AI product is deployed in the real world, it has seen enough of a variety of data in its development to ensure that it can perform safely, consistently, and accurately.
The problems with bias and data quality for machine learning in this area are discussed in this recent Nature article: Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans.
The majority of scans in the database are chest X-rays. It also holds some extra data about people who tested positive for coronavirus, such as age and gender. Care has been taken to ensure that this data is anonymous and representative of the UK population.
A portion of this data has been set aside as ‘unseen’, or hidden from the researchers and technology companies, so that it can be used to validate, or test the performance of AI models. In the same way that pupils should not have access to an exam before it is administered, the ‘unseen’ aspect of this data is important to ensure that the test is fair, with no one AI model having an advantage over others.
The representative nature of the data set gives researchers and technology companies the confidence that the results of the test will be a good reflection of how their model might perform in a real world healthcare setting. This in turn allows the public to trust in the safety and efficacy of these AI technologies should they eventually be deployed in the NHS.
The safety of AI technologies for health and care
At the NHS AI Lab we want to encourage the development of new AI technologies for health and care, but not at the expense of safety and quality. Assessing the performance of AI models on a resource like NCCID, which has data representative of the UK population, provides NHS commissioners, healthcare regulators and end users with the evidence to judge the safety, efficacy, and generalisability of AI models before they are used in clinical practice.
Working closely with NHS commissioners, regulators and end-users, the NHS AI Lab hopes to use this performance assessment exercise to inform how this process may be carried out in the future, which will help new AI technologies to get deployed into use more quickly and start making a real difference to people’s health and care.
