Artificial intelligence (AI) and machine learning (ML) have had little impact on the effort to thwart the COVID-19 pandemic to date, but that may be changing with the FDA’s grant of an emergency use authorization (EUA) for an algorithm intended for screening of asymptomatic individuals. Developers should note that the EUA was granted only after two studies of a total of nearly 900 subjects, suggesting that the FDA expects a large amount of data for ML algorithms that infer the presence of disease by indirect means.
A number of developers have attempted to write algorithms for use in the pandemic, but the evidence suggests that driving an ML algorithm to market is more difficult than might be commonly appreciated. A March 2021 review of ML algorithms designed to process thoracic images for diagnostic purposes states that among the difficulties in the included studies are:
- The risk of bias often encountered small data sets;
- The variability in large, internationally sourced data sets; and
- The need to work with clinicians in the development phase to ensure clinical relevance, and to ensure that the algorithm can be readily implemented in routine clinical practice.
These lessons should be taken to heart by developers of algorithms for other purposes, even in the context of the lowered evidentiary standards associated with the FDA’s EUA program relative to conventional premarket review requirements.
The FDA granted an EUA for the Tiger Tech COVID Plus monitor March 19 as a screening system that uses two photoplethysmography sensors in an armband for direct acquisition of pulsatile signals over a period of three to five minutes. The algorithm analyzes the signals for evidence of characteristics associated with infection with the SARS-CoV-2 virus, such as hypercoagulation and hyperinflammatory states. These symptoms are associated with other diseases as well, and thus the algorithm is not a substitute for a diagnostic test, nor is it intended for use on symptomatic individuals.
Instead, the Tiger Tech product is indicated for use in screening of individuals older than five years when those individuals exhibit higher than normal temperatures that do not qualify as fever. The company studied the clinical performance of the system in a hospital-based validation study that enrolled 467 asymptomatic individuals, including 69 confirmed positive cases. This study yielded a positive percent agreement (PPA) of 98.6% and a negative percent agreement (NPA) of 94.5% against reverse transcription PCR testing.
The confirmatory school study enrolled seven confirmed positive cases and 412 confirmed negative cases, providing a point estimate of 100% for both PPA and NPA. Going forward, Tiger Tech Solutions Inc., of Miami, FL, must provide comparative outcomes for 100 consecutive cases presumed positive for the SARS-CoV-2 virus and 1,000 cases presumed negative over the span of a year, but these 1,100 subjects must also be verified with PCR on the same day.
The company must also have a process in place for data collection at sites that had not taken part in the clinical studies of the device, although these reports need not be accompanied by PCR testing if such services are not immediately available. All in all, the postmarket study is a considerable expense for the developer, although the potential market is considerable. The agency said the monitor could be used in public settings such as schools, workplaces, theme parks, stadiums and airports, a good sampling of the types of activities that will have to resume if the economy is to return to any semblance of normalcy.
BioFire Breaks Ground as First EUA Conversion
The week of March 15 was a week of two firsts for FDA-regulated products for the pandemic, although the demands placed on the Tiger Tech algorithm at first glance seem substantially greater than those applied to the BioFire Respiratory Panel 2.1. The BioFire diagnostic was the subject of a successful de novo petition announced March 17, but one of the critical differences is that the BioFire test was initially routed through the EUA pathway. This approach allowed the sponsor to concurrently generate both revenues and data for the de novo petition instead of having to prospectively finance all the development costs for the de novo petition.
The FDA’s Tim Stenzel has mentioned that the agency is working on a guidance for converting EUAs to conventional premarket applications, but he has also tried to prod developers to convert their EUAs to conventional premarket filings in the meantime. This may have been part of the reason the agency made a point of highlighting the BioFire EUA event with the March 17 press release.
The agency pointedly reminded the reader, however, that the BioFire de novo was based on additional data “showing validation beyond what is needed” for an EUA. The agency said BioFire Diagnostics LLC presented data from a clinical study of more than 500 test samples and multiple analytical studies, although the press release did not specifically describe this as entirely additive to the EUA studies. The grant of de novo petition was made the same day the FDA revoked the associated EUA.
One of the advantages of successful de novo petitions is that the FDA must provide a series of special controls, which gives other developers and manufacturers a guide for development of similar products. The de novo created a new product code (QOF) and a new regulation (Title 21, Part 866.3981), but this was perhaps no ordinary filing as the multi-analyte test is cleared for several pathogens in addition to the SARS-CoV-2 virus. Among these are several coronavirus and influenza pathogens for a total of 19 microbials. The test is subject to an overarching regulation (Title 21, §809.10) as well as the specific regulation, with Part 809.10 calling for product labeling to provide a detailed description of the device and its functions, including reagents, ancillary controls and a list of all pre-analytical methods.
As might be predicted, a test that covers such a large number of pathogens comes with a wide range of mandated special controls, such as that design verification and validation must include:
- Documentation of a clinical study that includes sequential samples for each claimed specimen type;
- Documentation of a risk analysis covering failure mode effects analyses; and
- A detailed description of the identity, phylogenetic relationships, and other recognized characteristics of the pathogens listed on the device label.
This is far from an exhaustive list of the requirements featured in the de novo for the BioFire test, and test makers may wish to consider the possibility that the burgeoning number of variants of the SARS-CoV-2 virus may require an ever-larger number of analytes within a test to remain competitive in what is certain to become a hotly contested market for infectious disease diagnostics in the years ahead.
