This story has been updated to include comments from the US Food and Drug Administration.
NEW YORK – Combining a survey of diagnostics developers with a deep review of the US Food and Drug Administration's pandemic response, the US Department of Health and Human Services' Office of the Inspector General has recommended that the FDA adjust its course for future health emergencies.
In a 51-page report published on Wednesday, the OIG detailed the policy adaptations that the FDA made as it developed its COVID pandemic response, oftentimes on the fly, and recommended six tweaks be implemented before the next pandemic.
The OIG at HHS is the largest office of the inspector general in the US federal government and is tasked with evaluating the efficiency and effectiveness of more than 100 HHS programs. As part of its review of the FDA's Emergency Use Authorization process, the OIG team conducted analyses of FDA data and documents, interviewed FDA staff, and also surveyed diagnostics developers.
Specifically, Elizabeth Sandefer, a social science research analyst and OIG's team leader for the study, said in an email that the survey was sent to every developer that contacted the FDA about tests for COVID-19. This included every developer that reached out to the FDA with questions about the EUA process as well as all that submitted a formal EUA request.
"The intention of this evaluation is to provide stakeholders — including FDA, Congress, and the public — with timely, useful, and reliable information on this significant issue," Sandefer said.
In total, 237 developers provided feedback. The OIG team also interviewed test developers and representatives from a professional lab association for context about the developers' experiences, according to the report.
Diagnostics were necessary because without sufficient testing, efforts to control a virus are significantly limited, the OIG report noted. Nevertheless, what began with a single case of "2019 novel coronavirus" in Washington state in January 2020 ultimately resulted in more than 83 million SARS-CoV-2 cases and 1 million deaths in the US.
And although the FDA pivoted to use its EUA capabilities to help resolve the initial rollout failures of the US Centers for Disease Control and Prevention's RT-qPCR assay, the FDA's need to engage with private developers "revealed vulnerabilities in the federal approach to early emergency testing," the OIG report concluded.
Overall, the report found that some of the FDA's approaches were effective, while others were less so.
The generation of templates for EUA submission and the creation of forums for communication between the FDA and developers — such as the weekly town hall calls, guidance documents, a dedicated email inbox, and FAQ webpages — were counted in the win column by OIG.
But OIG also noted in the report that diagnostic developers were sometimes "confused and frustrated" by the FDA's evolving policies. Some spent a long time in the review queue only to find out that the agency decided to stop evaluating submissions, for example.
Most damning, the report suggested that the FDA's initial take-all-comers approach diluted the agency's focus and increased its workload, and an emphasis on rapidly increasing the total number of tests may have led to authorization of substandard assays.
Indeed, "FDA’s decision to accept all EUA requests resulted in a record number of submissions — often low-quality and from developers lacking experience with FDA's processes," the report noted.
The report highlighted that between January and May of 2020, a total of 731 tests were submitted to the FDA. The agency reviewed and authorized 117 of these tests and granted EUA to 101 more over the subsequent 12 months.
But FDA staff told OIG that most of the submissions were "low-quality" and lacking essential data, particularly attributing this to smaller device makers and labs that had not previously had experience working with the FDA.
Supporting this, the OIG report found that only 28 percent of the survey responders had previously gone through any FDA approval or clearance process. And experience helped, as 91 percent of those with past experience received EUA for their first submitted COVID-19 test, according to the report, while only 40 percent of those with no prior experience did.
On the other hand, developers told the OIG team that the FDA's evolving policies were sometimes not clearly communicated, and that templates sometimes changed after submissions were sent. Also, the FDA's prioritization criteria were murky and poorly communicated, with some developers reporting getting bumped to the back of the line and then receiving a "decline to review" notice after months of patiently waiting.
And still, "The FDA eased EUA requirements to expedite getting COVID-19 tests to market … [and] its decisions to reduce or forgo EUA requirements allowed problematic tests to be distributed," HHS's OIG team wrote.
The report specified that the FDA made "calculated decisions to increase availability of COVID-19 testing, but these decisions often came at a potential cost to test quality," basing authorization on lower levels of evidence when developers struggled to access clinical samples.
This led to a feedback cycle, whereby "FDA's policies allowed diagnostic and serology tests to get on the market quickly; however, that resulted in some problematic tests on the market, requiring further action by FDA," the OIG authors also wrote.
For example, of the seven labs that contacted the FDA under its initial notification policy — which allowed labs to start testing prior to submitting an EUA request — one lab withdrew its test due to contamination while three others had design or validation problems, the report found.
"A later FDA analysis of 125 EUA requests for lab-developed diagnostic tests found that 82 tests had design or validation problems," the OIG report authors wrote, adding that the FDA identified similar issues with commercial tests.
And, "problematic tests" were on the market and being used for months in some cases between notification and FDA review, according to the report. This was a particular issue for a "flood" of "poorly performing and inappropriately marketed" serology tests. Subsequent testing found that two-thirds of serology tests did not have adequate performance data.
In addition to these issues, the OIG assessment revealed that the FDA needs to have "clear, direct communication with the lab community during an emergency," according to the report.
It also found that developers are likely to require further FDA guidance on how to validate tests during shortages of clinical samples, and that, in general, inexperienced assay developers require more support from the FDA during the EUA process. The report further found that the FDA needs contingency plans to handle any increased workloads in a future emergency.
Importantly, "FDA addressed early testing challenges using EUA flexibilities, but this experience underscores the need for a national testing strategy," the OIG team wrote in the report.
For the next pandemic
The OIG report recommended six steps the FDA should take in anticipation of future pandemics.
OIG's Sandefer said the recommendations include ones that the FDA can implement internally but also ones that "emphasize the need for FDA to work with federal partners to implement lessons learned from COVID-19 that go beyond the EUA process," she said.
The recommendations included assessing and revising the guidance for EUA submissions, and developing a suite of EUA templates for future emergencies as well as a suite of resources to help guide test developers through the EUA process.
The OIG also suggested the FDA revise its device-tracking platform to facilitate EUA submission and monitoring, and that it establish formal communication channels between the FDA and the lab community to be engaged during emergencies.
Finally, OIG guided the FDA to work with federal partners to develop a national testing strategy.
At a minimum, this should involve determining the feasibility of contracting with test manufacturers in advance of the next emergency, as well as determining the feasibility of establishing a program to conduct all test validation to support EUAs and of precertifying labs for emergency test development and use, the OIG report authors wrote.
The OIG report emphasized that the FDA was consulted and concurred with all of the recommendations.
HHS and OIG will now follow up with the FDA within six months to get a "Final Management Decision," which outlines how the agency plans to execute the recommendations, OIG's Sandefer said.
And, HHS and OIG intend to continue to follow up with the FDA annually about the status of the recommendations.
In a statement, FDA said it appreciates OIG's recognition of the extensive efforts made by the agency since the beginning of the pandemic, as well as the recommendations identified by OIG to further enhance its approach to tests used in a public health emergency.
"We have already taken steps to implement many of these recommendations," the statement noted.
With the current monkeypox public health emergency, the agency "has been closely engaged with commercial manufacturers, as well as labs through the new CDC/private sector [memorandum of understanding] on infectious disease tests," and has adopted recommendations from the developer community in the monkeypox validation templates, the FDA said.
FDA intends to "continue to take an appropriately flexible approach to COVID-19 tests to meet consumer needs at this stage of the pandemic," and will also continue to consider OIG's recommendations as it responds to the monkeypox public health emergency "as part of our efforts to be best prepared for future outbreaks," the statement said.
In a separate report issued earlier this month, OIG also evaluated the FDA's participation in the Tri-Agency Task Force, a working group which assigns distinct roles to FDA, CDC, and the Centers for Medicare and Medicaid Services in any health crisis requiring labs to implement tests.
That evaluation was motivated by the fact that the pandemic was the first emergency since the 2019 establishment of the task force. It concluded that "multiple setbacks plagued the early testing rollout for COVID-19," but that this "failure" was offset by the fact that the FDA's engagement with the task force showed "the value of interagency coordination and collaboration to facilitate emergency test implementation in labs."