RWD - Greenleaf Health

This article was originally published as a guest column in Clinical Leader.

Over the past month, the FDA has continued to build out its real-world evidence program with the release of the final guidance, Submitting Documents Using Real-World Data and Real-World Evidence to FDA for Drug and Biological Products. The brief process-focused guidance describes the recommended approach sponsors should use to identify if and how real-world data (RWD) and real-world evidence (RWE) are incorporated in their regulatory submissions. The FDA notes that this guidance specifically applies to investigational new drug applications (INDs), new drug applications (NDAs), and biologics license applications (BLAs) that contain RWD and RWE. The guidance does not, however, address or comment on review standards for submissions containing RWD or RWE.

Focus on Product Safety and Effectiveness

In this guidance, the FDA signals that its primary focus regarding RWD and RWE is on their use to support product effectiveness and safety. Particularly, the FDA advises that sponsors only flag a submission’s inclusion of RWD and RWE if they are intended to support a regulatory decision on a product’s safety and efficacy. As a rule of thumb, the FDA suggests “sponsors and applicants not identify submissions that contain RWD/RWE if those data are not intended to support product labeling.”¹ Examples of submission types that do not need to be flagged for incorporating RWD or RWE include those that use RWD to generate hypotheses or studies that include RWD in exploratory models or simulations, studies that use RWD to validate an endpoint, and studies that use RWE to qualify a Drug Development Tool (DDT).

These guiderails will likely have implications for the type of public data that both sponsors and stakeholders will have access to on the use and regulatory acceptance of RWE. The FDA notes that the internal tracking of these submissions is intended to inform the “FDA’s understanding of the scope and use of RWD and RWE submitted to support regulatory decisions…”² However, this focus likely limits the scope of the FDA’s public reporting of RWE submissions. Under PDUFA VII, the FDA has committed to reporting aggregate and anonymized information on RWE submissions to CDER and CBER starting no later than June 2024. Thus, sponsors and stakeholders can expect the future report to focus only on RWE submissions supporting product labeling rather than other uses.

Key Data Points for Tracking

The sample submission checklist included in the appendix titled Sample Presentation of Information to be Included with Submissions Containing RWD/RWE provides further insight into the data points the FDA is looking to collect on RWD and RWE use. In addition to the product name and targeted disease or medical condition, the FDA suggests sponsors identify their purpose for using RWD or RWE as part of the submission, the study design using RWD to generate RWE, and RWD sources used to generate RWE. Per the sample list of purposes for RWD and RWE, the FDA anticipates the most common uses of RWD and RWE will be:

to support safety and/or effectiveness for a product not previously approved by the FDA;
to support labeling changes for an approved product; and
to support or satisfy a postmarketing requirement or postmarketing commitment (PMR/PMC).

In both the body and appendix of the guidance, the FDA also lists the anticipated study designs likely to involve RWD to support a regulatory approval. The list broadly aligns with the current clinical trial landscape and includes randomized controlled trials with pragmatic elements, single-arm trials using RWD in an external control arm, and non-interventional or observational studies. The FDA is also interested in collecting data on the types of RWD sources sponsors use to generate RWE. The FDA’s sample list of RWD sources, which is not intended to be exhaustive, includes electronic health records data, medical claims data, product or disease registry data, and data collected from digital health technologies in non-research settings.

As noted, the submission checklist included in the appendix is helpful in providing a glimpse into the FDA’s current understanding of the RWE landscape and a preview of the data points the agency is looking to track on RWD and RWE use for regulatory purposes. Just as important, however, are the data points that the FDA did not include in the submission checklist. Notably, the FDA does not request information on the data standard sponsors used to structure large data sets. To date, the agency has remained silent on preferred common data standards. However, per the FDA’s 2021 draft guidance titled Data Standards for Drug and Biological Product Submissions Containing Real-World Data,³ “FDA plans to issue further guidance and/or to update the Catalog with standards for study data that are derived from RWD sources.”⁴

The guidance was authored by the Office of Medical Policy in the Center for Drug Evaluation and Research (CDER), in consultation with the Center for Biologics Evaluation and Research (CBER) and the Oncology Center of Excellence (OCE). Notably, the Center for Devices and Radiological Health (CDRH) is not listed as a co-author, signaling that the FDA may not be taking an “inter-center” approach to RWE regulatory policy. This is further underscored by the fact that the 2017 guidance on RWE use for medical device applications, Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices,⁵ was authored by only CDRH and CBER, not CDER. As of now, the FDA has not provided recommendations on how to flag device submissions as using or containing RWD or RWE.

References

FDA, Guidance for Industry, “Submitting Documents Using Real-World Data and Real-World Evidence to FDA for Drug and Biological Products,” September 2022, https://www.fda.gov/regulatory-information/search-fda-guidance-documents/submitting-documents-using-real-world-data-and-real-world-evidence-fda-drug-and-biological-products.
Ibid.
https://www.fda.gov/media/153341/download.
Ibid.
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/use-real-world-evidence-support-regulatory-decision-making-medical-devices.

This article was originally published as a guest column in Clinical Leader.

The FDA’s release of four draft guidances on real-world evidence (RWE) at the tail end of 2021 is a productive step toward implementing FDA’s RWE program. The draft guidances are expansive, providing recommendations to sponsors seeking to use real-world data (RWD) derived from medical claims data, electronic health records (EHRs), and registries to support regulatory submissions. However, despite the scope of these documents, a survey of comments letters submitted in response to the draft guidances reveals a widespread need, specifically among drug developers, to understand how the agency will implement these guidances in practice. In an attempt to solicit concrete examples, several sponsors requested the agency develop an online dashboard that publishes all of the agency’s decisions, both positive and negative, on regulatory submissions that incorporate RWE.

Despite the lack of a central catalog of all RWE-related regulatory decisions, there are a handful of product approvals and FDA reviews that highlight both best practices and common pitfalls sponsors face when incorporating RWE in regulatory submissions. These decisions show that the use of RWE to support regulatory decision-making is often challenged by both methodological and process issues. In terms of process, the primary mistake made by sponsors is failing to share a prespecified protocol and statistical analysis plan (SAP) with the agency prior to submission. From a methodological perspective, unsuccessful RWE submissions are often plagued by missing data, small patient populations, and a lack of objective endpoints. Approvals based on or supported by RWE, such as Ibrance (palbociclib) for male breast cancer and Prograf (tacrolimus) to prevent organ rejection post lung transplant, demonstrate how to successfully address the myriad challenges of working with RWD.

1. Failing to Share a Prespecified Protocol and SAP

As noted, the most common mistake sponsors make when seeking to leverage observational studies to support or demonstrate a product’s safety and efficacy is failing to share a prespecified protocol and SAP with the agency. The FDA emphasizes the importance of up-front transparency to guard against this risk, noting in its real-world evidence framework that multiple analyses in electronic data sets can be done quickly and inexpensively, “making it possible to conduct numerous retrospective studies until the desired result is obtained….”¹ The four recent guidances also recognize this as a potential risk to undermining study validity and recommend sponsors:

“Provide draft versions of their proposed protocol and statistical analysis plan (SAP) for Agency review and comment, prior to finalizing these documents and before conducting the study analyses.”²

2. Missing Data

One of the most common methodological deficiencies identified in FDA reviews of RWE studies is missing data. Unlike clinical trials, wherein patient visits and data collection are prespecified, “real-world” patient visits are often irregular and data collection uneven, leading to spotty and inconsistent electronic health record (EHR) data. In turn, incomplete EHR data can make it difficult to define baseline characteristics for observational cohorts, obfuscating efforts to establish comparability with an active trial cohort. This problem has hobbled several attempts to use RWE studies to support oncology product approvals. In these cases, the sponsor was unable to provide complete documentation on essential baseline characteristics such as previous treatment regimens, tumor stage, and Eastern Cooperative Oncology Group (ECOG) scores. FDA reviewers noted that the failure to capture previous treatment regimens introduces confounding bias, while missing dates, such as the date of diagnosis or start of first treatment, can introduce bias into the study in favor of the investigational product.

In successful RWE applications, the majority of these important clinical details are well documented. For example, FDA notes in the supplemental review of palbociclib for male breast cancer that “In this study, real world tumor response data was generally available for several lines of therapy.”³ The reviewer also notes that real-world data sets were complete and included birth date, metastatic sites, and first metastatic treatment. Unstructured source data in the form of clinician assessments and radiology reports were also included in the submission.⁴

3. Small Patient Cohorts

Incomplete EHR data has significant downstream effects on a sponsor’s ability to use RWE for regulatory purposes. In an attempt to achieve comparability with active trial cohorts, sponsors often have to disqualify patients with incomplete data elements from analyses, further whittling down a data set that may already be pretty small. For oncology and rare disease applications, small patient populations are already a common issue, which the use of RWE can compound. This has hindered several attempts to use RWE to support approval. Even in successful RWE applications, like palbociclib, the reviewer still called out the small sample size as a key deficiency.

To address the issue of small patient populations, FDA has in recent guidance identified both data linkage and the combining of data as possible solutions. However, the guidance recognizes that these methods also introduce a new set of methodological problems, notably data heterogeneity. In the draft guidance Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products (September 2021), FDA states:

“For studies that require combining data from multiple data sources or study sites, FDA recommends demonstrating whether and how data from different sources can be obtained and integrated with acceptable quality, given the potential for heterogeneity in population characteristics, clinical practices, and coding across data sources.”⁵

In addition to undermining attempts to combine two different data sets, differences in population characteristics, clinical practice, and coding have derailed efforts to use observational cohorts to define standard clinical practice in the context of superiority trials. Again, many attempts to use RWE for oncology indications have failed due to differences in treatment locations and treatment classifications which prevented sponsors from being able to draw direct comparisons between the investigational product and the standard of care.

4. Lack of Objective Endpoints

A third common roadblock to the acceptance of RWE studies for regulatory purposes is the difficulty in uniformly capturing clinical outcomes in EHR or claims data. In recent guidance, FDA emphasized the enormous impact that variability in physician practice, specifically in regard to diagnoses and coding, have on capturing clinical outcomes. To address this risk, FDA recommends sponsors use RWD sources to capture outcomes with more objective “well-defined diagnostic criteria,” such as death, stroke, or myocardial infarction.

Using RWE for oncology indications is particularly tricky as many commonly used oncology endpoints are subjective and time-bound. Tumor response rates and progression free survival (PFS), for example, are inherently subjective and rely on visual assessments of radiographic images, often generating wide variability. These outcomes are also difficult to validate, as most EHR records do not incorporate the radiographic image. Meanwhile, time to treatment discontinuation (TTD) and overall survival (OS) is particularly subject to confounding bias due to treatments beyond disease progression. As noted, FDA’s answer to these issues is to encourage sponsors to use RWD for conditions or diseases with straightforward and objective outcomes. It is telling, then, that the only drug to have received approval based on an observational study is tacrolimus for lung transplant recipients, wherein the primary endpoint was post-transplant survival.

Considering the methodological issues sponsors face in working with RWD, FDA’s cautious stepwise approach to implementing the RWE program is understandable, but it is frustrating to some. As drug developers and the agency continue to wrestle with these methodological issues, we can expect that successful RWE regulatory submissions will continue to be for drugs with well-established safety and efficacy profiles.