Metrics are essential to determine the performance of a clinical trial. However, defining what those metrics are, how to select them, and why they’re valuable is not a straightforward process.
Technology used in clinical trials has made standardization of metrics far simpler as data is more easily collected and aggregated. Having standardized metrics makes it easier for all clinical trial stakeholders to understand why these measurements are being used. However, as we will see later on, standardization of metrics can sometimes result in missed key performance metrics and key risk indicators specific to a particular trial.
In this post, we explore the importance of getting the selection of clinical trial metrics correct and how best to ensure that all stakeholders understand what they mean and why they’re being used.
Know What You Want To Answer
Before study staff can decide on which metrics matter, they need to know what answers they want from the research. This strategy helps determine the most important metrics. The next step is to base further and more specific metrics on those fundamental metrics. Second-tier metrics include measuring performance and processes.
Stakeholders should focus on several important factors to determine what metrics to measure. One focus area is enrollment rates, contrasting participant accrual with target enrollment as well as the number of days since the last patient enrolled.
Another is looking at the study’s start-up phase and how it’s progressing towards activation. Pay attention to the time between contract receipt and execution turnaround, as well as the time between submitting to the IRB and being granted approval.
The representativeness of the study sample is also important. Look at patient demographics and diversity to ensure you’re meeting requirements.
Key Metrics To Focus On
Recruiting patients to trials is notoriously challenging, as is retaining them until study completion: 80 percent of studies do not reach their recruitment targets, and one-half of trials recruit only a single patient or none, writes clinical researcher Robert Dannfeld.
One of the most significant metrics to focus on, then, is patient recruitment. Researchers at BMC stress that patient retention to study completion, data quality and protocol compliance should also be added to this list.
Other sources go further, recommending a uniform set of rules be created for improving recruitment and retention. In a study in the Journal of Clinical Oncology, H.J. Durivage and K.D. Bridges note that more than one-half of the therapeutic clinical trials studied failed to accrue any patients, while enrollment tended to be concentrated in specific studies and through specific sources. A clinical trial that sets its own metrics related to patient criteria may be more successful at finding and recruiting patients, as well as in addressing issues that cause patient noncompliance or dropout.
Developing Study Metrics
There are a few common pitfalls when it comes to developing clinical trial metrics. Stakeholders can sometimes select too many metrics that are often overly complex, which can be time-consuming to manage, explains Ann McGee at PharmaLex Ireland.
Other mistakes include pursuing metrics so doggedly that wrong behaviors creep in such as taking actions without context to validate the metrics. Researchers should rather focus on factors such as operational quality performance, indicators that show compliance with regulations, and the differences between quantitative and qualitative data, McGee writes.
Developing study metrics can also help clinical trial teams manage costs. For instance, the right metrics and associated data can help clinical trial centers ensure they have the right staff to meet trial goals, write Carrie Lee and fellow researchers in an article in Business of Oncology.
Choosing the right metrics and defining them appropriately requires careful planning. Metrics should be defined at the beginning of the study and tracked throughout.
Using An Analogy To Discuss Metrics
An analogy is an effective means of explaining complex topics. So when it comes to convincing stakeholders about which metrics to use, try talking about the weather. This is what Alex Hsieh, director of predictive analytics at Pfizer, did to explain the organization’s implementation of machine learning and AI technology to predict quality risk and events.
As with weather forecasting, Hsieh says predicting clinical trial risks requires breaking risk into certain categories using existing data, and then elevating or downplaying that risk with any new data that arrives. Inputting new data will affect changes in the predictive model.
Measuring Tumors and Performance
Knowing the size of cancer patients’ tumors is critical to an oncology study’s success, of course, but researchers and study stakeholders need additional metrics too. Measuring study performance and quality is essential, according to the team at Median Technologies.
The best way for pharma organizations to measure these metrics is to adopt standardized measurements of operational time, cost, and quality. Essential to any successful metrics is the need for agreed definitions of terms, study milestones, and performance targets. Standardized metrics help improve planning and predicting a study’s performance.
The need for standardized metrics, of which all stakeholders should be aware, is best underscored by the pursuit of validity and reproducibility of results. In the case of imaging trials, specifically, it’s vital to equip staff with devices calibrated to standard specifications, as well as strategies for gathering images and following protocol.
The key performance metrics in an imaging trial are cycle times, timeliness, quality, and efficiency, explains the Median Technologies team.
Standardization Helps but Is Not Sufficient
An industry survey shows that 83 percent of biopharmaceutical companies use standardized performance metrics for clinical operations and data management. Nearly 70 percent use metrics for quality management, explains Moe Alsumidaie, head of research at integrated research organization CliniBiz.
He points to the Metrics Champion Consortium, now part of the WCG Avoca Quality Consortium, as an example. The Consortium has developed its analytics into therapeutic-specific study metrics. These standardized measurements present huge potential benefits to the whole industry. Most notable are clinical trial oversight, risk management, and performance.
But Alsimidaie urges caution. In the pursuit of standardization, study-specific KPIs and KRIs might be neglected, leading to the omission of significant study issues such as therapeutic indication, study design, drug or device, and study phase.
Outcome-Based and Goal-Based Metrics
Measuring whether a trial achieves its desired strategic objectives requires that outcome-based metrics are used. But because they are often broad with many different stakeholders contributing to a study’s outcomes, metrics can be difficult to decide upon.
Goal-based metrics, on the other hand, concern specific activities and initiatives that help to bring about the desired outcome. These goal-based metrics should be linked with a complementary outcome-based metric.
Outcome-based and goal-based metrics are related but distinct. They should be considered simultaneously and connect to the study’s objectives. Embedded as sub-categories of both outcome-based and goal-based metrics are quantitative and qualitative metrics. However, outcome-based metrics tend to be qualitative, and goal-based metrics quantitative.
The Value of Site-Based Metrics
Multiple stakeholders can benefit from using site-based metrics. Milestones can be set and progress measured based on performance data from past trials. Similarly, cycle times and progress from previous trials allow for a more accurate prediction of timelines of current trials.
But it’s not just sponsors that benefit from site metrics. Sites themselves can rely on this data to improve their processes. That information helps sites choose trials for which they will be able to deliver and meet sponsors’ needs.
Data from robust metrics also tend to give more weight to requests that sites make to sponsors and other stakeholders.
Metrics that should be focused on include past site performance measured against industry averages, volumetrics (number of trials conducted, cycle-times), and the historic speed to randomize the first study participant.
Relevance Depends on Stakeholders’ Perceptions
Quantifiable outcomes of investigational treatments that are easy to interpret and clinically relevant are fundamental conditions to meet when bringing new therapies to market.
However, what is considered relevant depends on which stakeholder you ask, says molecular geneticist Claudia Dall’Osso and Ian Love, a business insights analyst at Decision Resources Group. Developers and patients may well differ in this regard.
The key is defining clinical endpoints. Developers have traditionally chosen endpoints that are not always relevant to patients. This is changing, of course, with a new focus on patient-centric study designs. As a result, patients’ perspectives are being included in drug development.
These patient-centric outcomes are clinical metrics that are relevant to patients and caregivers. This requires understanding the patient’s journey, needs, and pain points. Determining these relevant endpoints, particularly in the treatment of rare diseases, requires drug manufacturers to first identify specific disease manifestations that negatively affect patients.
Next is to determine what amount of change to these diseases patients consider as meaningful. Collaboration with patients and patient advocacy groups, regulatory agencies, payers and physicians will help to determine the relevant metrics to focus on, write Dall’Osso and Love.
What is most important to a patient may not be the same for sponsors, CROs or researchers when it comes to measuring the performance and success of a clinical trial. However, taking patient-centric approaches to research better aligns the desired outcomes of multiple stakeholders. Choosing the right metrics and ensuring their relevance and meaning are explained will empower all study stakeholders.
Updated 9/26/22. Originally published 8/6/19.
Images by: ammentorp/©123RF Stock Photo, wavebreakmediamicro/©123RF Stock Photo, dolgachov/©123RF Stock Photo