University of Houston College of Pharmacy (UHCOP) Ph.D. candidate, Chijioke M. Okeke, has earned a $5,000 PhRMA Foundation Trainee Challenge Award for his research examining how well artificial intelligence (AI) models can evaluate potential bias in real-world evidence (RWE) studies.
The PhRMA Foundation partners with scientific journals on its Challenge Award competitions to recognize promising early-career researchers addressing emerging and timely scientific questions. An expert review panel selected the winning articles based on their innovative contributions and their implications for future AI research and application.
A Ph.D. candidate in the Pharmaceutical Health Outcomes and Policy concentration of the Pharmaceutical Sciences Ph.D. program, Okeke was one of only three trainee first authors selected for the award. The competition was held in conjunction with a special September issue of the Journal of Managed Care & Specialty Pharmacy focused on artificial intelligence in managed care pharmacy.
His award-winning paper, “Comparative Performance of Large Language Models for Appraising Bias in Real-World Evidence Studies,” examines whether large language models (LLMs) can reliably perform structured assessments of potential bias in RWE studies, a process that, according to Okeke, is time-intensive and requires specialized expertise.
Putting AI to the Test
RWE studies use real-world data to generate evidence about healthcare treatments, interventions, diseases, utilization, safety or effectiveness outcomes in routine clinical practice, rather than under the tightly controlled conditions of a traditional clinical trial. RWE is increasingly used alongside evidence from clinical trials to inform decisions about the safety, effectiveness and value of healthcare interventions, with findings that can influence clinical practice, regulatory decisions, payer coverage and health policy.
The project grew from the researchers’ interest in the increasing use of RWE in healthcare decision-making and the potential for LLMs to support quality assessment of published studies. Because identifying potential sources of bias in RWE studies is critical but can be time-intensive and require methodological expertise, Okeke and his collaborators set out to answer a practical question: Can LLMs help researchers and other healthcare stakeholders assess potential bias in RWE studies accurately and efficiently?
Why Bias Matters
Unlike randomized clinical trials, RWE studies typically do not randomly assign patients to treatments, which can make them susceptible to several sources of bias. Selection bias can arise when patients in treatment groups differ systematically, while confounding bias can occur when factors such as age, disease severity or other health conditions influence both treatment selection and outcomes. Misclassification or measurement bias can occur when diagnoses, treatments or outcomes are inaccurately or differently captured across groups.
If these potential biases are not appropriately identified and addressed through study design or analysis, they can lead to incorrect conclusions about a treatment’s benefits or risks. Accurately identifying bias is therefore important because healthcare stakeholders need to understand how much confidence they can place in a study’s findings.
For patients, the quality of that evidence can affect which treatments are considered safe and effective and ultimately made available to them. Reliable evidence also supports better-informed treatment decisions for healthcare professionals. For payers, regulators and policymakers, understanding potential sources of bias helps ensure that decisions about healthcare coverage, access and policy are based on credible evidence.
Measuring AI Against Human Expertise
To investigate whether AI tools could help with this quality assessment, Okeke and his collaborators evaluated 40 LLMs from six major AI technology providers. The models were asked to appraise 10 previously published RWE studies representing a range of pharmacoepidemiologic study designs and data sources.
Each AI model evaluated the studies using the Appraisal of Potential Bias in Real-World Evidence Studies (APPRAISE) framework, a structured approach for assessing potential bias in RWE studies of treatment effectiveness and safety. The researchers then compared the models’ assessments with those made by human reviewers trained in pharmacoepidemiology. The results demonstrated both the potential and current limitations of artificial intelligence.
Across the 40 models, overall accuracy ranged from 12.9% to 66.1%. The highest-performing model achieved 66.1% accuracy, while performance varied considerably depending on the specific AI model used. Where AI showed a clear advantage was speed.
Human reviewers required an average of about 61 minutes to assess each study. The AI models completed the same assessments in an average of less than one minute to approximately 17 minutes per study. Despite that substantial increase in efficiency, the findings suggest that speed alone is not enough when evaluating the methodological quality of research.
AI as a Research Support Tool
The researchers concluded that LLMs show considerable promise for automating portions of the RWE appraisal process, but their variable performance means human expertise remains essential. Rather than replacing trained reviewers, the researchers suggest AI may be most useful as a decision-support tool that helps experts complete complex evaluations more efficiently.
The study also reflects Okeke’s broader interest in exploring how AI can be integrated into healthcare research. He is particularly interested in investigating how AI tools can complement human expertise and improve efficiency across the research continuum, from conducting studies to evaluating their methodological rigor and scientific integrity.
A Collaborative Approach
Okeke conducted the study with Regina N. Nechi, B.Sc., B.Pharm., a Ph.D. student in Outcomes and Translational Sciences at The Ohio State University College of Pharmacy; Javeria Khalid, Ph.D. (’26), Pharm.D., M.Phil.; Ibraheem M. Karaye, M.D., DrPH, MPH, CPH, associate professor of population health and director of the Health Science program at Hofstra University; J. Douglas Thornton, Pharm.D., Ph.D., BCPS, UHCOP associate professor and director of The Prescription Drug Misuse Education and Research (PREMIER) Center; and Ismaeel Yunusa, Pharm.D., Ph.D., assistant professor in the Department of Clinical Pharmacy and Outcomes Sciences at the University of South Carolina College of Pharmacy.
The paper appears in Volume 32, Issue 9 of the Journal of Managed Care & Specialty Pharmacy, the Academy of Managed Care Pharmacy’s official peer-reviewed journal.