Journal of Psychosocial Nursing and Mental Health Services

The articles prior to January 2012 are part of the back file collection and are not available with a current paid subscription. To access the article, you may purchase it or purchase the complete back file collection here

Psychopharmacology 

What You See Depends on Where You’re Looking and How You Look at It: Publication Bias and Outcome Reporting Bias

Robert H. Howland, MD

Abstract

Study publication bias is the decision to publish or not publish a study based on its results. Compared to unpublished work, published studies are more likely to have positive or statistically significant findings. Outcome reporting bias is opting to publish only a subset of the original variables recorded for a study, such that the inclusion of the variables in the published work is selectively based on the results. Statistically significant results have a higher likelihood of being fully reported compared to nonsignificant results, and a significant proportion of published articles describe outcome variables or data analyses that differ from the pre-specified trial protocol as originally conceived. Recognition that publication bias and outcome reporting bias contribute to a distorted perception of drug effects—inflated estimates of efficacy and underreporting of adverse events—has led to the development and expansion of publicly accessible databases that contain transparent information about clinical trials and their results.

Abstract

Study publication bias is the decision to publish or not publish a study based on its results. Compared to unpublished work, published studies are more likely to have positive or statistically significant findings. Outcome reporting bias is opting to publish only a subset of the original variables recorded for a study, such that the inclusion of the variables in the published work is selectively based on the results. Statistically significant results have a higher likelihood of being fully reported compared to nonsignificant results, and a significant proportion of published articles describe outcome variables or data analyses that differ from the pre-specified trial protocol as originally conceived. Recognition that publication bias and outcome reporting bias contribute to a distorted perception of drug effects—inflated estimates of efficacy and underreporting of adverse events—has led to the development and expansion of publicly accessible databases that contain transparent information about clinical trials and their results.

Dr. Howland is Associate Professor of Psychiatry, University of Pittsburgh School of Medicine, Western Psychiatric Institute and Clinic, Pittsburgh, Pennsylvania.

The author discloses that he has no significant financial interests in any product or class of products discussed directly or indirectly in this activity. The author discloses past grant support from Novartis.

Address correspondence to Robert H. Howland, MD, Associate Professor of Psychiatry, University of Pittsburgh School of Medicine, Western Psychiatric Institute and Clinic, 3811 O’Hara Street, Pittsburgh, PA 15213; e-mail: HowlandRH@upmc.edu.

Posted Online: July 15, 2011

Bias can be defined broadly as any tendency that prevents an unprejudiced consideration of a question or issue. Unbiased information about the effectiveness of newer and older medications—their efficacy, tolerability, and safety—is critically important for nurses in clinical practice. In this article, I will briefly describe the concepts of publication bias and outcome reporting bias. I will then describe how overcoming these biases can be done by accessing clinical trials results directly.

Publication Bias and Outcome Reporting Bias

Useful medication information can be gained from a clinician’s personal experience in treating patients, but the limitations of acquiring knowledge based solely on individual experience should be inherently obvious. For drugs approved by the U.S. Food and Drug Administration (FDA), information on a drug for a specific indication is available according to the product labeling. This does not necessarily incorporate all information about a drug, such as off-label uses and any updated information. A broader perspective on medication effectiveness for approved and off-label uses comes from many types of clinical studies, but randomized controlled trials (RCTs) are considered the most important source of information. To minimize bias, RCT protocols are planned in advance to have explicit blinding and randomization procedures, specific inclusion and exclusion criteria, definite primary and secondary outcome measures and endpoints, and a pre-determined statistical and data analysis plan (Pannucci & Wilkins, 2010).

Study publication bias is the decision to publish or not publish a study based on its results (Dwan et al., 2008; McGauran et al., 2010). Compared with unpublished work, for example, published studies are more likely to have positive or statistically significant findings. Investigators, study sponsors, and journal editors each have separate as well as overlapping reasons for not being interested in publishing negative or insignificant findings; they all share responsibility for publication bias (Young, 2009).

Outcome reporting bias is opting to publish only a subset of the original variables recorded for a study, such that the inclusion of the variables in the published work is selectively based on the results (Dwan et al., 2008; McGauran et al., 2010). Statistically significant results have a higher likelihood of being fully reported compared with nonsignificant results. Also, a significant proportion of published articles describe outcome variables or data analyses that differ from the pre-specified trial protocol as originally conceived. After reviewing the raw data from a completed study, those variables or analyses having more favorable outcomes might then be selected for inclusion in a published report of the study.

Clinical Trial Registries

Finding published studies is easily accomplished by conducting literature searches (e.g., through MEDLINE or other databases). Until recently, finding summaries of trial protocols and the results from completed studies that are not published has been a much more difficult task. Recognition that publication bias and outcome reporting bias contribute to a distorted perception of drug effects has led to the development and expansion of accessible databases that contain transparent information about clinical trials and their results.

FDA

The FDA Modernization Act of 1997 mandated the development of a publicly accessible database on clinical trials (FDA, 2009). As a result, the website http://ClinicalTrials.gov was developed in 2000 to provide access to the Clinical Trials Data Bank, which contains useful information on studies of drugs regulated by the FDA. The website was created and is managed by the National Library of Medicine. According to a final guidance rule issued by the FDA in 2002, the Clinical Trials Data Bank is required to contain information about federally and privately funded clinical trials for experimental treatments, a description of the purpose of each experimental drug, the basic study design (including primary and secondary outcome measures), patient eligibility criteria, a description of the location of clinical trial sites, a point of contact for patients interested in enrollment, and the study sponsor. The FDA Amendments Act of 2007 further mandated that the Clinical Trials Data Bank be expanded to provide “basic-results” data for clinical trials registered on ClinicalTrials.gov (Tse, Williams, & Zarin, 2009). The type of “basic results” information to be reported includes scientific information on participant flow (e.g., number of participants who enrolled, dropped out, and completed the study), baseline demographic and clinical characteristics, outcome measures and statistical analyses (for all pre-specified primary and secondary outcomes), and adverse events, as well as administrative information (i.e., a point of contact to obtain more information about the reported results).

European Medicines Agency

For the European Union, the European Medicines Agency (EMA; http://www.ema.europa.eu) is the agency responsible for the evaluation of medicinal products. The Committee for Medicinal Products for Human Use (CHMP) of the EMA prepares scientific opinions on all questions concerning medicines for human use. The CHMP publishes a European Public Assessment Report (EPAR) for drugs that it evaluates. The EPAR summarizes all preclinical and clinical studies, including unpublished data—submitted with the drug application—and describes the scientific reasoning for CHMP opinions. For drugs that are granted authorization (approved by the EMA), the EPAR includes a summary of product characteristics, the labeling and package leaflet (patient information sheet) for the drug, and details of the procedural steps taken during the assessment process by the CHMP (EMA, 2010). The EPARs are published on the EMA website and generally provide more unpublished data than are available through the FDA.

Pharmaceutical Companies and Trade Groups

The Pharmaceutical Research and Manufacturers of America (PhRMA) is a trade group representing pharmaceutical research and technology companies. In 2004 (in response to the FDA guidance rule of 2002), PhRMA created the Clinical Study Results Database to improve the transparency of clinical studies on new medications. The resulting website ( http://www.ClinicalStudyResults.org) is a centralized, publicly accessible, and searchable repository for clinical study results. This database was developed for the purpose of making clinical trial results for many marketed medications more transparent by including unpublished data. The database provides a link to an electronic version of the drug label (if available), a bibliography of published articles on the drug with a link to each article (if available), and a complete summary of each trial. The trial summary includes information on the purpose of the study, the basic study design, patient eligibility criteria, location of clinical trial sites, a point of contact for patients interested in enrollment, and the study sponsor. For completed studies, the database includes a summary of the results in a standard nonpromotional format that includes a description of the trial design and method, results of the primary and secondary outcome measures, and safety results. With the expansion of some individual company databases and the ClinicalTrials.gov website, the PhRMA database is scheduled to be phased out by the end of 2011.

The International Federation of Pharmaceutical Manufacturers & Associations (IFPMA) is a global nonprofit, nongovernmental organization representing research-based pharmaceutical, biotechnology, and vaccine companies. The IFPMA has a publicly accessible Clinical Trials Portal ( http://clinicaltrials.ifpma.org), which serves as a repository for information on new or ongoing clinical trials. The database provides information on the purpose of the study, the basic study design, patient eligibility criteria, location of clinical trial sites, a point of contact for patients interested in enrollment, and the study sponsor. When studies are completed, the database includes a summary of the results in a standard nonpromotional format that includes a description of the trial design and method, results of the primary and secondary outcome measures, and safety results.

Some manufacturers have their own publicly accessible clinical trials database websites. Interested individuals should contact a company directly to determine whether they have such a website and how to access the information. One example is the Novartis Clinical Trial Results Database ( http://www.novctrd.com). Similar to other clinical trial databases, this site contains information on the purpose of the study, the basic study design and method, patient eligibility criteria, and a summary of the efficacy and safety results.

Conclusion

Publication bias and outcome reporting bias contribute to a distorted perception of drug effects, especially inflated estimates of efficacy and underreporting of adverse events (Ioannides, 2009; Young, Ioannides, & Al-Ubaydli, 2008). Accessible databases that contain transparent information about clinical trials and their results—regardless of publication or outcome—are an important resource for nurses to obtain unbiased information about drug effectiveness. Because these databases are publicly available, nurses should be aware that patients and families might access this information and have questions or concerns about what they find.

References

Authors

Dr. Howland is Associate Professor of Psychiatry, University of Pittsburgh School of Medicine, Western Psychiatric Institute and Clinic, Pittsburgh, Pennsylvania.

The author discloses that he has no significant financial interests in any product or class of products discussed directly or indirectly in this activity. The author discloses past grant support from Novartis.

Address correspondence to Robert H. Howland, MD, Associate Professor of Psychiatry, University of Pittsburgh School of Medicine, Western Psychiatric Institute and Clinic, 3811 O’Hara Street, Pittsburgh, PA 15213; e-mail: .HowlandRH@upmc.edu

10.3928/02793695-20110705-06

Sign up to receive

Journal E-contents