Skip to Main Content

Systematic reviews and other evidence synthesis projects

To Do during this step

  • Check for retractions
  • Critically appraise the quality of the remaining studies
  • Record the data from this step in PRISMA flow diagram

Check for Retractions

Retraction is a mechanism for correcting the literature when an article has errors or flaws so serious that it should not be relied upon. It's a means for correcting the literature – part of a healthy science ecosystem!
A retraction doesn't necessarily mean intentional wrongdoing on the part of the authors; it can just be honest error.

Ways to check for retractions:

  • Retraction Watch database: search by DOI or PubMed ID
    • Zotero and EndNote have partnered with Retraction Watch to identify retracted articles in your citation manager library.
  • Terms to watch for in abstracts and article records: retraction, retracted, erratum, errata, “expression of concern”, comments, author responses to comments, "Update in: [article title]”.
  • Search for retraction information on this set of articles in databases using filters like "Retracted Publication" in PubMed, "Tombstone" in Embase, or filters called any of the words/phrases in Terms to watch for.

Critical appraisal of results

After the full-text screening, you will have a much smaller pool of results. Before you start to extract and analyze your data, you will need to to assess risk of bias in your results; sometimes referred to as critical assessment or quality appraisal. This step is an important part of why systematic reviews and meta-analysis are considered to be a high level of evidence.

According to the Institute of Medicine standards, appraisal for a systematic review requires the authors to:

  • 3.6.1 Systematically assess the risk of bias, using predefined criteria
  • 3.6.2 Assess the relevance of the study’s populations, interventions, and outcome measures
  • 3.6.3 Assess the fidelity of the implementation of interventions
About bias and appraisal in general:

Appraisal Checklists

Different types of studies have different appraisal criteria. There are appraisal support tools to assist you with many of them.

Collections of appraisal checklists for a variety of study types:

Appraisal checklists for specific study types:

Appraising Grey Literature

As with journal articles included in reviews, grey literature must be appraised before incorporating it into a review.

  • Unpublished studies and clinical trials: appraise using the same tools as published studies and trials.
  • Other types of grey literature: AACODS checklist
    • provides guidance for evaluating Authority, Accuracy, Coverage, Objectivity, Date, Significance in the material.
    • developed by Jess Tyndall, Medical Librarian and Head of the Gus Fraenkel Medical Library at Flinders University as an evaluation and Critical Appraisal Tool specifically for use with grey literature sources
    • can be applied to materials across formats and disciplines

Still not sure about an article?

There can be a lag time of 1-5 years between publication and retraction. If something about an article still seems off, you can:

  • Check for information about the article on listervs, electronic bulletin boards on the topic, colleagues, journal comments, altmetrics (Twitter, blogs, news--though use careful judgement about these!)
  • Apply REAPPRAISED checklist

     

How will YOUR review be appraised?

Guides to critically appraising systematic reviews and meta-analyses are a great resource for learning what needs to be present in a high-quality SR.
Try appraising your own review before submitting it for publication!

Guides: