Skip to Main Content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

Evidence Synthesis: Steps in a Review

1. Develop a Research Question

Formulating a research question is key to a systematic review. It will be the foundation upon which the rest of the research is built. At this stage in the process, you will have identified a knowledge gap in your field, and you are aiming to answer a specific question. For example:

If X is prescribed, what happens to Y patients?

or assess and intervention:

How does X affect Y?

or synthesize existing evidence:

What is the nature of X?

Developing a research question takes time. You will likely go through different versions before settling on a final question. Once you've developed your research question, you will use it to create a search strategy.

Frameworks help to break your question into parts so you can clearly see the elements of your topic. Depending on your field of study, the frameworks listed in this guide may not fit the types of questions you're asking. There are dozens of frameworks you can use to formulate your specific and answerable research question. To see other frameworks you might use, visit the University of Maryland's Systematic Review guide.

The most common framework for systematic reviews is PICO, which is often used within the health sciences for clinical research, or in education. It is commonly used for quantitative studies.

PICO

P: Population

I: Intervention/Exposure

C: Comparison

O: Outcome

Example: In 11-12 year old children (Population), what is the effect of a school-based multi-media learning program (Intervention) on an increase in real-world problem solving skills compared with analog-only curriculum (Comparison) within a one-year period (Time)?

Source: Richardson, W. S., Wilson, M. C., Nishikawa, J., & Hayward, R. S. (1995). The well-built clinical question: A key to evidence-based decisionsACP journal club, 123(3), A12-A12.

PICo

P: Population/problem

I: Phenomenon of Interest

Co: Context

Example: What are the experiences (phenomenon of interest) of caregivers providing home based care to patients with Alzheimer's disease (population) in Australia (context)?

Source: Methley, A.M., Campbell, S., Chew-Graham, C. et al. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res 14, 579 (2014). https://doi.org/10.1186/s12913-014-0579-0

________________________________________________________________________

CHIP

C: Context

H: How

I: Issues

P: Population

Example: 

Source: Shaw, R. (2010). Conducting literature reviews. In M. A. Forester (Ed.), Doing Qualitative Research in Psychology: A Practical Guide (pp. 39-52). London, Sage.

SPICE

S: Setting

P: Perspective

I: Intervention/Exposure/Interest

C: Comparison

E: Evaluation

Example: What are the benefits (evaluation) of a doula (intervention) for low income mothers (perspective) in the developed world (setting) compared to no support (comparison)?

Source: Booth, A. (2006). Clear and present questions: Formulating questions for evidence based practice. Library Hi Tech, 24(3), 355-368. https://doi.org/10.1108/07378830610692127

________________________________________________________

SPIDER

S: Sample

PI: Phenomenon of Interest

D: Design

E: Evaluation

R: Research Type

Example: What are the experiences (evaluation) of women (sample) undergoing IVF treatment (phenomenon of interest) as assessed?

Design:  questionnaire or survey or interview

Study Type: qualitative or mixed method

Source: Cooke, A., Smith, D., & Booth, A. (2012). Beyond PICO: The SPIDER tool for qualitative evidence synthesis. Qualitative Health Research, 22(10), 1435-1443. https://doi.org/10.1177/1049732312452938

2. Select a Reporting Guideline

If you plan to have your systematic review published, you will want to follow reporting guidelines that outline what you will include in your manuscript. Reporting guidelines provide some consistency to published systematic reviews, and they increase the likelihood that your research could be replicated by other researchers. There are different reporting guidelines for different disciplines. Below are a few to consider:

3. Select Databases

Librarians can assist you with selecting databases for your systematic review. Each database is different and will require a different search syntax. Some databases have controlled vocabulary and thesauri that you will want to incorporate into your searches. We recommend creating one master search strategy and then translating it for each database. 

To begin browsing databases, visit the A-Z Database List:

4. Select Grey Literature Sources

Grey (or gray) Literature is "A variety of written materials produced by organizations outside of traditional commercial and academic publishing channels, such as annual reports, [theses and dissertations], white papers, or conference proceedings from government agencies, non-governmental organizations, or private companies. Grey literature may be difficult to access because it may not be widely distributed or included in bibliographic databases." 

Your research question and field of study will guide what type of grey literature to include in your systematic review. 

Source: Byrne, D. (2017). Reviewing the literature. Project Planner. 10.4135/9781526408518.

The purpose of a systematic review is to identify and synthesize all available evidence. There is significant bias in scientific publishing toward publishing studies that show some sort of significant effect. In fact, according to Campbell Collaboration Guidelines on Information Retrieval, more than 50% of studies reported in conference abstracts never reach full publication. While conference abstracts and other grey literature is not peer-reviewed, it is important to include all available research on the topic you're studying.

Finding grey literature on your topic may require some creativity, and may involve going directly to the source. Here are a few tips:

  • Find a systematic review on a topic similar to yours and see what grey literature sources they used. You can find existing systematic reviews in subject databases, The Campbell Library, and the Cochrane Library. In databases such as PsycINFO, you can use the Methodology search tool to narrow by Systematic Review or Meta-Analysis; otherwise check the thesaurus for controlled vocabulary or use the keyword search to add ("systematic review" OR meta-analysis OR "scoping review") to your search string.
  • Ask colleagues and other experts in the field for sources of grey literature in your discipline.
  • Contact known researchers in the field to learn if there are any unpublished or ongoing studies to be aware of.
  • On the web, search professional associations, research funders, and government websites.
In addition to the sources below, you may find grey literature such as conference proceedings in subject databases such as Scopus and ACM Digital Library.

5. Write a Search Strategy

Use the keywords from your research question and begin to create a core keyword search that can then be translated to fit each database search. Since the goal is to be as comprehensive as possible, you will want to identify all terms that may be used for each of the keywords, and use a combination of natural language and controlled vocabulary when available. Librarians are available to assist with search strategy development and keyword review.

Your core keyword search will likely include some or all of the following syntax:

  • Boolean operators (AND, OR, and NOT) 
  • Proximity operators (NEAR or WITHIN)
  • Synonyms, related terms, and alternate spellings
  • Controlled vocabulary (found within the database thesaurus)
  • Truncation (ex: preg* would find pregnant and pregnancy)

Search filters that are built into databases may also be used, but use them with caution. Database articles within the social sciences tend not to be as consistently or thoroughly indexed as those within the health sciences, so using filters could cause you to miss some relevant results.

Source: Kugley S, Wade A, Thomas J, Mahood Q, Jørgensen AMK, Hammerstrøm K, Sathe N. Searching for studies: A guide to information retrieval for Campbell Systematic Reviews. Campbell Methods Guides 2016:1 DOI: 10.4073/cmg.2016.1

6. Register a Protocol

A protocol is a detailed explanation of your research project that should be written before you begin searching. It will likely include your research question, objectives, and search methodology, but information included within a protocol can vary across disciplines. The protocol will act as a map for you and your team, and will be helpful in the future if you or any other researchers want to replicate your search. Protocol development resources and registries:

7. Translate Search Strategies

Each database is different and will require a customized search string. We recommend creating one master keyword list and then translating it for each database by using that database's subject terms and search syntax. Below are some tools to assist with translating search strings from one database to the next.

8. Manage your Citations

When conducting a systematic review, you will likely be exporting hundreds or even thousands of citations from databases. Citation management tools are useful for storing, organizing, and managing your citations. They can also perform de-duplication to remove doubles of any citations you may have. The Libraries provide training and support on EndNote, Zotero, and Mendeley. Visit the links below to get started. You may also reach out directly to BIBSUPPORT@LISTSERV.IT.NORTHWESTERN.EDU with questions or consultation requests.

9. Article Screening

During the screening process, you will take all of the articles you exported from your searches and begin to remove studies that are not relevant to your topic. Use the inclusion/exclusion criteria you developed during the protocol-writing stage to screen the title and abstract of the articles you found. Any studies that don't fit the criteria of your review can be deleted. The full text of the remaining studies will need to be screened to confirm that they fit the criteria of your review.

It is highly recommended that two independent reviewers screen all studies, resolving areas of disagreement by consensus or by a third party who is an expert in the field. Listed below are tools that can be used for article screening.

10. Assess the Risk of Bias

Bias refers to factors that can systematically affect the observations and conclusions of the study, causing them to be inaccurate. When compiling studies for systematic reviews, it is best practice to assess the risk of bias for each of the studies included, and then include the assessment in your final manuscript. The Cochrane Handbook recommends presenting the assessment as a table or graph.

In general, scoping reviews don't require a risk of bias assessment, but according to the PRISMA Scoping Review checklist, scoping reviews should include a "critical appraisal of individual sources of evidence." In a final manuscript, a critical appraisal could be an explanation of the limitations of the studies included.

Source: Andrea C. Tricco, Erin Lillie, Wasifa Zarin, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med.2018;169:467-473. [Epub ahead of print 4 September 2018]. doi:10.7326/M18-0850

11. Extract the Data

Once you and your team have screened all of the studies to be included in your review, you will need to extract the data from the studies in order to synthesize the results. You can use Excel or Google Forms to code the results. Additional resources below.

12. Synthesize, Map, or Describe the Results

In the data synthesis section, you will present the main findings of your evidence synthesis. There are multiple ways you could go about synthesizing the data, and that decision will depend largely on the type of studies you're synthesizing. In any case, it is standard to use the PRISMA flow diagram to map out the number of studies identified, screened, and included in your evidence synthesis project.

Meta-Analysis

A quantitative statistical analysis that combines the results of multiple studies. The studies included must all be attempting to answer the same research question and have a similar research design. According to the Cochrane Handbook, "meta-analysis yields an overall statistic (together with its confidence interval) that summarizes the effectiveness of an experimental intervention compared with a comparator intervention."

Narrative or Descriptive

If you've included studies that are not similar in research design, then a meta-analysis is not possible. You will then use a narrative or descriptive synthesis to describe the results.