Formulating a research question is key to a systematic review. It will be the foundation upon which the rest of the research is built. At this stage in the process, you will have identified a knowledge gap in your field, and you are aiming to answer a specific question. For example:
If X is prescribed, what happens to Y patients?
or assess and intervention:
How does X affect Y?
or synthesize existing evidence:
What is the nature of X?
Developing a research question takes time. You will likely go through different versions before settling on a final question. Once you've developed your research question, you will use it to create a search strategy.
Frameworks help to break your question into parts so you can clearly see the elements of your topic. Depending on your field of study, the frameworks listed in this guide may not fit the types of questions you're asking. There are dozens of frameworks you can use to formulate your specific and answerable research question. To see other frameworks you might use, visit the University of Maryland's Systematic Review guide.
The most common framework for systematic reviews is PICO, which is often used within the health sciences for clinical research, or in education. It is commonly used for quantitative studies.
PICO
P: Population
I: Intervention/Exposure
C: Comparison
O: Outcome
Example: In 11-12 year old children (Population), what is the effect of a school-based multi-media learning program (Intervention) on an increase in real-world problem solving skills compared with analog-only curriculum (Comparison) within a one-year period (Time)?
Source: Richardson, W. S., Wilson, M. C., Nishikawa, J., & Hayward, R. S. (1995). The well-built clinical question: A key to evidence-based decisions. ACP journal club, 123(3), A12-A12.
PICo
P: Population/problem
I: Phenomenon of Interest
Co: Context
Example: What are the experiences (phenomenon of interest) of caregivers providing home based care to patients with Alzheimer's disease (population) in Australia (context)?
Source: Methley, A.M., Campbell, S., Chew-Graham, C. et al. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res 14, 579 (2014). https://doi.org/10.1186/s12913-014-0579-0
________________________________________________________________________
C: Context
H: How
I: Issues
P: Population
Example:
Source: Shaw, R. (2010). Conducting literature reviews. In M. A. Forester (Ed.), Doing Qualitative Research in Psychology: A Practical Guide (pp. 39-52). London, Sage.
S: Setting
P: Perspective
I: Intervention/Exposure/Interest
C: Comparison
E: Evaluation
Example: What are the benefits (evaluation) of a doula (intervention) for low income mothers (perspective) in the developed world (setting) compared to no support (comparison)?
Source: Booth, A. (2006). Clear and present questions: Formulating questions for evidence based practice. Library Hi Tech, 24(3), 355-368. https://doi.org/10.1108/07378830610692127
________________________________________________________
SPIDER
S: Sample
PI: Phenomenon of Interest
D: Design
E: Evaluation
R: Research Type
Example: What are the experiences (evaluation) of women (sample) undergoing IVF treatment (phenomenon of interest) as assessed?
Design: questionnaire or survey or interview
Study Type: qualitative or mixed method
Source: Cooke, A., Smith, D., & Booth, A. (2012). Beyond PICO: The SPIDER tool for qualitative evidence synthesis. Qualitative Health Research, 22(10), 1435-1443. https://doi.org/10.1177/1049732312452938
Scoping reviews generally have a broader scope that systematic reviews, but it is still helpful to put scoping and mapping reviews within a framework. The Joanna Briggs Institute offers guidance on forming scoping review questions in Chapter 11 of their manual for evidence synthesis. They recommend using the PCC framework:
P: Population
C: Concept
C: Context
Example: What are the trends (concept) in MOOCs (context) that support the interactions of learners with disabilities (population)?
Source: Peters MDJ, Godfrey C, McInerney P, Munn Z, Tricco AC, Khalil, H. Chapter 11: Scoping Reviews (2020 version). In: Aromataris E, Munn Z (Editors). JBI Manual for Evidence Synthesis, JBI, 2020. Available from https://synthesismanual.jbi.global. https://doi.org/10.46658/JBIMES-20-12
Librarians can assist you with selecting databases for your systematic review. Each database is different and will require a different search syntax. Some databases have controlled vocabulary and thesauri that you will want to incorporate into your searches. We recommend creating one master search strategy and then translating it for each database.
To begin browsing databases, visit the A-Z Database List:
Grey (or gray) Literature is "A variety of written materials produced by organizations outside of traditional commercial and academic publishing channels, such as annual reports, [theses and dissertations], white papers, or conference proceedings from government agencies, non-governmental organizations, or private companies. Grey literature may be difficult to access because it may not be widely distributed or included in bibliographic databases."
Your research question and field of study will guide what type of grey literature to include in your systematic review.
Source: Byrne, D. (2017). Reviewing the literature. Project Planner. 10.4135/9781526408518.
The purpose of a systematic review is to identify and synthesize all available evidence. There is significant bias in scientific publishing toward publishing studies that show some sort of significant effect. In fact, according to Campbell Collaboration Guidelines on Information Retrieval, more than 50% of studies reported in conference abstracts never reach full publication. While conference abstracts and other grey literature is not peer-reviewed, it is important to include all available research on the topic you're studying.
Finding grey literature on your topic may require some creativity, and may involve going directly to the source. Here are a few tips:
Use the keywords from your research question and begin to create a core keyword search that can then be translated to fit each database search. Since the goal is to be as comprehensive as possible, you will want to identify all terms that may be used for each of the keywords, and use a combination of natural language and controlled vocabulary when available. Librarians are available to assist with search strategy development and keyword review.
Your core keyword search will likely include some or all of the following syntax:
Search filters that are built into databases may also be used, but use them with caution. Database articles within the social sciences tend not to be as consistently or thoroughly indexed as those within the health sciences, so using filters could cause you to miss some relevant results.
Source: Kugley S, Wade A, Thomas J, Mahood Q, Jørgensen AMK, Hammerstrøm K, Sathe N. Searching for studies: A guide to information retrieval for Campbell Systematic Reviews. Campbell Methods Guides 2016:1 DOI: 10.4073/cmg.2016.1
A protocol is a detailed explanation of your research project that should be written before you begin searching. It will likely include your research question, objectives, and search methodology, but information included within a protocol can vary across disciplines. The protocol will act as a map for you and your team, and will be helpful in the future if you or any other researchers want to replicate your search. Protocol development resources and registries:
Each database is different and will require a customized search string. We recommend creating one master keyword list and then translating it for each database by using that database's subject terms and search syntax. Below are some tools to assist with translating search strings from one database to the next.
When conducting a systematic review, you will likely be exporting hundreds or even thousands of citations from databases. Citation management tools are useful for storing, organizing, and managing your citations. They can also perform de-duplication to remove doubles of any citations you may have. The Libraries provide training and support on EndNote, Zotero, and Mendeley. Visit the links below to get started. You may also reach out directly to BIBSUPPORT@LISTSERV.IT.NORTHWESTERN.EDU with questions or consultation requests.
During the screening process, you will take all of the articles you exported from your searches and begin to remove studies that are not relevant to your topic. Use the inclusion/exclusion criteria you developed during the protocol-writing stage to screen the title and abstract of the articles you found. Any studies that don't fit the criteria of your review can be deleted. The full text of the remaining studies will need to be screened to confirm that they fit the criteria of your review.
It is highly recommended that two independent reviewers screen all studies, resolving areas of disagreement by consensus or by a third party who is an expert in the field. Listed below are tools that can be used for article screening.
Bias refers to factors that can systematically affect the observations and conclusions of the study, causing them to be inaccurate. When compiling studies for systematic reviews, it is best practice to assess the risk of bias for each of the studies included, and then include the assessment in your final manuscript. The Cochrane Handbook recommends presenting the assessment as a table or graph.
In general, scoping reviews don't require a risk of bias assessment, but according to the PRISMA Scoping Review checklist, scoping reviews should include a "critical appraisal of individual sources of evidence." In a final manuscript, a critical appraisal could be an explanation of the limitations of the studies included.
Source: Andrea C. Tricco, Erin Lillie, Wasifa Zarin, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med.2018;169:467-473. [Epub ahead of print 4 September 2018]. doi:10.7326/M18-0850
Once you and your team have screened all of the studies to be included in your review, you will need to extract the data from the studies in order to synthesize the results. You can use Excel or Google Forms to code the results. Additional resources below.
In the data synthesis section, you will present the main findings of your evidence synthesis. There are multiple ways you could go about synthesizing the data, and that decision will depend largely on the type of studies you're synthesizing. In any case, it is standard to use the PRISMA flow diagram to map out the number of studies identified, screened, and included in your evidence synthesis project.
Librarians can help write the methods section of your review for publication, to ensure clarity and transparency of the search process. However, we encourage evidence synthesis teams to engage statisticians to carry out their data syntheses.
Meta-Analysis
A quantitative statistical analysis that combines the results of multiple studies. The studies included must all be attempting to answer the same research question and have a similar research design. According to the Cochrane Handbook, "meta-analysis yields an overall statistic (together with its confidence interval) that summarizes the effectiveness of an experimental intervention compared with a comparator intervention."
Narrative or Descriptive
If you've included studies that are not similar in research design, then a meta-analysis is not possible. You will then use a narrative or descriptive synthesis to describe the results.