Additional Resources for Study Designs
Experimental Design by
Experimental Design: Procedures for Behavioral Sciences, Fourth Edition is a classic text with a reputuation for accessibility and readability. It has been revised and updated to make learning design concepts even easier. Roger E. Kirk shows how three simple experimental designs can be combined to form a variety of complex designs. He provides diagrams illustrating how subjects are assigned to treatments and treatment combinations. New terms are emphasized in boldface type, there are summaries of the advantages and disadvantages of each design, and real-life examples show how the designs are used.
Publication Date: 2013
Appraising the Literature
Critical appraisal is a systematic analysis of research articles to determine the strength of the evidence in reference to your clinical question. Systematically review the different aspects (below) of each study to answer the following questions:
- What were the strengths of the study and the overall rigor of the study in terms of credibility?
- Quality of evidence is based on the level and strength of the study.
- Level of evidence: Various scales have been developed to rank; e.g., Pedro scale
- Strength of evidence is based on the methodological limitations and/or threats to validity that may affect interpretation of findings and generalization of results.
- Quality of evidence is based on the level and strength of the study.
- How are the results of this study relevant to your clinical question? How might the results influence clinical practice?
- Applicability of evidence is based on its relevance to your question
- How are the results of this study relevant to your clinical question? How might the results influence clinical practice?
Study Designs & Level of Evidence
- What is the study design?
- Nock MK. Single-case research designs (Chap 22 in Handbook of research methods in abnormal and clinical psychology)
- Types of designs in Center for Social Research Methods Research Methods Knowledge Base
- Understanding Research Study Designs, University of Minnesota LibGuide
- Study designs Short article from CEBM describing different study designs
- Qualitative study design examples: phenomenology, ethnography, grounded theory, participatory action research.
- Quantitative study design examples: randomized (RCT), cohort, single-case, before and after, case-control, cross-sectional or case study
- What is the level of evidence?
- The level of evidence can be illustrated using an Evidence-Based Practice Pyramid.
Setting & Participants
Setting: Was the setting appropriate to answer the research question?
Participants/Sample:
- How were participants recruited?
- What were the inclusion/exclusion criteria?
- How many participants participated? How many were lost through attrition?
- What were the participant demographics?
- If more than one group, was there similarity between groups?
Intervention & Outcome Measures
Intervention(s):
- Was the intervention clearly described?
- Who delivered the intervention and how often?
- Was there cross-contamination between interventions?
- Was there a break-in period?
Outcome measure(s):
- What instruments or methods were used to measure the variables? Examples include participant observation, interviews, focus groups, instruments, devices & questionnaires.
- Did the authors use measures with documented evidence of validity and reliability?
- Was the procedural reliability documented?
- How frequent were the participants measured?
Results & Conclusions
Main Results or Key Findings:
- What were the results?
- Was there statistical difference? What was the effect size?
- How were the results analyzed and were the analysis methods appropriate?
Authors interpretation/conclusion: What was the clinical relevance of the study and the results?
Critical Appraisal
- What are the strengths and weaknesses of the study?
- Were interventions delivered and data collected systematically, objectively and with fidelity?
- What were the potential threats to internal validity?
- Examples of potential threats to internal validity: history, maturation, testing, instrumentation, statistical regression, selection, mortality, interactions with selection, ambiguity about causal influence, and diffusion of intervention.
- What were the potential threats to external validity?
- Examples of potential threats to external validity: interaction of testing and treatment, interaction of selection and treatment, interaction of setting and treatment, interaction of history and treatment, multiple-treatment interference.
- Description of external validity
- Were there limitations of the study to answer your clinical question?
- Rate the study quality – this may be rated on a scale of 1-5 or 1-3.
- Level of evidence is based on the study design(s)
- Quality of evidence is based on the methodological strengths and weaknesses.
- Applicability of evidence is based on its relevance to your question
- Various scales have been developed to rank; e.g., Pedro scale
- Your overall summary and interpretation of the study:
- What were the strengths of the study and the overall rigor of the study in terms of credibility?
- How are the results of this study relevant to your clinical question? How might the results influence clinical practice?
