PREMIER Planning of Experiments
Objectives
PREMIER's objective is to support researchers in the detailed planning of projects / experiments before the beginning in order to implement a suitable test procedure in which all risks are considered and avoided in advance and feasibility is implemented with a suitable experimental design.
Background
In recent years, Meta-Research has identified significant deficits in the planning, implementation, analysis and reporting of results from biomedical research. The lack of reproducibility, even of findings published in well-known journals, has brought up the term "replication crisis" and is probably at least partly responsible for the low success rate in translating often spectacular findings from preclinical research into clinically effective therapies. To overcome this phenomenon, experimental neurology has developed structured processes to make projects transparent and comprehensible in their planning, implementation, analysis and reporting.
Accomplishment of projects/experiments requires careful planning as the first step, since a large number of variables can influence the project. The PREMIER approach to this challenge is described in detail below.
Tasks / Actions
In order to create a lab specific action plan, the first step is an assessment, which will be carried out by the PREMIER team. The assessment will determine the status quo of the laboratory in regard to existing quality tools. Here you find the general tasks / actions that are necessary to implement the module.
Issues
The design of experiments is carried out in several steps and starts with following questions.
- What is the goal of the research project?
- Is it realistically achievable?
- Is the research question relevant in the research context?
- Who will benefit from the project/results?
Two Study Modes: Exploration vs. Confirmation
In the exploratory investigation, researchers should aim to develop robust pathophysiological theories of diseases.
In the confirmatory investigation (confirmation of hypotheses), researchers should collect strong and reproducible treatment effects in relevant animal models.
We should separate these two modes and adapt the design and reporting guidelines for each mode to the requirements.
The table gives a first overview of how the distinction between exploratory and confirmatory studies can lead to different study designs.
| Exploratory (Discovery) | Confirmatory | |
| Hypothesis | (+) | (+++) |
| Establish pathophysiology | (+++) | (+) |
| Sequence and detail of experiments at onset | (+) | (+++) |
| Defined primary and point | (-) | (++) |
| Sample size calculation | (+) | (+++) |
| Blinding | (+++) | (+++) |
| Randomization | (+++) | (+++) |
| External validity (aging, comorbidities) | (-) | (++) |
| Predefined inclusion/exclusion criteria | (++) | (+++) |
| Test statistics | (+) | (+++) |
| Preregistration | (-) | (++) |
| High sensitivity (high type I error rate, low type II error rate): find what might work | (+++) | (+) |
| High specifity (low type I error rate, high type II error rate): weed out false-positives | (+) | (+++) |
Process of Experimental Design
Depending on the type of project, the following steps must be considered and answered for a comprehensive and complete design of experiments:
- Search
- Hypothesis / Counter (null)-Hypothesis
- Target Parameters
- Sample Size Calculation
- Model Planning / Study Design
- Feasibility Study
- Nesting and Pseudoreplication
- Randomisation and Blinding
- Resource Plan (financing, capacities, personnel)
- Schedule
- Accompanying Training and Courses
- Planning of Data Preparation / Analysis
- Data Storage
- Clarification of Authorship
- Pre-registration
Template Experimental Design
This background information is also linked to a template created included here. Thus, it is possible for every scientist to enter, edit and save the project directly in the template. If this template is included in any form of electronic laboratory notebook, all information will be located in one place and stored safely (at least 10 years) according to the Good Scientific Practice guidelines.
A video tutorial on the use of templates for design of experiments and the export as PDF can be found under Template
Changes
Changes that have occurred during the design of the experiment and during the project must always be documented and explained in the ELN, e.g. if something has changed in the progress of the project with regard to the previously agreed authorship or the core method. Transparency and traceability of the entire research process are absolutely necessary and mandatory in order to generate robust and reliable results.
Sustainability
Every researcher should finally think about how and with what means the project can achieve sustainability. Are there factors that could affect the project and its success in the long run? If so, how can these factors be reduced in advance?
References
- Experimental Design Assistant (EDA) Website at the NC3Rs https://eda.nc3rs.org.uk/
- Experimental Design Assistan EDA Video (Youtube video. Only accessible if you agree to Google conditions)
- IACUC: Resources and Links for Animal Subjects: Experimental Design and Statistical Analysis in Animal Studies (PDF) http://blink.ucsd.edu/_files/sponsor-tab/iacuc/Guidelines.pdf
- Dirnagl U. Thomas Willis Lecture: Is Translational Stroke Research Broken, and if So, How Can We Fix It? Stroke. 2016 Aug;47(8):2148-53. doi:10.1161/STROKEAHA.116.013244. PubMed PMID: 27354221.[1]
- Kimmelman J, Mogil JS, Dirnagl U. Distinguishing between exploratory and confirmatory preclinical research will improve translation. PLoS Biol. 2014 May 20;12(5):e1001863. doi: 10.1371/journal.pbio.1001863. PubMed PMID: 24844265. [2]
- Lindner MD, Torralba KD, Khan NA. Scientific productivity: An exploratory study of metrics and incentives. PLoS One. 2018 Apr 3;13(4):e0195321. doi: 10.1371/journal.pone.0195321. PubMed PMID: 29614101. [3]
- Discuss at PREMIER forum