Introduction to the design and analysis of experiments
“A useful property of a test of significance is that it exerts a sobering influence on the type of experimenter who jumps to conclusions on scanty data, and who might otherwise try to make everyone excited about some sensational treatment effect that can well be ascribed to the ordinary variation in [their] experiment.”
Key phrases
Key phrases
An experiment is a procedure (or set of actions) where a researcher intentionally changes some factor/treatment/variable to observe the effect of their actions. As mentioned above, the collection of observational data is not experimentation.
An experimental unit is the smallest portion of experimental material which is independently perturbed. This is the item under study for which some variable (treatment) is changed. For example this could be a human subject or an agricultural plot.
An observational unit (or subsample) is the smallest unit on which a response is measured. If the experimental unit is split after the treatment has been applied (e.g., multiple samples taken from one human subject) then this sample is called a subsample or observational unit. If one measurement is made on each experimental unit then the observational unit = the experimental unit. If multiple measurements are made on each subject (e.g., human) then each experimental unit has >1 observational unit. This is then pseudo- or technical replication (see below).
A treatment (or independent variable or factor or treatment factor) is an experimental condition independently applied to an experimental unit. It is one of the variables that is controlled by the researcher during the experiment (e.g., drug type). The values of the treatments within a set are called levels.
The dependent variable or response is the output (or thing) that is measured after an experiment. This is what the researcher measures and assesses if changing the treatment(s) (i.e., independent variable(s)) induces any change.
An effect is the change in the response variable caused by the controlled changes in the independent variable. Whether the magnitude of the effect (it’s size) is significant (or or any practical interest) is determined by the researcher after carrying out some appropriate analyses.
Setting up an experiment
In the previous chapter where we discussed [Reproducibility]. This also applies to the design and analysis of experiments. In order to future proof our research we should make every effort to ensure others can reproduce it. To do so we should be specific about our goals and procedures by
- Defining the goals/objectives of our research,
- Formulating a specific hypothesis,
- Specifying the response variable(s) that will be measured,
- Specifying the treatments that will be tested and describing the process of applying these treatments to the experimental units,
- Outlining the procedure for observing and recording responses to assess treatment effects,
- Identifying factors that may contribute to variability in the results (expected and otherwise), and
- Describing the statistical or methods that will be employed to the our hypothesis.
Following these guidelines not only helps our experiment, but makes sure others can reproduce it. For example, defining specific objectives directs you towards writing focused statements about the investigative questions you want your experiment to answer. Listing the experimental factors (or treatments or independent variables) you will study in your experiment helps to organize variables and work out how they may help to explain observed changes in your measurable response(s). It is important that the experimental factor can be controlled during and between experimental runs. Variables that are thought to affect the response, but cannot be controlled, cannot be included as an experimental factor.
An example
| Experiment Title | Barista Brews |
|---|---|
| Researcher Name | Dr Java |
| Researcher Institute | Central Perk |
| Objective | The objective of this experiment is to determine what type of grind and brew ratio (i.e., amount of coffee in relation to water) leads to the strongest coffee. |
| Hypothesis | That a finer grained coffee with a higher brew ratio (i.e., less water to coffee) will lead to the strongest coffee. |
| Response variable | Total Dissolved Solids (TDS), a measure of soluble concentration (%). |
| Treatments | A. Brew ratio i) 2 parts coffee to 1 part water (2-1) ii) 1 part coffee to 1 part water (1-1) iii) 1 part coffee to 2 parts water (1-2) B. Grind i) Fine ii) Coarse |
| Experimental material | Individual cups, maintaining consistency in cup size, water amount, and boiling temperature. |
| How treatments will be applied | Independently to individual cups. Each cup will be made independently, not as a batch distributed among several cups. |
| How measurements will be taken | Treatments applied independently to individual cups. Each cup with the same treatment (brew ratio and grind) will be subject to the same environmental conditions. |
| Experimental units | Replicate each treatment twice. Assign 2 experimental units (cups) to each unique treatment combination. |