Statistics · Unit 3: Collecting Data · 14 min read · Updated 2026-05-11
How to Experiment Well — AP Statistics
AP Statistics · Unit 3: Collecting Data · 14 min read
1. Core Principles of Experimental Design★★☆☆☆⏱ 4 min
A well-designed experiment allows researchers to isolate the effect of an explanatory variable (called a factor) on a response variable, enabling valid causal inference that cannot be drawn from observational studies. Unlike observational studies, experiments impose treatments on experimental units (called subjects if they are human) to measure response. Per the AP Statistics CED, this topic makes up ~40% of Unit 3 and 4-6% of the total AP exam score.
**Control**: Account for lurking variables by including a comparison group, eliminating confounding where treatment effects are mixed with lurking variable effects. Control groups may receive no treatment, a placebo, or an existing standard treatment.
**Randomization**: Randomly assign experimental units to treatment groups, balancing both known and unknown lurking variables across groups on average.
**Replication**: Assign each treatment to multiple independent experimental units to reduce sampling variability, making it easier to detect real treatment effects.
2. Common Experimental Designs★★★☆☆⏱ 4 min
Experiments are structured into three common designs based on whether researchers know of any nuisance variables (variables that affect response but are not of interest) that need to be accounted for.
**Completely Randomized Design (CRD)**: The simplest design, where all experimental units are randomly assigned to treatments with no pre-grouping. Used when no known systematic differences between units exist.
**Randomized Block Design (RBD)**: Units are grouped into blocks by a known nuisance variable, so all units in a block are similar on the nuisance variable. Random assignment of treatments happens *within each block*. Blocking reduces unwanted variability, making it easier to detect treatment effects.
**Matched Pairs Design**: A special case of RBD where each block has exactly two units matched on similar characteristics. One unit gets each treatment. Alternatively, each unit gets both treatments in random order (repeated measures matched pairs), with the unit acting as its own block.
3. Scope of Inference★★★★☆⏱ 3 min
A key AP Statistics skill is identifying what types of valid conclusions can be drawn from an experiment, based on its design. There are two separate questions to answer for any study.
**Can we conclude the treatment caused the difference in response?**: Causal conclusions are only valid if treatments were *randomly assigned* to units. Without random assignment, you can only conclude association, not causation.
**Can we generalize results to a larger population?**: Generalization is only valid if experimental units were *randomly sampled* from the population of interest. A convenience sample (like volunteer students) does not allow generalization.
4. Concept Check★★★☆☆⏱ 3 min
Common Pitfalls
Why: Students confuse random sampling (for generalization) with random assignment (the core requirement for a valid experiment that allows causation).
Why: The simplified definition ignores that control groups often get a standard existing treatment or placebo, not no treatment.
Why: Students think blocks are another factor to test, when blocks are nuisance variables grouped to reduce unwanted variability, not variables of interest.
Why: Students confuse post-experiment confirmation with replication within the original experiment.
Why: Students think matching removes the need for randomization, but order effects can confound results.
Why: Students assume any study called an experiment can support causation, but only random assignment enables causal inference.