Topic 7.1: Introducing Statistics: Should I Worry About Error?
The Logic of Significance Testing for a Difference in Means
Video Follow-Along Worksheet
↻ Already did this one? It stays open even after it’s checked off — just change any answer and click ✓ Check Answers again (no need to Reset). ✨ Grade with AI only ever raises your grade. Make sure you’re signed in so your changes save.
Learning Objectives:
Identify the evidence in a study comparing two group means.
Describe the two possible explanations for an observed difference in sample means.
Use a rerandomization simulation to estimate how likely an observed result is by chance alone.
Decide whether the data provide convincing evidence for a claim about a difference in means.
Essential Knowledge:
If a treatment makes no difference, the expected difference in sample means is 0.
An observed difference can be explained by chance variation in random assignment or by a real treatment effect.
A simulation under the "no difference" assumption estimates how unusual the observed statistic would be by chance alone.
If the observed result is very unlikely under chance alone, the chance explanation can be ruled out and the treatment explanation is supported.
Random assignment allows a causal conclusion about students like the ones in the study.
Key Vocabulary
Claim
A statement being evaluated with data, such as whether one wording causes more spending than another.
Evidence
The observed result from the study that supports or fails to support the claim.
Difference in Means
The difference between two sample averages, used here to compare spending in the two groups.
Chance Variation
Natural random differences that can appear just because subjects were randomly assigned to groups.
Random Assignment
Assigning subjects to treatments by chance so the groups are comparable.
Rerandomization
Shuffling subjects into groups again and again under the assumption that the treatment makes no difference.
Simulation
A repeated random process used to estimate how likely a result would be if chance alone were operating.
Convincing Evidence
Evidence strong enough that the chance-alone explanation is not very believable.
Video 1: Bonus vs. Rebate and the Logic of Significance Testing
[0:00 – 7:43]
1.[0:00]
This lesson introduces the logic of significance testing for a difference in .
2.[0:09]
One goal is to identify evidence for a .
3.[0:13]
Another goal is to decide whether that evidence is .
Context — Bonus vs. Rebate[0:20 – 0:56]
Volunteer college students were given $50 with no strings attached. Twenty-five of the 47 students were randomly assigned to hear that the money was "bonus income," and the other 22 were told it was a "tuition rebate."
Group
Students
Mean Amount Spent
Bonus Income
25
$22.04
Tuition Rebate
22
$9.55
4.[0:20]
The students were randomly to the two wording groups.
5.[0:24]
One group heard the money called a , while the other group heard it called a tuition .
6.[1:41]
If the wording made no difference in spending, the expected difference in means would be .
7.[2:03]
In the study, the observed difference in means was $, which is greater than .
Reasoning Structure:[1:41 – 3:15]
If the treatment makes no difference, we expect a difference in means of 0.
The observed statistic provides evidence for the claim if it goes in the predicted direction.
To decide if the evidence is convincing, ask how likely that observed result would be by chance alone.
8.[2:26 – 2:30]
One explanation is that the wording made no and the observed result happened because of chance .
9.[2:52]
The other explanation is that the bonus wording caused students to spend on average.
10.[3:00]
To be convinced that explanation 2 is correct, we need to know the of getting a difference in means of 12.49 or more by chance alone.
11.[3:33]
To estimate that probability, the lesson uses a .
12.[3:47]
In this simulation, we assume the wording makes no , so subjects are expected to spend the amount no matter which group they are in.
Simulation Process:[3:55 – 5:30]
Rerandomize the same subjects into the two groups.
Calculate the mean amount spent for each group.
Find the difference in means.
Repeat many times to see what differences happen by chance alone.
13.[3:55]
Each simulation trial reassigns the subjects to groups, calculates the group means, and records the in means.
14.[4:33]
In one rerandomization, the difference in means was negative $.
15.[4:57]
Another rerandomization produced a difference in means of $.
16.[5:18]
A later rerandomization produced a difference in means of negative $.
17.[5:43]
The full dotplot shown in the lesson came from simulation trials.
18.[6:12]
About out of 1000 trials were 12.49 or greater, giving an approximate probability of .
19.[6:48]
Because that probability is very , we can rule out explanation .
20.[7:43]
The data provide convincing that the bonus wording causes students like the ones in this study to spend money on average.
Key Takeaways
[Summary]
Evidence for a claim comes from the observed statistic, such as the difference in sample means.
If there is no treatment effect, the expected difference in means is 0.
A rerandomization simulation estimates how likely the observed statistic would be by chance alone.
If that probability is very small, the chance explanation is not convincing and the treatment explanation is supported.
Reflection Questions
[AI-Graded]
R1.Using the bonus vs. rebate study, explain what the evidence for the claim is and describe the two possible explanations for that evidence.
R2.Explain how the rerandomization simulation works in this lesson and how the approximate probability of 0.013 helps decide whether the evidence is convincing.
Exit Ticket
In the bonus vs. rebate experiment, the observed difference in mean amount spent was $12.49, and a rerandomization simulation produced 13 results of $12.49 or greater out of 1000 trials.
(a) State the evidence for the claim that wording matters.
(b) Describe the two possible explanations for the observed difference.
(c) Use the simulation result to decide whether the data provide convincing evidence, and explain the conclusion in context.
Class Responses (Esc to close)
Click a "Class" button next to a question to see responses.