4
Lesson 4.5
Observational Studies and
Experiments
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Learning Targets
After this lesson you should be able to:
Explain the concept of confounding and how it limits
the ability to make cause-and-effect conclusions.
Identify the treatments in an experiment.
Explain the purpose of a control group in an
experiment.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
In Lesson 4.1 , you learned how to distinguish observational studies from
experiments.
??
The goal of an observational study can be
to describe some group or situation,
?Y
to compare groups, or
to examine relationships between variables.
X→
Unfortunately, with observational studies it is very
difficult to identify cause-and-effect relationships.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
Confounding
In Lesson App 4.1 , an observational
study revealed that teenagers who eat
with their families at least 5 times a
week are more likely to get good
grades in school.
Unfortunately, it’s not appropriate to
conclude that increasing the number
of meals you eat with your family will
cause your grades to improve.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
Can you think of a reason why teens who eat often
with their families are more likely to get good grades?
As shown in the following table, there are many differences between teens who
eat often with their families and teens who don’t. Any of these differences could
be the cause of the difference in grades.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
Perhaps the teens who eat more meals with their families are less likely to have
parttime jobs or participate in after-school activities, allowing more time for
studying—and for dinner with their families.
In this case, we wouldn’t know which variable was causing the good grades—the
number of meals with their family or the amount of available study time.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
Recall that a response variable measures an outcome of a study.
An explanatory variable may help predict or explain changes in a response variable.
When it is impossible to tell if the explanatory variable or some other variable is
causing a change in a response variable, we say the two variables are confounded.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
DEFINITION Confounding
Confounding occurs when two variables are associated
in such a way that their effects on a response variable
cannot be distinguished from each other.
Observational studies cannot definitively show a cause-and-effect
relationship between an explanatory variable and a response
variable because of confounding. You learned a similar lesson in
Chapter 3: association does not imply causation.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Example: Can ADHD be linked to mothers who smoke?
In a study of more than 4700 children, researchers from Cincinnati Children’s
Hospital Medical Center found that those children whose mothers smoked during
pregnancy were more than twice as likely to develop ADHD as children whose
mothers had not smoked. Based on this study, is it reasonable to conclude that a
mother’s smoking during pregnancy causes an increase in the risk of ADHD in her
children? Explain your reasoning.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Example: Can ADHD be linked to mothers who smoke?
In a study of more than 4700 children, researchers from Cincinnati Children’s
Hospital Medical Center found that those children whose mothers smoked during
pregnancy were more than twice as likely to develop ADHD as children whose
mothers had not smoked. Based on this study, is it reasonable to conclude that a
mother’s smoking during pregnancy causes an increase in the risk of ADHD in her
children? Explain your reasoning.
SOLUTION:
No. It is possible that women who smoked during pregnancy had less
healthy diets and that eating poorly during pregnancy increases the
risk of ADHD in kids.
If both of these are true, then we would see a relationship between
smoking and ADHD even if smoking has no effect on ADHD.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
The easiest way to identify confounding in an observational study is to think
about other variables that are associated with the explanatory variable that might
cause a change in the response variable.
?
In the smoking and ADHD study, there were many
potential differences between the group of mothers
who smoked during pregnancy and the group of
mothers who didn’t.
?
Likewise, there are many potential differences between
the group of teens who eat often with their families and
the group of teens who don’t. Any of these differences
could be the cause of the change in the response variable.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
The Language of Experiments
Conducting a well-designed experiment is one of the best ways to prevent
confounding. Recall that an experiment deliberately imposes some treatment on
individuals to measure their responses. Each of these individuals is called an
experimental unit, or subject.
DEFINITION Treatment, Experimental unit, Subject
A treatment is a specific condition applied to the individuals in an
experiment. An experimental unit is the object to which a treatment is
randomly assigned. When the units are human beings, they often are
called subjects.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Example: Do you want another brownie?
Researchers invited 186 university students to take part in
an experiment. While the students were watching a video,
they were given brownies to snack on. Unknown to the
students, the size and number of brownies differed from
student to student. There were three different sizes (8g, 16g,
or 32g) and four different quantities (1, 2, 4, or 8 brownies).
Researchers measured the total weight of brownies consumed
and whether or not each student finished the brownies.
(a) List the explanatory variable(s) and the number of possible values for each variable.
(b) If the researchers used every possible combination to form the treatments, how many
treatments were included in the experiment?
(c) List two of the treatments.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Example: Do you want another brownie?
Researchers invited 186 university students to take part in
an experiment. While the students were watching a video,
they were given brownies to snack on. Unknown to the
students, the size and number of brownies differed from
student to student. There were three different sizes (8g, 16g,
or 32g) and four different quantities (1, 2, 4, or 8 brownies).
Researchers measured the total weight of brownies consumed
and whether or not each student finished the brownies.
(a) List the explanatory variable(s) and the number of possible values for each variable.
SOLUTION:
Size of brownie (3 possible values) and quantity of brownies
(4 possible values).
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Example: Do you want another brownie?
Researchers invited 186 university students to take part in
an experiment. While the students were watching a video,
they were given brownies to snack on. Unknown to the
students, the size and number of brownies differed from
student to student. There were three different sizes (8g, 16g,
or 32g) and four different quantities (1, 2, 4, or 8 brownies).
Researchers measured the total weight of brownies consumed
and whether or not each student finished the brownies.
(b) If the researchers used every possible combination to form the treatments, how many
treatments were included in the experiment?
SOLUTION:
3 × 4 = 12 treatments
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Example: Do you want another brownie?
Researchers invited 186 university students to take part in
an experiment. While the students were watching a video,
they were given brownies to snack on. Unknown to the
students, the size and number of brownies differed from
student to student. There were three different sizes (8g, 16g,
or 32g) and four different quantities (1, 2, 4, or 8 brownies).
Researchers measured the total weight of brownies consumed
and whether or not each student finished the brownies.
(c) List two of the treatments.
SOLUTION:
One 8-gram brownie; eight 32-gram brownies
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
To learn what it means to be a “well-designed” experiment, let’s start with a poorly
designed experiment.
Mr. Luckow’s class wants to investigate if caffeine affects pulse rates. One of his
students proposes the following plan:
1. Have each student measure his or her initial pulse rate.
2. Give each student 12 ounces of cola.
3. Wait for 15 minutes.
4. Have each student measure his or her final pulse rate.
5. Compare initial and final pulse rates.
Can you identify any problems with the proposed experiment?
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
If the cola has sugar as well as caffeine, we wouldn’t be able to tell if the
increase in pulse rates was due to the presence of caffeine or the presence
of sugar. (Can you say confounding?)
Sugar? Caffeine?
Likewise, there could be other events between the initial and final pulse
rate measurements that could be the cause of an increase in pulse rates,
such as a fire drill or an exciting statistics lesson.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
The solution for these problems is to have two groups in the experiment:
one group that receives the caffeine and a control group that doesn’t
receive the caffeine.
DEFINITION Control group
In an experiment, a control group is a group used to provide a
baseline for comparing the effects of other treatments. Depending on
the purpose of the experiment, a control group may be given an
inactive treatment, an active treatment, or not treatment at all.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
In all other ways, these two groups should be treated exactly the same so the only
difference is the caffeine. That way, if there is convincing evidence of a difference in the
average increase in pulse rates, we can conclude it was caused by the caffeine.
This means that one group could get regular cola with caffeine,
while the control group gets caffeine-free cola. Both groups
Caffeine
Regular
Free
would get the same amount of sugar, so sugar consumption
would no longer be confounded with caffeine intake.
Likewise, both groups would experience the same events during the experiment so what
happens during the experiment won’t be confounded with caffeine intake either.
Using a design that compares two or more treatments
is the first step in designing a good experiment.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Example: What devices get customers to conserve electricity?
Many utility companies have introduced programs to encourage energy
conservation among their customers. An electric company considers
placing small digital displays in households to show current electricity use
along with a projected monthly cost. Will the displays reduce electricity
use? One cheaper approach is to give customers a chart and information
about monitoring their electricity use from their outside meter. Would this
method work almost as well? The company decides to conduct an
experiment to compare these two approaches (display, chart) with a
control group of customers who receive information about energy
consumption but no help in monitoring electricity use. Explain why it is
necessary to include the control group.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Example: What devices get customers to conserve electricity?
SOLUTION:
A control group would show how
much electricity customers tend
to use with no help monitoring
electricity usage. This would serve
as a baseline to determine how
much electricity is saved by using
the display or chart.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
Not all experiments require a control group.
In the brownie experiment, researchers were interested in the relationships
between size of brownie, number of brownies, and amount consumed.
Having a control group that received no brownies wouldn’t help—we know
this group would eat no brownies.
?
While not all experiments have a control group, all good experiments
involve comparison of two or more treatments.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
In the caffeine experiment, we used comparison to help prevent confounding. But even
when there is comparison, confounding is still possible.
If each student in Mr. Luckow’s class knows what type of soda he or she is receiving, the
expectations of the two groups will be different.
Caffeine
Regular
Free
The expectation of an increase in pulse rate may increase a subject’s pulse rate, apart from
the caffeine itself. This is an example of the placebo effect.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
DEFINITION Placebo effect
The placebo effect describes the fact that some subjects in an experiment
will respond favorably to any treatment, even an inactive treatment.
To make sure the expectations of all subjects are the same in Mr. Luckow’s caffeine
experiment, we give one group cola with caffeine and the other group a
placebo—a cola that looks and tastes identical, but without caffeine.
DEFINITION Placebo
A placebo is a treatment that has no active ingredient, but is otherwise
like other treatments.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
Placebos are often “sugar pills,” though they can also be any other type of
substance as long as it looks, tastes, smells, and feels the same as the active
treatment. Placebos can even be a procedure such as placebo surgery!
Placebo Treatment
A placebo isn’t that useful if a subject knows he or she is getting a placebo. It is
important that the placebo look, feel, and taste just like the other treatments, so
the subjects will not know which treatment they are receiving.
Otherwise, the expectations of the subjects receiving the placebo will differ from
the expectations of other subjects, creating confounding between the explanatory
variable and the expectations of the subjects.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
If neither the students in the caffeine experiment nor the people running
the experiment know who is getting which type of cola, the experiment is
double-blind. Other experiments are single-blind.
DEFINITION Double-blind, Single-blind
In a double-blind experiment, neither the subjects nor those who
interact with them and measure the response variable know which
treatment a subject is receiving.
In a single-blind experiment, either the subjects or the people who
interact with them and measure the response variable don’t know
which treatment a subject is receiving.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
In the electricity example, were the subjects blind?
No. They knew what treatment they were assigned.
The experiment can still be single-blind, however, if the person recording
electricity usage doesn’t know which household received which treatment.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Observational Studies and Experiments
THINK ABOUT IT .
Do placebos work?
One study found that 42% of balding men maintained or increased the amount of hair on
their heads when they took a placebo.
In another study, researchers zapped the wrists of 24 test subjects with a painful jolt of
electricity. Then, they rubbed a cream with no active medicine on subjects’ wrists and told
them the cream should help soothe the pain. When researchers shocked them again, 8
subjects said they experienced significantly less pain.
With ailments like depression, some experts think that the placebo effect accounts for
about three-quarters of the effect of the most widely used drugs. Others disagree. In any
case, “placebos work” is a good place to start when you think about planning medical
experiments.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
LESSON APP 4.5
Does regularly taking aspirin help protect people against heart
attacks? Does regularly taking beta-carotene help prevent
cancer? The Physicians’ Health Study I was a medical
experiment that helped answer these questions. The subjects
in this experiment were 21,996 male physicians. Each of the
physicians was assigned to take an aspirin or an aspirin-like
placebo. Likewise, each of the physicians was assigned to take
beta-carotene or a beta-carotene-like placebo. The physicians
were followed for several years and researchers recorded if
each of the physicians had a heart attack or developed cancer.
1. Why was it necessary to perform an experiment rather than simply asking the
doctors whether or not they take aspirin and beta-carotene regularly?
2. Identify the four treatments in this experiment.
3. Explain the purpose of the control group in this experiment.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
Learning Targets
What did you learn?
Explain the concept of confounding and how it limits
the ability to make cause-and-effect conclusions.
Identify the treatments in an experiment.
Explain the purpose of a control group in an
experiment.
Starnes/Tabor/Wilcox, Statistics and Probability with Applications, 4e © 2021 BFW Publishers
