Exercises: Designing Studies — Sampling and Experiments

These exercises progress from concept checks to scenario analysis and critical evaluation. Estimated completion time: 2.5 hours.

Difficulty Guide: - ⭐ Foundational (5-10 min each) - ⭐⭐ Intermediate (10-20 min each) - ⭐⭐⭐ Challenging (20-40 min each) - ⭐⭐⭐⭐ Advanced/Research (40+ min each)

Part A: Conceptual Understanding ⭐

A.1. In your own words, explain the difference between an observational study and an experiment. Give one example of each from everyday life.

A.2. What is a confounding variable? Use the ice cream and drowning example to explain the concept to someone who has never taken a statistics course.

A.3. Why is it that a biased sample of 2.4 million can give worse results than a well-designed sample of 50,000? This seems counterintuitive — explain it.

A.4. Explain the difference between random sampling and random assignment. What does each one protect against?

A.5. What is the placebo effect? Why do experiments need placebo groups rather than simply comparing treatment to no treatment?

A.6. Define selection bias, response bias, nonresponse bias, and survivorship bias. Give a fresh example of each (not from the chapter).

A.7. Why can't we randomly assign people to smoke cigarettes for 20 years to study the effect of smoking on lung cancer? If we can't run that experiment, how did scientists establish that smoking causes cancer?

Part B: Identifying Study Types ⭐⭐

For each scenario, identify: (a) whether it is an observational study or an experiment, (b) the explanatory variable and response variable, (c) one potential confounding variable, and (d) whether the study can support causal conclusions.

B.1. A researcher at a university notices that students who attend office hours tend to get higher grades. She records attendance at office hours and final exam scores for 200 students.

B.2. A pharmaceutical company randomly assigns 500 patients with high blood pressure to receive either a new drug or a placebo. Blood pressure is measured after 12 weeks.

B.3. A school district implements a new reading program in 10 schools and compares reading scores to 10 schools that continue with the old program. The schools were not randomly selected — the new program was given to schools with the lowest test scores.

B.4. StreamVibe randomly shows half of its users a "Continue Watching" banner at the top of the homepage and the other half the standard layout. They measure average session length over two weeks.

B.5. A news article reports that people who own pets have lower blood pressure than people who don't. The data comes from a health survey of 10,000 adults.

B.6. Professor Washington examines arrest records from before and after a city implemented a predictive policing algorithm. He compares arrest rates by neighborhood demographics.

Part C: Sampling Methods ⭐⭐

For each scenario, identify the sampling method used and evaluate whether the sample is likely to be representative of the target population.

C.1. A market research company wants to survey adult residents of a city. They use the city's voter registration database, assign each registered voter a number, and use a random number generator to select 800 people.

C.2. A university wants to assess student satisfaction. They email a survey link to all 25,000 students. 1,200 students complete the survey.

C.3. A polling firm wants to survey Americans about healthcare policy. They divide the population into four age groups (18-29, 30-44, 45-64, 65+), then randomly sample 500 people from each group.

C.4. A quality control inspector checks every 50th item coming off an assembly line.

C.5. An education researcher wants to study math achievement across a state. She randomly selects 30 school districts, then tests every 8th grader in those districts.

C.6. A food delivery app wants customer feedback. They add a pop-up survey that appears after every 10th order. Is this systematic sampling? Why or why not?

C.7. A journalist covering a political rally interviews attendees to gauge support for a candidate. She claims "most Americans support this candidate." What's wrong with this conclusion?

Part D: Bias and Confounding ⭐⭐⭐

D.1. A study finds that children who play video games for more than 3 hours a day have lower grades than children who play for less than 1 hour a day.

(a) Is this observational or experimental? (b) Name two confounding variables that could explain the association. (c) Design an experiment that could establish a causal relationship. Is your experiment ethical? Why or why not? (d) If we can't run an ethical experiment, what kind of evidence would make the causal claim more convincing?

D.2. An online fitness company claims: "Our app users lose an average of 12 pounds in 3 months!" They support this claim with data from 500 users who completed a 3-month program.

(a) Identify at least two sources of bias in this claim. (b) What group is missing from this analysis? (c) How would you redesign this study to produce more reliable evidence?

D.3. A hospital administrator notices that patients admitted on weekends have higher mortality rates than patients admitted on weekdays. She concludes that weekend hospital staffing is inadequate.

(a) Name two confounding variables that could explain this pattern. (b) How could you investigate whether the administrator's conclusion is correct?

D.4. A social media platform reports that users who see more diverse content in their feeds report higher satisfaction. The data comes from analyzing user behavior logs and satisfaction surveys.

(a) Is this observational or experimental? (b) What confounding variables might explain the association? (c) How could the platform use A/B testing to get a stronger answer? (d) What ethical considerations arise from manipulating users' feeds?

D.5. Consider this claim: "Students who use laptops in class have lower GPAs than students who take notes by hand."

(a) Identify three confounding variables. (b) A researcher designs an experiment where she randomly assigns half the class to use laptops and half to use paper. What are the strengths and limitations of this design? (c) How does this study relate to the concept of blinding? Is blinding possible here?

Part E: Critical Evaluation ⭐⭐⭐

E.1. Read the following (simplified) study summary and apply the Causal Claims Checklist from Section 4.8:

"Researchers surveyed 5,000 adults and found that those who reported eating organic food at least three times per week had 25% lower cancer rates over a five-year follow-up period. They concluded that eating organic food reduces cancer risk."

Evaluate this claim by addressing all eight checklist items.

E.2. A tech company blog post states: "We ran an A/B test and found that users shown personalized recommendations spent 15% more time on our platform."

(a) What information is missing from this claim that you would need to evaluate it? (b) Even if the A/B test was well-designed, what ethical questions does this raise? (c) Can you think of a scenario where a well-designed A/B test still produces misleading results?

E.3. Design a study to answer the following question: "Does listening to music while studying improve exam performance?"

(a) Describe both an observational study and an experiment that could address this question. (b) For the experiment, specify: the treatment, control, randomization procedure, response variable, and how you would handle blinding. (c) What confounding variables would your experiment control for? What confounders might remain? (d) How would you ensure your sample is representative of the population you want to generalize to?

E.4. A news headline reads: "Living near parks adds 5 years to your life, study finds."

(a) Without reading the study, what's your initial reaction? What type of study do you suspect this is? (b) Name three confounding variables that could explain this association. (c) Rewrite the headline to be more accurate based on what you know about observational studies. (d) What would it take for you to be convinced that parks actually cause longer lives?

Part M: Interleaved Practice (Mixed review from Chapters 1-3) ⭐⭐

These problems intentionally mix concepts from previous chapters with new material. Interleaving strengthens your ability to recognize which concept applies in a given situation.

M.1. (Ch.1 + Ch.4) In Chapter 1, we discussed the four pillars of statistical investigation. Consider a pharmaceutical company that wants to test whether a new vaccine prevents the flu. For each pillar, describe what a well-designed study would look like — and identify one way each pillar could go wrong.

M.2. (Ch.2 + Ch.4) A researcher collects the following variables in a study of college students' mental health: age (years), GPA (0-4.0), gender (male/female/non-binary), number of close friends, anxiety level (1-10 scale), and whether they received counseling (yes/no).

(a) Classify each variable as categorical (nominal/ordinal) or numerical (discrete/continuous). (b) This is an observational study. If the researcher finds that students who received counseling report lower anxiety, can she conclude counseling reduces anxiety? Why or why not? (c) Name two confounding variables. (d) Design an experiment to test whether counseling reduces anxiety. What ethical issues might arise?

M.3. (Ch.3 + Ch.4) You load a dataset of 10,000 customer reviews into pandas and run .describe(). You notice that the average rating is 4.2 out of 5.

(a) Is 4.2 a parameter or a statistic? (b) What type of sampling likely produced this data? (Hint: who writes reviews?) (c) Name two sources of bias that might make this average unrepresentative of all customers' opinions. (d) If the company concludes "Our customers are highly satisfied," what's wrong with this conclusion?

M.4. (Ch.1 + Ch.2 + Ch.4) A headline reads: "Longitudinal study of 20,000 adults finds that people who volunteer regularly have 22% lower mortality over 10 years."

(a) Is this an observational study or an experiment? How do you know? (b) What does "longitudinal" mean, and why is it important here? (Ch.2) (c) Name two confounding variables. (d) A friend says, "So volunteering makes you live longer." How would you respond using what you've learned about confounding and causation?

M.5. (Ch.2 + Ch.4) Sam Okafor is trying to determine if Daria's new shooting coach made a difference. He has data on all of Daria's three-point attempts: make/miss (categorical — what subtype?), game number (numerical — what subtype?), the opponent's defensive ranking (numerical — continuous), and whether the attempt was in the first or second half of the season (categorical — what subtype?).

(a) Classify each variable. (b) Is Sam's analysis observational or experimental? Why? (c) Name the most important confounding variable in Sam's analysis. (d) What additional data would help Sam make a stronger case?

M.6. (Ch.3 + Ch.4) You are given a CSV file containing survey data from a convenience sample of 200 college students. You load it into pandas and plan your analysis.

(a) Write the Python code to load the file and check its dimensions. (b) Before running any statistical analysis, what is the most important question you should ask about this dataset? (Hint: think about this chapter.) (c) If you find that students who sleep more than 7 hours per night have higher GPAs, what can you conclude? What can you not conclude? (d) If you wanted to generalize your results to "all college students in the U.S.," what would need to be true about the sample?