Scenario Based Exam Questions for Intro Statistics

A scenario based question is one that includes a brief description of a dataset or model, along with some information like a plot, set of summary statistics, or computer output. From this information, the student needs to answer several questions about the data, such as 'what to these parameters represent?' or 'would the correlation be stronger or weaker without the outlier?'. It's my preferred way of asking questions on exams and assignments, as it more closely mimics the sort of problems that someone would actually encounter in a non-classroom setting.

Below are four examples with post-mortem commentary, as well as download links to all 94 such questions I've made so far that are worth keeping and drawing from.




The difficulty is in compartmentalizing the questions in such a way that there is a clear way to mark them that isn't too open to interpretation. These problems need to be consistently gradable by senior undergraduate teaching assistants.


Example 1:

Consider the following regression output of the amount of flowers in an area in response to the number of beehives in the area. The explanatory variable is between 0 and 6 beehives.

A) What does the intercept describe in the situation in real-world terms? (2 pts)

B) What does the slope describe in the situation in real-world terms? (2 pts)

C) Predict the number of flowers in an area with 2 beehives. (2 pts)

D) Is the prediction in Part C reasonable in real-world terms? Why or why not? (2 pts)

E) Which prediction would have more uncertainty, one for a single area with 2 beehives, or the average of many areas with 2 beehives each? (1 pts)

Commentary: 

I've made the explanatory variable 'number of beehives' instead of 'thousands of bees', or something similar because I've found that in the past that students get confused about inserting '2' or '2000' into a regression equation. 'Amount of flowers' is ambiguous, as 'thousands of flowers' would cause similar trouble and 'number of flowers' would cause confusion surrounding negative numbers and rounding.

The first two parts are to prime the student into seeing what each variable and parameter in the model is doing. In class, I would have told them that 'in real-world terms' should include at least some mention of the question context, in this case beehives and flowers.

Part C is the only calculation. This particular question is light on calculation, but it's not unusual for my exam questions. The emphasis is on understanding the numbers the computer gives out, not on reproducing them. 

Part D refers to interpolation vs extrapolation. Part E is about the difference between a prediction band and a confidence band, or equally about the sources of uncertainty (model uncertainty vs. natural variation between observations).

Example 2: We have a sample of vet records of 50 purebred dogs and  mutts' lifespans and are interested in comparing the mean life expectancies. Assume all factors other than purebred vs. Mutt are irrelevant.  (7 pts total)

   Purebred    Mutt    
 Life Expectancy (yrs)     10.6    14.3           
s    6.4    1.8           
n    16    34        

a) Describe the test to be done to test the given hypothesis.  (2 pts, 0.5 pts per part):

b) Get the t-score, assume you cannot pool standard deviation. (3 pts) (show work)

c) Get the t-critical at alpha = 0.05, assume you cannot pool standard deviation.  (1 pt)

d) State your conclusion in terms of dogs and lifespans. (1 pt)


Commentary:
Part A is just asking what should be done, which is again meant to prime students and reinforce a process of considering such things before jumping into mathematics. The '0.5 pts per part' is meant to indicate that at least four facts should be stated (The answer was any four of the following: this is a t-test, between two groups, of their means, as a two-tailed test, independent groups, and we cannot assume equal variance). Each part is written to avoid ambiguities if possible.

Another thing to be avoided when possible is the potential for 'cascading failures'. These are questions were an incorrect answer in one part dooms the student to give an incorrect answer in another part. Here, the answer for part C does not depend on the answer for part B, even though they pertain to the same hypothesis test. If student gets their answer in part A wrong, they likely would have employed the wrong statistical test and the wrong formula regardless. In a worst case scenario, having part A there has not made the subsequent parts harder; in most cases it should make them easier.

Example 3:
Last year, Canada helped develop a vaccine to Ebola, to be tested in Sierra Leone. They know it will work, but they don't know which dosage will be the most effective. They set up a randomized trial and split the treatments into 3 groups of 1000 people each. The treatments were low, medium, and high dose. For each group, they measured the proportion of people that ended up infected after a year.


a) What is the explanatory/independent variable? The response/dependent variable? (1 pt)


b) Draw an experiment diagram outlining this experiment (2 pts)


c) What are some lurking variables that could affect the outcome? List at least 2. (2 pts)


d) What is the name of this design? (1 pt)

Commentary: Part A again is there to reinforce the process or identifying the problem and solution before executing it. Note that due to differences in terminology between fields, the x variable is referred to as both explanatory AND independent.

Example 4:
A baby can be born either pre-term, at the normal time, or late term. Every baby fits exactly one of these categories (no overlap, no missing values). Without knowing anything else about a given baby, the following probabilities apply:

Pr(pre-term) = 0.20        Pr(normal) = 0.70        Pr(late) = 0.10

a) Find Pr(pre-term OR normal)


b) Find Pr(pre-term AND normal)


c) Find Pr(late | not pre-term)


d) Assume each baby is independent. Find the probability that three particular, unrelated babies are all born at the normal time?

Commentary:The ordering of the question parts here was a tough decision, and in retrospect I messed up. Ideally the question parts progress from easy to hard, but part C requires multiple steps that few people were able to parse. A recurring misconception is that 'conditional on' means the same thing as 'and'; I haven't been able to fix this issue before exam time yet. There was a Part E that asked about independence and common genetic factors, but it has been removed because during the exam, it was found to be badly ambiguous.

The links below are downloads of all 43 final exam questions (mostly regression, chi-squared tests, and ANOVA), 27 midterm 1 questions (mostly descriptive statistics, sampling, and experimental design), and 24 midterm 2 questions (mostly t-tests and related hypothesis tests).

Midterm 1 PDF DOCX

Midterm 2 PDF DOCX

Final  PDF DOCX

These are also found in the ASA Statistical Commons