5.1.1 Check out the data

We have loaded into R a dataset called kobe that consists of every shooting attempt Kobe made during that game, including whether or not it went in (i.e., was the shot a “Hit” or a “Miss”). Here’s the first few rows:

5.1.2 Framing the hypotheses

We can translate the claim of the hot hand into a null hypothesis and an alternative hypothesis. For us to believe the “hot hand” claim, we first have to rule out the possibility that Kobe’s hit proportion is the same regardless of whether his previous shot went in or not. This possibility is the null hypothesis. The alternative hypothesis is that Kobe really had a hot hand and he made a greater proportion of hits after having already made a hit than after missing.

5.1.3 Summarize the data

Now that we’ve framed our hypotheses, let’s see whether our data suggest that we could reject the null hypothesis. We’ve already seen how to do this using a frequency table, like this:

kobe %>%
    group_by(prev_shot, shot) %>%
    summarize(n = n()) %>%
    mutate(p = n / sum(n))

## `summarise()` has grouped output by 'prev_shot'. You can override using the
## `.groups` argument.

## # A tibble: 4 × 4
## # Groups:   prev_shot [2]
##   prev_shot shot      n     p
##   <chr>     <chr> <int> <dbl>
## 1 prev_H    H        18 0.36 
## 2 prev_H    M        32 0.64 
## 3 prev_M    H        25 0.410
## 4 prev_M    M        36 0.590

Here, prev_shot refers to whether Kobe’s previous shot was a hit (“prev_H”) or a miss (“prev_M”). The final column p gives us the proportions of Hits (“H”) or Misses (“M”) following either a Previous Hit or Previous Miss.

We could use the proportions in the final column to calculate by hand the difference between the proportion of Kobe’s made shots (Hits) following either a hit (prev_H) or a miss (prev_M). But we can use R to do this work for us using a chunk like this one:

kobe %>%
    specify(shot ~ prev_shot, success = "H") %>%
    calculate(stat = "diff in props", order = c("prev_H", "prev_M"))

## Response: shot (factor)
## Explanatory: prev_shot (factor)
## # A tibble: 1 × 1
##      stat
##     <dbl>
## 1 -0.0498

The chunk works like this:

1. Tell R the name of the dataset we are working with (kobe).
2. We specify which variable is the explanatory variable (prev_shot) and which variable is the response variable (shot). We do this using the same squiggly thing we used for linear regression ([response variable] ~ [explanatory variable]). Because the response variable is a binary (i.e., two-level) categorical variable, we also need to tell R that a “success” is a Hit (abbreviated “H”).
3. Tell R to calculate a particular summary statistic. By saying stat = "diff in props" we are telling R that we want to calculate a difference in proportions. Because we already told R that our response variable is shot and a success is a Hit, R knows that we want to calculate a difference in the proportion of shots that were hits. By saying order = c("prev_H", "prev_M"), we tell R that we want that difference calculated in a specific order. In math shorthand, we could write it \(\hat{p}_{\text{Prev. H}} - \hat{p}_{\text{Prev. M}}\).

This may seem like a lot, but remember that computers needs to be told everything in detail. We can’t trust a computer to figure out anything on its own (yet?). Moreover, we will see that these lines of code can be “remixed” in various ways that will be very convenient for us as we go.

kobe %>%
    specify(shot ~ prev_shot, success = "M") %>%
    calculate(stat = "diff in props", order = c("prev_H", "prev_M"))

## Response: shot (factor)
## Explanatory: prev_shot (factor)
## # A tibble: 1 × 1
##     stat
##    <dbl>
## 1 0.0498

kobe %>%
    specify(shot ~ prev_shot, success = "H") %>%
    calculate(stat = "diff in props", order = c("prev_M", "prev_H"))

## Response: shot (factor)
## Explanatory: prev_shot (factor)
## # A tibble: 1 × 1
##     stat
##    <dbl>
## 1 0.0498

5.1.4 Simulating a possible dataset if the null were true

If the null hypothesis were true, whether Kobe does or does not make a shot should not depend on whether he did or did not make his previous shot. We can model what Kobe’s performance would look like if this null hypothesis were true. To do this, we treat Kobe’s shooting history as if it were random. If the null hypothesis were true, we should be able to randomly permute Kobe’s shooting history without changing the overall relationship between Kobe’s previous shots and his current shot. By “overall relationship”, we mean the difference in proportions that we just calculated. If the null hypothesis were true, the difference in proportions from our permuted data should not be too different from the difference in proportions in our actual data. On the other hand, if the null hypothesis were false, the difference in proportions in the permuted data should be very different from what we actually saw.

We can use R to permute Kobe’s shot history and thereby model how Kobe’s shots might have gone if the null hypothesis were true. To do this, we need to specify the relevant variables as well as the hypothesis. Let’s see that all at once, then unpack it:

kobe %>%
    specify(shot ~ prev_shot, success = "H") %>%
    hypothesize(null = "independence") %>%
    generate(reps = 1, type = "permute")

## Response: shot (factor)
## Explanatory: prev_shot (factor)
## Null Hypothesis: independence
## # A tibble: 111 × 3
## # Groups:   replicate [1]
##    shot  prev_shot replicate
##    <fct> <fct>         <int>
##  1 M     prev_H            1
##  2 M     prev_M            1
##  3 H     prev_M            1
##  4 M     prev_H            1
##  5 M     prev_H            1
##  6 M     prev_M            1
##  7 H     prev_M            1
##  8 M     prev_M            1
##  9 M     prev_H            1
## 10 M     prev_H            1
## # … with 101 more rows

The first and second lines are just like what we used already.

The third line is how we tell R what our null hypothesis is, namely, that the explanatory and response variables are independent (not associated).

Finally, the fourth line generates a simulated dataset by randomly permuting—that is, shuffling—the columns of the original data containing the explanatory and response variables. We’ll see the function of the reps = 1 setting momentarily.

kobe %>%
    specify(shot ~ prev_shot, success = "H") %>%
    hypothesize(null = "independence") %>%
    generate(reps = 1, type = "permute") %>%
    calculate(stat = "diff in props", order = c("prev_H", "prev_M"))

5.1.5 Simulating many possible datasets if the null were true

All we did was simulate one possible way Kobe’s shots could have turned out if the null hypothesis were true. There are many possible ways Kobe’s shots could have turned out, and each of these corresponds to a different random permutation of Kobe’s shot history. What we need to do is keep track of all the differences in proportions we get from each of those permutations. It’s a good thing that computers are good at repetitive tasks, because we can use the computer to repeat the random permutation process many times to simulated many possible datasets.

The next chunk of code is the same as before except we have changed reps in the generate line from 1 to 1000. This way, we can simulate 1000 different datasets, each of which is a different random permutation of Kobe’s shot history. We tell R to remember these all under the name kobe_null_distribution because this represents the distribution of differences that we would expect to see if the null hypothesis were true.

kobe_null_distribution <- kobe %>%
    specify(shot ~ prev_shot, success = "H") %>%
    hypothesize(null = "independence") %>%
    generate(reps = 1000, type = "permute") %>%
    calculate(stat = "diff in props", order = c("prev_H", "prev_M"))

The first few rows of the result look like this:

The “replicate” column is a label for each simulated dataset, and the new “stat” column is the difference \(\hat{p}_{\text{Prev. H}} - \hat{p}_{\text{Prev. M}}\) for each simulated dataset.

We can now make a histogram to examine the distribution of differences in proportions that result when the null hypothesis is true:

kobe_null_distribution %>%
    visualize()

5.1.6 Find the \(p\) value

Remember that the \(p\) value is the proportion of simulated datasets which are at least as extreme as our actual data. In this case, that means the proportion of simulated datasets with a difference in proportions that is bigger than what we observed. Let’s tell R to remember that difference by running the chunk of code below:

kobe_obs_diff <- kobe %>%
    specify(shot ~ prev_shot, success = "H") %>%
    calculate(stat = "diff in props", order = c("prev_H", "prev_M"))

Now, let’s see how many simulated datasets were bigger than this. In the following line of code, we have to tell R that the observed “statistic” (obs_stat) is the difference we just told it to remember (kobe_obs_diff) as well as the fact that we are interested in how many simulations produced results that were “greater” than what was observed:

kobe_null_distribution %>%
    get_p_value(obs_stat = kobe_obs_diff, direction = "greater")

## # A tibble: 1 × 1
##   p_value
##     <dbl>
## 1   0.757

Finally, it will help to visualize where the observed difference falls relative to the distribution of differences from our simulated data (note that this uses shade_p_value rather than get_p_value):

kobe_null_distribution %>%
    visualize() +
    shade_p_value(obs_stat = kobe_obs_diff, direction = "greater")

The red line is the value that was actually observed and the parts of the histogram that are shaded pink represent the simulated datasets that were “more extreme” than what was observed.

5.1.7 Form a conclusion

Remember that we were testing the null hypothesis that Kobe did not have a hot hand. We only reject the null hypothesis if our data would be very unlikely if the null hypothesis were true, i.e., if the \(p\) value is low. Assume we adopt a significance level of 0.05, so we would reject the null hypothesis if the \(p\) value were less than this level.

5.2.1 Check out the data

The relevant data are stored in R under the name asc_choice. Here’s what the first few rows look like.

The response variable is Choice, which is either “Consistent” (if the participant’s choice was not affected by the decoy) or “Inconsistent” (if the participant’s choice was affected by the decoy).

The explanatory variable is Group, which is either “ASC” (for “Autism Spectrum Condition”) or “NT” (for “Neuro-Typical” control).

5.2.2 Framing the hypotheses

5.2.3 Summarize the data

___ %>%
    specify(___ ~ ___, success = "___") %>%
    calculate(stat = "___", order = c("___", "___"))

5.2.4 Model the null hypothesis

If the null hypothesis were true, a participant’s choice shouldn’t depend on which group they are in. In the Kobe example, we modeled the null hypothesis by randomly permuting Kobe’s shot history, since that was the explanatory variable. In this research scenario, group is the explanatory variable. So by randomly shuffling participants between groups, we can simulate how the data would look if the null hypothesis were true.

asc_null_distribution <- ___ %>%
    specify(___ ~ ___, success = "___") %>%
    hypothesize(null = "___") %>%
    generate(reps = ___, type = "permute") %>%
    calculate(stat = "___", order = c("___", "___"))

asc_null_distribution %>%
    visualize()

5.2.5 Find the \(p\) value

To find the \(p\) value, we first need to get the observed difference in proportions and tell R to remember it with the label asc_obs_diff.

asc_null_distribution <- ___ %>%
    specify(___ ~ ___, success = "___") %>%
    hypothesize(null = "___") %>%
    generate(reps = ___, type = "permute") %>%
    calculate(stat = "___", order = c("___", "___"))

asc_obs_diff <- ___ %>%
    specify(___ ~ ___, success = "___") %>%
    calculate(stat = "___", order = c("___", "___"))

asc_null_distribution %>%
    get_p_value(obs_stat = asc_obs_diff, direction = "___")

5.2.6 Form a conclusion

Finally, we are in a position to visualize where the observed difference in proportions falls relative to the differences that would be expected if the null hypothesis were true. This will enable us to form a conclusion about what these data tell us about our research question.

asc_null_distribution <- ___ %>%
    specify(___ ~ ___, success = "___") %>%
    hypothesize(null = "___") %>%
    generate(reps = ___, type = "permute") %>%
    calculate(stat = "___", order = c("___", "___"))

asc_obs_diff <- ___ %>%
    specify(___ ~ ___, success = "___") %>%
    calculate(stat = "___", order = c("___", "___"))

asc_null_distribution %>%
    visualize() +
    shade_p_value(obs_stat = asc_obs_diff, direction = "___")