Greg Foletta - Bits and Blobs

A melange of networking, data science and programming.

Where's the Bias: In the Die, or in Me?

It’s the classic mistake: trying to run before you can walk. And despite knowing my limitations when it comes to Bayesian modelling, I’ve still been making that mistake, trying to tackle problems that are outside of my capability. I needed something relatively simple and bounded, and while playing Yahtzee with my son I had a question that (I thought) fit these parameters: could I tell if a die is biased?

As often happens, peel back the first layer of an apparent simple question and you will find a…

LLM Disclosure

All commentary and code in this post was written by myself. An LLM was used to help me generate the Stan models for each of the scenarios, so please consider this post not as an expert lecturing to you ia innate knowledge, but rather as a student trying to understand and learn about these models by explaining how they work to someone else.

As a result, a significant amount of LLM time was used asking questions, clarifying, and trying to educate myself on the model and other supporting statistical aspects. This is my preferred method of interacting with a model: treating it as a tutor, not as a servant.1.

Rolling as Meditation

The Die in Question

The first step was to generate some data, which I acquired by rolling a die one-thousand times2. This didn’t take as long as I thought it would - only about 30 minutes - it was actually quite a meditative process. Here’s a table showing the first 10 rolls:

die_rolls <-
    read_file('die_rolls.txt') |>
    str_split(pattern = '', simplify = FALSE) |>
    pluck(1) |>
    as.integer() |>
    as_tibble() |>
    rename(roll = value) |> 
    mutate(roll_id = 1:n()) |>
    select(roll_id, roll) |> 
    # Remove the newline (last value) which converted to NA
    slice_head(n = -1)
Die Rolls - First 10 of 1,000
roll_id12345678910
roll1343555556

What catches my eye straight away is the ominous run of five fives (a probability of 1/77763), which had me concerned about my rolling technique. I then began swapping between my right and left hands every 100 rolls, and ensuring I put a fair amount of momentum into rolling the die.

After rolling, here’s the distribution of the number of rolls per face of the die:

With apologies to Alexander Pope, “to pattern match is to be human”. What I’m drawn to is face six, and the fact that it’s peeking its head above all the others. Could this be an indication that the die has a bias in it? To find out, we need to model the die rolls, and try and put some numbers (and some uncertainty) on the die roll probabilities.

A Trip to Monte Carlo

We’ll start with a very simple model of a die rolling. Here’s the Stan code:

data {
  int<lower=1> n;
  array[n] int<lower=1, upper=6> roll;
  vector[6] alphas; 
}

parameters {
  simplex[6] theta;
}

model {
  theta ~ dirichlet(alphas);
  roll  ~ categorical(theta);
}

We model the die \(roll\) as draws from a \(categorical\) distribution with with a simplex of theta. Our prior distribution for each \(theta\) comes from a Dirichlet, and we can pass the six \(alpha\) priors in at run-time. You’ll see below that I pass all ones into Stan for \(alpha\). By choosing this prior I’m saying that prior to seeing any data, I expect that all combinations of face probabilities have an equal probability density.

The posterior distributions of the six theta we get by running our Stan sampler are the probability distribution over the six die face probabilities, representing our uncertainty of the probabilities of each die face.

After compiling the Stan program, we feed it the data and sample from the posterior distributions of each of the theta parameters. Here’s a dotplot visualisation of these posterior draws, with a 90% credible interval and a vertical line at 1/6 (the probability of a side of a fair die).

Our posterior distribution for \(\theta_6\) is shifted to the right as compared to what we consider a fair die’s \(theta\) to be, which is \(1/6\). The above graph shows a 90% credible interval, and again almost all of that is above the \(1/6\) as well.

How much of the mass is above 1/6?

posterior_mass <-
    die_roll_draws |>
    filter(face == 6) |>
    summarise(mean(theta > 1/6))

posterior_mass
## # A tibble: 1 × 2
##    face `mean(theta > 1/6)`
##   <int>               <dbl>
## 1     6               0.944

Around 94%. Surely the mass shifted over this far means that there is a 95% probability that the real value of \(\theta_6\) is greater that 1/6, and thus the die is biased?

Well, not quite. There’s two problems here; one is me, the other is the model.

Enter the Texan Sharpshooter

This is why I - a statistical dilettante - am very nervous asserting anything about probability in public. Around every corner seems to be a wrong assumption or human foible that catches me out. I’d fallen foul of the Texan Sharpshooter fallacy. Like the sharpshooter drawing the target around the bullet holes after he’d shot, I’d chosen face 6 after I’d seen the data.

In fact, before I’d even rolled the full one-thousand rolls I’d already fallen foul: those five fives I noticed in the first ten rolls were the sharpshooter as well. The 1/7776 probability I mentioned previously is correct for five pre-specified rolls all landing on five, but I would have noticed five rolls of any face. So the question was really “five rolls of any face”, which increased the probability to 1/1296. Within the first ten rolls, there are six positions where the run could have occurred, further reducing the probability to around 1/251, or .4%, far less than the 1/7776 I incorrectly assumed was the case. In fact over the total 1,000 rolls, a run of five of any face has around a 47% chance of occurring.

After the full thousand rolls, face six being higher wasn’t a hypothesis I had going in; I saw the data and then chose it. To compound things, I then used a model which has a flat prior, so each face’s posterior only depends on its own count, and isn’t compared to the other five. The model can’t tell or compensate for the fact I’d fallen prey to the picked face six.

To get a better intuition about this fallacy we can perform a simulation. In the code below we use rmultinom() to generate 100,000 simulations of 1,000 dice rolls. These come out in matrix format, so we do a bit of wrangling to turn it into a tibble:

simulations <- 100000
rolls <- 1000

die_roll_sims <-
    rmultinom(simulations, rolls, rep(1/6, 6)) |>
    as_tibble(.name_repair = 'unique_quiet') |> 
    mutate(face = 1:n()) |>
    pivot_longer(cols = starts_with('..'), names_to = 'sim', names_pattern = '..(\\d+)', values_to = 'count')

This data is slightly different to my original, manually rolled data; for each simulation we get a count of how many times each face was rolled. Now for each of those 100,000 simulations we, take consider two scenarios

  1. Imitate the fallacy: pick the face that had the maximum number of rolls out of the thousand.
  2. Chose a face beforehand (I’ve selected face 6).
die_roll_sim_summary_choices <-
    die_roll_sims |> 
    group_by(sim) |> 
    summarise(
        # Chosing the die with the max rolls (i.e. our face 6)
        chosen_after = max(count),
        # Chosing the face before we've seen the data
        chosen_before = count[face == 6]
    ) |>
    pivot_longer(cols = c(chosen_before, chosen_after), values_to = 'count')

Now we can visualise the distribution of counts for each of the scenarios, overlaying the counts for each of the die faces from or original data:

Now we get a better view of differences in distributions of count. Looking at the count of face six from our original data, we can see it sits near the centre of the distribution of counts when chose the maximum face after we’ve rolled. But if we look at where the counts sit in the distribution when the face was chosen beforehand, we see it sits quite fair right in the distribution. Chosing the die after we’ve seen the data and our six is at the 67.7% percentile, chosing beforehand and it’s at the 94.4% percentile.

The Burden of Choosing a Prior

A second problem is the model itself, and I briefly mentioned it before. I chose a \(Dirichlet(1,1,1,1,1,1)\) as the prior, which means that every die - loadedbiased or not - has equal prior density.4. A die that rolled all sixes all the time is equally as likely as a fair die.

I could crank up the alphas, but what do I pick: ten? One-hundred? One-thousand? This burden of choice is why I often go back to uninformative priors. I could do better with my diagnostics and run prior-predictive checks (I could do better with my diagnostics across the whole pipeline!) which would remove inplausible values, but I’m still having to make a decisions about my belief in each face of the die, which is tough and fixed.

Refining the Model

Let’s remove (well, move) that burden of choice and switch to a hierarchical model:

data {
  int<lower=1> n;
  array[n] int<lower=1, upper=6> roll;
}
parameters {
  real<lower=0> sigma;
  vector[6] eta_raw;
}
transformed parameters {
  vector[6] eta = sigma * eta_raw;
  simplex[6] theta = softmax(eta);
}
model {
  sigma   ~ normal(0, 0.1);
  eta_raw ~ std_normal();
  roll    ~ categorical(theta);
}

We’re still modelling \(roll\) as draws from a categorical distribution, but we’ve changed our prior structure. Our \(theta\) is not a direct parameter, but rather a transformed parameter. It’s derived from \(eta\), which lives on \((-\infty, \infty)\) rather than \([0, 1]\) and is transformed into probabilities via softmax().

Up until here there aren’t any substantial differences between the this and the first model, just a different path to get there. The difference in this model is that it’s hierarchical: we don’t specify the prior on \(eta\), but rather assume \(eta\) is drawn from a \(Normal(0, sigma)\), where \(sigma\) - what we could call the ‘lopsidedness’ of the die - is inferred from the data.5

We still need to specify a prior on \(sigma\), which is still “how alike the faces are”, but it’s a parameter. We can specify a broad prior of a plausible ranges, and the data can rule out the values that don’t fit.

What value are we going to give the prior on \(sigma\)? I’m going to hand-wave my rationale, but I’ve made a choice of .1. What I can do though is run prior-predicative checks to find out how this choice of prior affects the die, and what beliefs its about the die it’s encoding into the model.

# 4000 different dice with 4000 different sigmas 
prior_predictive <-
    tibble(
        die = 1:4000,
        sigma = rhnorm(die, .1) 
    ) |>
    mutate(
        # Each die face gets an eta
        eta = map(sigma, ~rnorm(6, 0, .x)),
        # Eta gets softmaxed into thetas
        theta = map(eta, ~{ exp(.x) / sum( exp( .x) ) }),
        # How far away from fair is (in both directions) is the worst face
        worst_face_pp = map_dbl(theta, ~{ 100 * max( abs( .x - (1/6) ) ) })
    ) |>
    select(sigma, worst_face_pp) |>
    pivot_longer(everything(), names_to = 'Variable') |>
    group_by(Variable) |>
    # Look at the quantile for sigma and the worst face
    summarise(
        Q5 = quantile(value, .05),
        Median = median(value),
        Q95 = quantile(value, .95)
    )
VariableQ5MedianQ95
sigma0.0062402920.06550730.1964192
worst_face_pp0.1294310781.47627665.3545630

So the .1 prior on sigma says that, before seeing the data, I’d expect the one of the faces of the die to be around 1.5 percentage points above fair, with a 90% interval of between .1% and 5%. In other words, I don’t think the die has is reasonably fair, but there could be one face that has a decent amount of bias.

We now run our model with the roll data, and take a look at the posterior distributions for the \(theta\) paramters:

You can see that the marginal posterior distributions for all of the `\(theta_{1..6}\)` have been pulled toward the fair value of 1/6. Why? Let’s look at the table of statistics of the parameters:
die_roll_hierarchical_fit |> print(c('sigma', 'theta'), 'median', 'mean', ~{ quantile(.x, probs = c(.05, .95)) })
##  variable median mean   5%  95%
##  sigma      0.03 0.04 0.00 0.11
##  theta[1]   0.17 0.17 0.16 0.18
##  theta[2]   0.17 0.17 0.16 0.18
##  theta[3]   0.17 0.17 0.16 0.18
##  theta[4]   0.17 0.16 0.15 0.17
##  theta[5]   0.17 0.17 0.15 0.17
##  theta[6]   0.17 0.17 0.16 0.19

The model has determined, when looking at the joint posterior distributions of \(sigma\) and \(theta_{1..6}\) (via \(eta_{1..6}\)) that \(sigma\) is likely small, with a median of 0.03 and a 90% interval of between 0 and .11. This then pulls or regularises the \(theta\) parameters back towards a fair value.

Here’s a better view, comparing \(\theta_6\) marginal posterior distributions from the original flat model and our hierarchical model. , This also answers the initial question I had at the start of the post: is the die biased? The 90% range of \(sigma\) includes the best case of a completely fair die, and worst case of .11, which translates to on average one face being around 2.6% higher.

Testing a Loaded Die

What we want to se3e now is how this hierachical model performs when we give it data from a die that is actually biased. I don’t have a weighted die at hand, so I turn to a simulation. A loaded die would shift the centre of mass within the die towards one of the faces, meaning it is more likely to face down, decreasing its probability and increasing the opposite side’s. In the simulation below I’ve shifted the probability of rolling a six up by 0.05 and then reduced face one by the same amount.

set.seed(3455357)
shift = 0.05
probs <- c(1/6 - shift, 1/6, 1/6, 1/6, 1/6, 1/6 + shift)

loaded_die_rolls <- 
    tibble(
        roll_id = 1:1000,
        roll = sample(1:6, 1000, replace = TRUE, prob = probs)
    )

Here’s the distribution of dice rolls for the simulated loaded die:

loaded_die_rolls |>
    ggplot() +
    geom_bar(aes(roll), fill = 'lightgreen') +
    labs(
        title = "Biased Die Simulation - Distribution of One-Thousand Die Rolls",
        subtitle = "Faces 6 & 1 +/- (1/6) - 0.050",
        x = "Die Face",
        y = "Count of Rolls"
    ) +
    scale_x_continuous(breaks = 1:6)
Let’s run this data through the same hierarchical model:

Here are the posterior distributions of each of the faces of the loaded die:

The posteriors are regularised far less, with faces one and six’s distributions sitting out to the left and right respectively. You’ll also notice that the 90% credible intervals does not overlap a fair value of 1/6. Remember: this is the same model, only the data has changed. But the model has been able to help us avoid the Sharpshooter Problem by pulling posterior likelihoods of a fair die back towards fair thetas, and conversely leaving in place posterior likelihoods of a known biased die.

For clarity, here’s a view of theta[6] and sigma for both my manual die rolls, and the simulated biased die:

die_roll_draws_unified <-
bind_rows(
    loaded_die_roll_hierarchical_draws,
    die_roll_hierarchical_draws,
    .id = 'die_type'
) |>
    ungroup() |> 
    mutate(
        die_type = die_type |> recode_values( 
            "1" ~ "Biased Sim.",
            "2" ~ "Real"
        )
    )
die_roll_draws_unified |>
    filter(face == 6) |>
    rename(`theta[6]` = theta) |> 
    pivot_longer(c(sigma, `theta[6]`), names_to = 'parameter') |> 
    ggplot() +
    stat_dotsinterval(aes(x = value, y = die_type), .width = .9) +
    facet_grid(~parameter, scales = 'free') +
    theme(axis.text.y = element_text(angle = 90, hjust = .5)) +
    labs(
        title = "Posterior Distribution Comparison - Fair vs Loaded Die Rolls",
        x = "Parameter Posterior Density",
        y = "Die Type"
    )
And here is the join posterior distributions of `\(theta_6\)` and `\(sigma\)` (on a log scale) from the fair and loaded die rolls. It shows how the posterior distribution of `\(sigma\)` and `\(theta_6\)` interact, pulling the joint posterior back towards a fiar probability of 1/6. \# Summary

Time for a recap: after rolling the die 1000 times, we tried to model the rolls using a very simple categorical model. But we saw how this simple model, which treated each face of the die independently, combined with a flawed human intuition, could lead us to to wrong conclusions.

We then shifted to a hierarchical model, which took into account the die as a whole, rather than six independent faces. With this model we were able to put some numbers to our uncertainty about the bias of the die.


  1. I’ve made public the two key conversations here and here ↩︎

  2. You can view the raw data here ↩︎

  3. More on this later in the post ↩︎

  4. Dirchlet is conjugate to our Categorical, so each \(\alpha\) in the prior works as a pseudo-count added to our actual roll data ↩︎

  5. There’s a slight detour via \(eta_raw\), which is an alternate way of specifying a prior of \(eta ~ Normal(0, sigma)\) (\(eta = sigma * eta_raw\) with \(eta_raw ~ Normal(0,1)\) is equivalent to that prior). This is a non-standard parameterisation to avoid Neal’s Funnel ↩︎