Last Updated: March 26, 2024 - Updated due date

Updated Due Date: April 11. 11:59 pm.

Please consult the syllabus about proper homework format, due date, and policy. For programming questions, you may use all of Pandas and Numpy, unless a problem states otherwise. Certain problems allow other packages and modules. You may not use packages and modules other than what is explicitly allowed.

Bernoulli - Binomial - Beta - Bayes (12 points total)

Added: January 28, 2024

<aside> ⚠️ For this problem, you may use scipy.stats.beta, matplotlib.pyplot, numpy

</aside>

1. Let $K$ be the number of heads of flipping a biased coin $n$ times. Let $P$ be the probability that the coin comes up heads.

   1. (1 pt.) Express $K$ as a sum iid Bernoulli random variables $X_i \sim \text{Bern}(P)$. What are the possible values of $X_i$?
   2. (1 pt.) What is $\mathbb E[X]$? $\text{Var}[X]$?

2. (2 pts.) In fact, $K$ is called a Binomial random variable with parameters $n$ and $p$, and write $K \sim \text{Binom}(n,P)$, which has a probability mass function:

   $$ f_K(k) = \binom{n}{k}P^k(1-P)^{n-k} $$

   Computing summary statistics from this PMF is difficult. Instead, using the summation above in part 1, derive the formulas for $\mathbb E[K]$ and $\text{Var}[K]$ for the Binomial distribution.

3. (3 pts.) Suppose we don’t know $P$, and instead treat it as a random variable. A reasonable distribution for $P$ is the Beta distribution, which has itself two parameters $\alpha, \beta$, that we’ll consider as integers for this problem. (Parameters like $\alpha, \beta$ describing distributions of a parameter $P$ of another distribution are often called hyperparameters.)

   We then can write $P \sim \text{Beta}(\alpha, \beta)$. The Beta distribution has the PDF:

   $$ f_P(p) = \begin{cases} c\cdot p^{\alpha-1}(1-p)^{\beta-1} & \text{ if } 0 < p < 1\\ 0 & \text{ otherwise} \end{cases} $$

   where $c$ is a normalizing factor that depends on $\alpha, \beta$, given by the formula:

   $$ c = \frac{(\alpha + \beta-1)!}{(\alpha-1)!(\beta-1)!} $$

   Using Bayes’s theorem, argue that the posterior distribution $P\mid K$ for the bias $P$ given the number of heads $K$ after $n$ throws is also Beta distributed: $P\mid K \sim \text{Beta}(\alpha^\prime, \beta^\prime)$.

   Hint: You can argue this by showing that the posterior PDF $f_{P\mid K}(p)$ satisfies:

   $$ f_{P\mid K}(p) \propto p^{\alpha^\prime-1}(1-p)^{\beta^\prime-1} $$

   What are $\alpha^\prime$ and $\beta^\prime$ in terms of $\alpha$ and $\beta$?

4. Suppose you toss the biased coin ten times, resulting in a total of seven heads. Assume a prior distribution $P \sim \text{Beta}(1, 1)$ for the coin’s bias.

   1. (1 pt.) What is the posterior distribution $P\mid K$?
   2. (1 pt.) Plot the prior and posterior distributions on the same plot. You may use the spicy.stats.beta module.
   3. (1 pt.) Briefly comment on what the posterior represents here, contrasting it with the frequentist estimator for $P$ given by $\bar P = K/n$. Indicate this estimator on your plot. What does the Bayesian treatment of this problem give you that this simple estimator does not?

5. (2 pts.) Let us now consider the impact of our prior assumption. Plot the prior and posterior distributions for the alternative priors $P \sim\text{Beta}(5,5)$, $P \sim \text{Beta}(2, 5)$ and $P\sim \text{Beta}(10, 2)$. Compare and contrast these priors and posteriors with the ones you obtained with the prior $P\sim \text{Beta}(1, 1)$.