Raising the bar

Chapter 5. Discrete Random Variables and Probability Distributions

Unit 5.11. Bernoulli variable

Intuitive definition. A Bernoulli variable is an unfair coin.

Formal definition. A Bernoulli variable is defined by Table 5.7 where $latex 0 < p < 1$. Often it is convenient to denote $latex q = 1 - p$.

Table 5.7. Bernoulli variable definition.

Property 1. Mean of the Bernoulli variable: $latex EB = 1 \cdot p + 0 \cdot (1 - p) = p$.

Property 2. Variance of the Bernoulli variable. Using the shortcut for variance,

$latex V(B) = E{B^2} - {(EB)^2} = {1^2}p + {0^2}(1 - p) - {p^2} = p - {p^2} = p(1 - p).$

$latex V(B)$ is a parabola with branches looking downward. It vanishes at $latex p = 0$ and $latex p = 1$ and is maximized at $latex p = 1/2$, which corresponds to a fair coin.

Some facts about sampling not many people will tell you.

Here we provide an example of an unfair coin and use it to discuss the notions of population and sample.

Take any book in English. All alphabetic characters in it are categorized as either consonants or vowels. Denote by $latex p$ the percentage of consonants. We assign 1 to a consonant and 0 to a vowel. The experiment consists in randomly selecting an alphabetic character from the book. Then the outcome is 1 with probability $latex p$ and 0 with probability $latex q$. We have a random variable described by Table 5.7. In this context, the population is the set of consonants and vowels, along with their percentages $latex p$ and $latex q$. This description fits the definition from NCT:

Population and sample. A population is a complete set of all items that interest the investigator. A sample is an observed subset of population values with sample size given by $latex n$.

It is important to imagine the sampling process step by step. To obtain a sample in our example we can randomly select $latex n$ characters and write the sample as $latex {X_1},...,{X_n}$ where some $latex X$'s are unities and others are zeros. The problem is that "random selection" can be understood in different ways. We consider the most important two.

Case I. We randomly select $latex {X_1}$ from the whole book. Suppose it's the first character in the table of contents, which is "C" in NCT. Then $latex {X_1} = 1$. Then we randomly select $latex {X_2}$ again from the whole book. It may well be the same "C" in the table of contents, in which case $latex {X_2} = 1$. We continue like this following two rules: (1) each selection is random and (2) the selection is made from the whole book.

Sampling with replacement. The second condition defines what is called a sampling with replacement: to restore the population, selected elements are returned to it.

After we have obtained a sample of size $latex n$, we can form a sum of the observed values

$latex {S_n} = {X_1} + ... + {X_n} \qquad (\textrm{Eq. 5.10})$

and a sample mean

$latex \bar X = \frac{{{X_1} + ... + {X_n}}}{n} = \frac{{{S_n}}}{n}. \qquad (\textrm{Eq. 5.11})$

After we obtain an actual sample, there will be nothing random. $latex {S_n}$, being a sum of unities and zeros, will be some integer from the range $latex [0,n]$ and $latex \bar X$ will be some fraction from $latex [0,1]$.

The subtlety of the theoretical argument in statistics is that $latex {X_1},...,{X_n}$ are treated not as realized (ex-post) values but as theoretical (ex-ante) random variables that are associated with the drawings from the population.

Let us rewind the movie and think about the selection process again. The first time you don't actually select a character, you just THINK about randomly selecting it from the whole book. The associated random variable is denoted $latex {X_1}$ and, clearly, is described by the same table as the Bernoulli variable $latex B$. Similarly, you think about what possibilities you would face in all other cases and come to the conclusion that all drawings are described by the same table, just the names of the variables will be different.

Identically distributed random variables. Random variables that have the same distributions (that is, values plus probabilities) are called identically distributed.

Comments. (1) Since "values plus probabilities" is all one needs to find the mean and variance of a random variable, identically distributed variables have identical means and variances.

(2) Instead of saying "$latex {X_1},...,{X_n}$ are identically distributed" we can equivalently say "$latex {X_1},...,{X_n}$ are drawn from the same population". In this case the population we draw the sample from is called a parent population.

(3) By construction, in Case I the variables are independent. The common abbreviation for "independent identically distributed" is i.i.d.

(4) In the sum Eq. (5.10) the variables $latex {X_1},...,{X_n}$ may take independently values 0 and 1. If all are zeros, $latex {S_n}$ is zero. If all are unities, $latex {S_n}$ will be equal to $latex n$. In general, $latex {S_n}$ may take any integer value from 0 to $latex n$, inclusive. To emphasize that we deal with variables, in the equation

$latex \mathop {{S_n}}\limits_{0,...,n} = \mathop {{X_1}}\limits_{0,1} + ... + \mathop {{X_n}}\limits_{0,1}$

below the variables I write their possible values. Now the next definition must be clear.

Binomial variable. A binomial variable is a sum of $latex n$ independent identically distributed Bernoulli variables.

Case II. Think about an opinion poll on a simple question requiring a simple answer: Yes or No. Suppose everybody has a definite opinion on the matter, and denote $latex p$ the percentage of voters who have an affirmative answer. A team is sent out to conduct a survey. They record 1 for Yes and 0 for No. The population will be again Bernoulli but there are problems with the sample. It doesn't make sense to ask the same person twice, so the elements in the sample cannot be repeated (this is sampling without replacement). In essence, after having randomly selected the first individual, you cross out that individual from the population before selecting the next one. This implies two things: the observations are dependent and their distributions are not identical.

The general statistical principle is that removing one element from a large population does not change its characteristics very much. So in the case under consideration, the observations can be assumed approximately i.i.d.

Conclusion. If $latex {X_1},...,{X_n}$ are drawn from the same parent population and $latex {\mu _X}$ and $latex {\sigma _X}$ denote the mean and standard deviation of the parent population, then

$latex E{X_1} = ... = E{X_n} = {\mu _X},\,\,\sigma ({X_1}) = ... = \sigma ({X_n}) = {\sigma _X}. \qquad (\textrm{Eq. 5.12})$

Previous: Standardization | Next: An obvious and handy result

Want the whole book (all 14 chapters, exercises, and the index) as a single PDF? Available on Lulu for $1.99.