Probability · Lesson 3 of 5

Combining Cases and Deriving Bayes’ Rule

“Combine mutually exclusive cases, then reverse a conditional probability using Bayes’ rule.”

Learning Objectives

• Build a partition from disjoint, exhaustive positive-probability cases. • Derive total probability by decomposing an event into disjoint pieces. • Distinguish a forward probability from a reverse conditional probability. • Derive Bayes’ rule and interpret prior and posterior probabilities. • Use probability trees and case tables to organise a multi-stage model.

Different routes to the same observation

A white ball may have come from either of two bags. A construction job may finish on time whether a strike occurs or not. To calculate the probability of the final observation, we must include every possible route to it. The routes should cover all possibilities without counting any outcome twice.

Suppose bag I contains two white and three red balls, while bag II contains four white and five red balls. Choosing a bag and then drawing a ball is a two-stage experiment. Knowing the chosen bag makes the colour probability straightforward. Learning only that the ball is white creates a reverse question: which bag was more likely to have produced it? We first develop the case structure needed for both questions.

A partition organises all possible cases

A useful case split tells us that exactly one case occurs. Cases must not overlap, and no possible outcome may be left out. We also require each conditioning case to have positive probability so that the conditional probabilities used in our calculations are defined.

Definition

Partition of a sample space

Events $E_{1}, \dots, E_{n}$ form a partition for the total-probability calculation when they are pairwise disjoint, their union is $S$ , and $P (E_{i}) > 0$ for every $i$ . Pairwise disjoint means $E_{i} \cap E_{j} = \emptyset$ whenever $i \neq = j$ . Exhaustive means their union contains every outcome.

An event $B$ and its complement $B^{'}$ make a two-case partition when both have positive probability. Two events can also split a sample space into the four regions $E \cap F$ , $E \cap F^{'}$ , $E^{'} \cap F$ , $E^{'} \cap F^{'}$ . Some regions might be empty or have zero probability; omit those when forming a positive-probability partition. A sample space can have many valid partitions: choose the one matching the information supplied.

An event crossing several cases— Each piece of A belongs to one partition event. Their union is A.

Add the probability contributed by each case

The event we want may stretch across several cases. Within a particular case, use a conditional probability. To recover its contribution to the original experiment, multiply by the chance of reaching that case. Then add the contributions, because the cases are disjoint.

Since $S = E_{1} \cup \dots \cup E_{n}$ , intersection distributes to give $A = (A \cap E_{1}) \cup \dots \cup (A \cap E_{n})$ . These pieces are disjoint because the partition events are disjoint. Therefore $P (A) = \sum_{i} P (A \cap E_{i})$ . Finally use multiplication on each piece: $P (A \cap E_{i}) = P (E_{i}) P (A ∣ E_{i})$ .

Theorem of total probabilityLaTeX

P (A) = i = 1 \sum n P (E_{i}) P (A ∣ E_{i})

The

E_{i}

form a partition.

P (E_{i})

is the weight of case

i

, and

P (A ∣ E_{i})

is the chance of

A

within that case. Each product is a joint probability in the original sample space.

This is a weighted average of the case-specific probabilities. The weights $P (E_{i})$ add to one, so $P (A)$ lies between the smallest and largest values of $P (A ∣ E_{i})$ . An unweighted average works only when all cases are equally likely. Having several named cases does not make their probabilities equal.

Example — A construction job and a possible strike

Problem
A strike has probability $0.65$ . A job finishes on time with probability $0.32$ if there is a strike and $0.80$ if there is no strike. Find the overall on-time probability.

1.Let $B$ mean strike and $A$ mean on-time completion. The two cases are $B, B^{'}$ , with probabilities $0.65, 0.35$ .
2.The strike-and-on-time route contributes $0.65 \times 0.32 = 0.208$ . The no-strike-and-on-time route contributes $0.35 \times 0.80 = 0.280$ .
3.Add the disjoint routes: $P (A) = 0.208 + 0.280 = 0.488$ . This lies between $0.32$ and $0.80$ , as a weighted average should.

Example — An urn changes after a first draw

Problem
An urn starts with five red and five black balls. Draw a ball, return it, add two more of its colour, and draw again. Find the probability that the second ball is red.

1.Partition by the first colour: $R_{1}$ and $B_{1}$ each have probability $1/2$ . After the first colour is known, there are $12$ balls in the urn.
2.Given a first red, seven of the $12$ are red. Given a first black, five of the $12$ are red.
3.Thus $P (R_{2}) = (1/2) (7/12) + (1/2) (5/12) = 1/2$ . The overall red probability remains one-half even though the second draw depends on the first. An unchanged marginal probability does not by itself establish independence.

Reverse the direction of the information

Total probability answers a forward question: how likely is an observation when the starting case is uncertain? Bayes’ rule answers a reverse question: after the observation, how likely is each starting case? We are conditioning the same joint probability in the other direction.

Suppose $E_{1}, \dots, E_{n}$ is our partition and an observed event $A$ has positive probability. By conditional probability, $P (E_{i} ∣ A) = P (E_{i} \cap A) / P (A)$ . The numerator is $P (E_{i}) P (A ∣ E_{i})$ by multiplication. The denominator is the sum of these joint contributions by total probability. Combining these substitutions gives Bayes’ rule.

Bayes’ ruleLaTeX

P (E_{i} ∣ A) = \frac{P ( E _{i} ) P ( A ∣ E _{i} )}{j = 1 \sum n P ( E _{j} ) P ( A ∣ E _{j} )}

The observed event must satisfy

P (A) > 0

. The numerator is the contribution from the requested case; the denominator includes the contributions from all cases.

Definition

Hypothesis, prior, and posterior

The partition events are possible hypotheses, or candidate cases. The prior probability $P (E_{i})$ describes a hypothesis before the observation. The posterior probability $P (E_{i} ∣ A)$ describes it after the observation. The likelihood $P (A ∣ E_{i})$ describes how likely the observation is if that hypothesis holds.

Calling a hypothesis a cause is convenient in some stories, but the formula requires a probability model, not a claim about physical causation. It compares how much probability each case contributes to the observed event. A case with zero likelihood contributes nothing, even when its prior is positive.

Example — Which bag supplied a red ball?

Problem
Bag I has three red and four black balls; bag II has five red and six black balls. Choose a bag with probability $1/2$ each and draw one ball. Given a red ball, find the probability it came from bag II.

1.Let $E_{1}, E_{2}$ name the selected bags and $A$ mean red. The priors are both $1/2$ ; the likelihoods are $3/7$ and $5/11$ .
2.The joint red contributions are $(1/2) (3/7) = 3/14$ and $(1/2) (5/11) = 5/22$ . Their sum is $P (A) = 34/77$ .
3.The requested posterior is $P (E_{2} ∣ A) = (5/22) / (34/77) = 35/68$ . It is slightly above $1/2$ because bag II has the slightly greater red proportion. The greater number of red balls alone is not the right comparison; the proportions matter.

Example — A gold coin and its partner

Problem
Three equally likely boxes contain gold-gold, silver-silver, and gold-silver. Choose a box, then choose either coin uniformly. If the selected coin is gold, find the probability that the other coin is also gold.

1.Use the three boxes as hypotheses with priors $1/3$ each. The likelihoods of a gold selection are $1, 0, 1/2$ .
2.The observed gold probability is $(1/3) (1) + (1/3) (0) + (1/3) (1/2) = 1/2$ . A gold partner means the first box was selected.
3.Bayes gives $P (E_{1} ∣ gold) = (1/3) / (1/2) = 2/3$ . Eliminating the all-silver box does not leave two equally likely boxes: the all-gold box supplies twice as much probability weight to the observation.

A likelihood is not a posterior

$P (A ∣ E_{i})$ asks how likely the evidence is under a chosen case. $P (E_{i} ∣ A)$ asks how likely that case is after seeing the evidence. Reversing them discards the priors and the alternative explanations. Equal priors can cancel from a Bayes ratio, but unequal priors cannot be ignored.

A reliable organisation method

Organising the numbers before calculating is often more useful than memorising the formula. A table makes clear which quantities are given and which must be built. Write the requested posterior in symbols first so that the final numerator uses the right case.

Case	Prior	Likelihood of A	Joint contribution
$E_{1}$	$P (E_{1})$	$P (A ∣ E_{1})$	$P (E_{1}) P (A ∣ E_{1})$
$E_{2}$	$P (E_{2})$	$P (A ∣ E_{2})$	$P (E_{2}) P (A ∣ E_{2})$
All cases	Priors sum to 1	Not added without weights	Contributions sum to $P (A)$

For a forward probability, stop after adding the final column. For a posterior, divide the requested row of that column by its total. Check that all posterior probabilities are nonnegative and add to one. These checks follow directly from dividing the disjoint pieces of $A$ by the same positive $P (A)$ .

Check your understanding

Choose disjoint, exhaustive cases before applying either theorem. Decide whether the question ends with an observation probability or asks for a case after an observation.

Quick check

Which pair is a valid positive-probability partition?

Quick check

$P (E_{1}) = 0.7$ , $P (E_{2}) = 0.3$ , and the likelihoods of $A$ are $0.2, 0.8$ . What is $P (A)$ ?

Quick check

In Bayes’ rule, what does the denominator represent?

Quick check

The prior weights are $1/2, 1/2$ and likelihoods are $1, 0$ . Given the evidence, what is the posterior of the first case?

Quick check

After selecting a gold coin in the three-box example, why is the all-gold box more likely than the mixed box?

Quick check

Which quantity is a prior?

Practice Problems

Repeat the changing-urn calculation when the starting urn contains five red and five black balls and two matching-colour balls are added after the first draw. Explain why the first and second colours are still dependent.
A route is chosen with probabilities $0.6$ and $0.4$ . Delay probabilities on those routes are $0.1$ and $0.3$ . Find the overall delay probability.
For the preceding routes, a delay occurs. Find the probability that the second route was used.
A box is chosen uniformly from three boxes with red proportions $1/4, 1/2, 3/4$ . Find the probability of drawing red and the posterior probability of the third box given red.
Two cases have priors $0.8, 0.2$ and equal likelihood $0.4$ for an observation. Calculate their posteriors and explain the result.
An event $B$ and its complement have positive probabilities. Derive total probability for this two-case partition and explain why overlapping cases cannot simply be added.

The red probability is $(1/2) (7/12) + (1/2) (5/12) = 1/2$ . Dependence is shown by $P (R_{2} ∣ R_{1}) = 7/12 \neq = 1/2 = P (R_{2})$ .

Key Takeaways

• A partition supplies exactly one positive-probability case for every possible outcome. • Total probability adds the weighted contributions $P (E_{i}) P (A ∣ E_{i})$ . • Bayes’ rule divides the requested case contribution by the total evidence probability. • A prior describes a case before evidence; a posterior describes it after evidence. • Keep every possible source of the evidence in the denominator, including cases with small priors.

Previous · Lesson 2

Joint Events, Sequential Trials, and Independence

Next · Lesson 4

Reasoning Backward from Evidence

Different routes to the same observation

A partition organises all possible cases

Add the probability contributed by each case

Reverse the direction of the information

A reliable organisation method

Check your understanding

Quiz

Practice Problems

Key Takeaways