1   Introduction

Objective of Probability Theory

❼ To define more precisely the nature of uncertainty by developing rules for computing the

probability or likelihood that a given event will occur

❼ Result: will be better able to deal with uncertainty

– In economic models

– In statistical inference

Plan

1. Basic concepts and terminology

2. Definition of probability

3. Basic postulates of probability

4. Counting rules to help us compute probabilities

5. More probability rules

6. Bivariate probabilities

7. Bayes’Theorem

2   Basic Concepts Random experiment

Definition. A random experiment is a process leading to at least two possible outcomes, with uncertainty as to which will occur.

❼ This definition more general than it seems

❼ In principle (but maybe not in practice) a random experiment can be repeated ❼ Examples:

1. Rolling a die

2. Tossing a coin

3. Daily observation of the prime rate of a large bank

❼ Example 3 not repeatable, because can’t replicate conditions of a particular day

Outcomes and events

Definition.  The possible outcomes of a random experiment are called the basic outcomes and the set of all basic outcomes is called the sample space .  We will use the symbol S to denote the sample space.

❼ Notation used to denote basic outcomes will depend on the random experiment

– Usually a capital letter, maybe with subscripts

Definition. An event E is a subset of basic outcomes from the sample space, and it is said to occur if the random experiment gives rise to one of its constituent basic outcomes.  The null event denoted by ∅ represents a situation in which none of the basic outcomes of an event occurs.

Examples

1. Random experiment: Toss a coin once

❼ Basic outcomes: H , T

❼ Sample space: S = {H,T}

❼ Possible events: {H}, {T}

2. Random experiment: Toss a coin twice

❼ Basic outcomes: HH , HT, TH , TT

❼ Sample space: S = {HH,HT,TH,TT}

❼ Possible events:

– A = At least one head = {HH,HT,TH}

– B = No heads = {TT}

3. Random experiment: Observe direction of daily change in prime rate

❼ Basic outcomes: up, down, no change

❼ Sample space: S = {up, down, no change}

❼ Possible events: A = a change in the prime rate = {up, down}

Some things to note

❼ Common to use set notation in describing events – will do this often

❼ Basic outcomes are always mutually exclusive

– Example: H and T cannot both occur at the same time

❼ Some random experiments have an infinite number of possible outcomes

– Example: Observing the value of the prime rate each day

– In these cases, S is infinite

3    Some Concepts from Set Theory

Intersection of Events

Definition. Let A  and B  be two  events in the sample space S .   Their intersection,  denoted A ∩ B, is the set of all basic outcomes in S that belong to both A and B . Hence, the intersection A ∩ B occurs if and only if both A and B occur.   More generally, given K events E1 ,E2 , . . . ,EK , their intersection, E1  ∩ E2  ∩ ... ∩ EK ,  is the set of all basic  outcomes that belong to  every Ei (i = 1, . . . ,K) .

Example: tossing a coin twice

❼ Let A = {HH,HT,TH}, B = {TT}, C = one head and one tail = {HT,TH} ❼ A ∩ C = {HT,TH}

❼ A ∩ B = ∅

Mutually Exclusive Events

Definition. If the  events A  and B  have no  common basic  outcomes,  they are  called mutually exclusive, and their intersection A ∩ B is said to be the empty set.  It follows, then, that A ∩ B cannot occur.

More generally, the K events E1 ,E2 , . . . ,EK  are said to be mutually exclusive if every pair of them is a pair of mutually exclusive events – that is, if Ei ∩ Ej  is the empty set for all i j .

❼ Venn diagrams helpful to illustrate relationships between events

❼ Sample space S represented by box

Venn Diagram of Intersection

Copyright ➞Pearson Education, publishing as Prentice Hall

Interpretation of intersection

❼ If A ∩ B ∅, both A and B have occurred

❼ If A ∩ B = ∅, neither A nor B have occurred

❼ Another possible event that might be of interest:  “either A or B or both have occurred”

Union of Events

Definition. Let A and B be two events in the sample space S .  Their union, denoted A ∪ B, is the set of all basic outcomes in S that belong to at least one of these two events. Hence, the union A ∪ B occurs if and only if either A or B or both occurs.

More generally, given K events E1 ,E2 , . . . ,EK , their union, E1 ∪ E2 ∪ ... ∪ EK , is the set of all basic outcomes that belong to at least one of these K events.

Example: Tossing a coin twice

❼ Union of A = {HH,HT,TH} and C = {HT,TH} is A

Venn Diagram of Union

Copyright ➞Pearson Education, publishing as Prentice Hall

Collectively Exhaustive Events

Definition. Let E1 ,E2 , . . . ,EK   be K events in the sample space S .  If E1 ∪ E2 ∪ ... ∪ EK  = S, these K events are said to be collectively exhaustive .

Example: Tossing a coin twice

❼ A = {HH,HT,TH}, B = {TT}

❼ A ∪ B = {HH,HT,TH,TT} = S

❼ Therefore, A and B are collectively exhaustive

Complement of an Event

Definition. Let A  be an event in the sample space S .   The set of basic outcomes of a random experiment belonging to S but not to A is called the complement of A and is denoted A .

Example: Tossing a coin twice

❼ A = {HH,HT,TH}, B = {TT}

❼ Event B is the complement of event A, i.e., A = B

❼ Also, B = A

Three Useful Results from Set Theory

(i) Let A and B be two events. Then the events A ∩ B and A ∩ B are mutually exclusive and their union is B .

(ii) Let A and B be two events. The events A and A∩B are mutually exclusive, and their union

is A ∪ B .

(iii) Let E1 ,E2 , . . . ,EK  be K mutually exclusive and collectively exhaustive events, and let A be

some other event.  Then the K events E1  ∩ A,E2  ∩ A, . . . ,EK  ∩ A are mutually exclusive, and their union is A.

Example: Result (ii)

❼ Tossing a coin twice

– Sample space: S = {HH,HT,TH,TT}

– A = {HH,HT,TH}, A = {TT}

– Let event D = at least one tail = {HT,TH,TT}

– Then A ∩ D = {TT} and A ∩ (A ∩ D) = ∅

– A ∪ (A ∩ D) = {HH,HT,TH,TT} = A ∪ D

❼ Important note: NEVER write 0 for the empty set

– 0 ∅

Another example

Question 3.2 of Newbold et al.  (2013) The sample space is

S = {E1 ,E2 ,E3 ,E4 ,E5 ,E6 ,E7 ,E8 ,E9 ,E10 } .

Let A = {E1 ,E3 ,E7 ,E9 } and B = {E2 ,E3 ,E8 ,E9 }.

a. What is A intersection B?

b. What is the union of A and B?

c. Is the union of A and B collectively exhaustive?

Exercise: Try question 3.6 (application of results (i) and (ii)).

4   Definition of Probability

Defining Probability

❼ Surprisingly, more than one definition of probability has been proposed

❼ Will look at three:

1. Subjective probability

2. Relative frequency definition

3. Classical definition

Subjective Probability

Definition. Subjective probabilities express an individual’s degree of belief about the likelihood that an event will occur.

❼ Based on judgement and experience of the decision-maker

❼ Compatible with many real world situations

❼ Problem: each individual’s subjective probability is likely to be different ❼ Therefore, difficult to operationalize

Relative Frequency Definition of Probability

Definition. Let NA   be the number of occurrences of event A in N repeated trials of a random experiment.  Then the probability that A occurs is the limit of the ratio NA /N  as the number of trials becomes infinitely large.

❼ Corresponding mathematical formula:

P(A) =  lim NA

❼ Objective approach to defining probability – everyone watching sequence of trials should

arrive at same value for P(A)

❼ Problem: also difficult to operationalize, because

1. Rarely possible to carry out an infinite number of trials

2. Some random experiments cannot be easily duplicated – e.g., economic conditions

Classical Definition of Probability

Definition. Let N be the number of basic outcomes in the sample space and NA  be the number of basic outcomes that belong to event A .  Then the probability that event A will occur is given by

P(A) = NA

❼ Classical approach studies probability in well-defined situation: number of outcomes of ran-

dom experiment is finite, each outcome is equally likely to occur

❼ Enables us to deduce probability of a particular event logically

❼ Value that is computed for P(A) is also the value to which relative frequency would converge

if random experiment were repeated

– Classical definition close in spirit to relative frequency definition

5   The Classical Probability Postulates The Three Probability Postulates

❼ In this course, will use the classical definition of probability

❼ Some notation: Let

– S = sample space of a random experiment

– Oi  = ith basic outcome

– A = an event

– P(A) = probability that event A occurs

❼ The Classical Probability Postulates:

1. 0 ≤ P(A) ≤ 1

2. P(A) =Pi∈A P(Oi )

3. P(S)=1

What do the postulates tell us?

❼ (1) says that probabilities must lie between 0 and 1, like relative frequencies

❼ P(A) = 0 implies that event A will never occur

❼ P(A) = 1 implies that event A is certain to occur

❼ Uncertainty about A exists only if 0 < P(A) < 1

❼ (2) gives us a basic rule for computing P(A) – sum P(Oi )

❼ (3) follows from definition of S – tells us that one of the basic outcomes must occur ❼ Postulates also have four important consequences

6    Consequences of the Probability Postulates

Consequence (i)

(i) If the sample space S consists of n equally likely basic outcomes, O1 ,O2 , . . . ,On , then each of these has probability 1/n; that is,

P(Oi ) = ,    i = 1, . . . ,n .

Proof :

If each basic outcome is equally likely, then P(Oi ) = k  ∀ i.

From postulates 2 and 3, we have

P(S) =XP(Oi ) = 1

i∈S

∴ X k = 1

i∈S

nk = 1      since there are n elements in S

∴ k = = P(Oi )

Consequence (ii)

(ii) If the sample space S consists of n equally likely basic outcomes and the event A consists of

nA  of these outcomes, then

A

Proof :

From the second postulate,

P(A) =XP(Oi ) .

i∈A

But from consequence (i), P(Oi ) = 1/n. Therefore,

P(A) =X = nA · = .

Consequence (iii)

(iii) Let A and B be mutually exclusive events. Then the probability of their union is the sum

of their individual probabilities; that is,

P(A ∪ B) = P(A) + P(B) .

More generally, if E1 ,E2 , . . . ,EK  are mutually exclusive events,

P(E1 ∪ E2 ∪ ... ∪ EK ) = P(E1 ) + P(E2 ) + ... + P(EK ) . This result also follows from postulate 2.

Consequence (iii)

Proof :

From the second postulate,

P(A ∪ B) =  X  P(Oi ) .

i∈ A∪B

But since A and B are mutually exclusive, we know that A ∩ B = ∅. Therefore,

P(A ∪ B) =XP(Oi ) +XP(Oi )

i∈A                       i∈B

= P(A) + P(B)

Consequence (iv)

(iv) If E1 ,E2 , . . . ,EK  are collectively exhaustive events, the probability of their union is P(E1 ∪ E2 ∪ ... ∪ EK ) = 1 .

Why? Because the union of collectively exhaustive events is the sample space, which according to postulate 3 has a probability of 1.

What have we learned about computing probabilities thus far?

❼ Rule for computing probabilities of events (consequence (ii))

❼ Rule for computing probability of union of mutually exclusive events (consequence (iii)) ❼ Important: All rules are valid only for case where all basic outcomes are equally likely

❼ To implement these rules, need to know how many basic outcomes are included in each event ❼ Problem of computing probabilities has been reduced to problem of counting basic outcomes

Examples

1. Tossing a fair die

2. Question  3.15  of Newbold  et  al.   (2013) A manager has available a pool of eight employees who could be assigned to a project-monitoring task.  Four of the employees are women and four are men.   Two of the men are brothers.   The manager is to make the assignment at random, so that each of the eight employees is equally likely to be chosen. Let A be the event“chosen employee is a man” and B the event“chosen employee is one of the brothers.”

a. Find the probability of A.

b. Find the probability of B .

c. Find the probability of the intersection of A and B .

d. Find the probability of the union of A and B .

7    Counting Rules        Counting the number of outcomes

❼ Examples considered thus far have relatively few basic outcomes → easy to compute prob-

abilities using the postulates and their consequences

❼ What can we do if the number of basic outcomes is large?

– Tree diagrams can help with multi-stage random experiments

– BUT: Still only useful if number of outcomes at each stage is relatively small ❼ Rules to facilitate counting of outcomes would be helpful

Example: Tree Diagram

The Multiplication Principle

❼ Will derive four counting rules

❼ First rule known as the Multiplication Principle

❼ Applies to composite experiment that is result of a sequence of random experiments ❼ Examples:

– Tossing a die, then tossing a coin (tree diagram example)

– Tossing a coin 10 times

The Multiplication Principle

Formal statement:

Rule. Multiplication Principle . Suppose that experiment E1  has n1  possible outcomes, exper- iment E2  has n2  possible outcomes, ..., and experiment EK  has nK  possible outcomes.  Then the total number of basic outcomes of the composite experiment E1 E2 . . . EK  is n1  · n2  · ... · nK .

❼ In other words: total number of basic outcomes is product of number of outcomes at each

step

Examples: Multiplication Principle

1. Tossing a die, then tossing a coin

❼ Step 1: Toss die → n1  = 6 outcomes

❼ Step 2: Toss a coin → n2  = 2 outcomes

❼ Therefore, n = n1  · n2  = 6 · 2 = 12

2. Tossing a coin 10 times

❼ Two outcomes at each step

❼ Therefore n = 2 · 2 · 2 · ... · 2 = 210  = 1, 024

❼ Each basic outcome is a sequence of heads and tails:  HHHHH TTTTT or HTHTH

THTTT or TTHTT HHTHT, etc.

Examples: Multiplication Principle

3. Draw a letter from the set {A,B,C,D} three times, with replacement

❼ nk  = 4 at each step, k = 1, 2, 3

❼ Therefore, n = 4 · 4 · 4 = 43  = 64

❼ Each basic outcome is a sequence of three letters: ABC,BCD,CDA,DAB, etc.

❼ What would be the number of outcomes if the letters were drawn without replacement?

Permutations

❼ Not uncommon to encounter problems like this one:  How many different orderings of the

letters {A,B,C,D} are possible?

❼ Could answer this question by tediously listing all possible orderings:  ABCD , BCDA,

CDAB , DABC, etc.

❼ Case of sampling without replacement

❼ Problem: As the number of items to be ordered rises, easy to miss some possibilities

❼ Would be helpful to have a formula for quickly computing number of possible orderings

Number of Permutations of n Objects

Definition. A permutation is an ordering or arrangement of objects.

❼ Now think of ordering problem as a sequence random experiments

– Start with n objects

– Let E1  = choose one of the n objects and place it in position 1 → n possible outcomes

– Let E2  = choose one of the remaining n − 1 objects and place it in position 2 → n − 1 possible outcomes

– Let E3  = choose one of the remaining n − 2 objects and place it in position 3 → n − 2 possible outcomes

– Continue in this fashion until all objects have been selected

❼ Now can apply Multiplication Principle

Number of Permutations of n Objects

Rule. Number of Permutations of n Objects . The number of possible permutations of n objects is

n(n − 1)(n − 2) ... 2 · 1 = n! .

❼ n! =“n factorial”

❼ By definition, 1! = 0! = 1

❼ Example: How many different permutations of the letters {A,B,C,D} are possible?

– Answer: 4! = 4 · 3 · 2 · 1 = 24

– Therefore, 24 permutations of the four letters are possible

– Could use a tree diagram to verify that this answer is correct

Permutations of x objects selected from n

❼ Sometimes the number of items/objects we wish to select is less than total number available – Wish to select x objects, where x < n

❼ Another example of sampling without replacement

❼ Can once again apply Multiplication Principle:

– Start with n objects

– n possible choices in the 1st round (E1 )

– n − 1 possible choices in the 2nd round (E2 )

– n − 2 possible choices in the 3rd round, etc.

– In round x, will be n − x + 1 objects to choose from

✯ x − 1 is number of objects that have already been chosen

Number of Permutations of x Objects Chosen from n

❼ Now can apply the multiplication principle again:  total number of possible orderings is

n(n − 1)(n − 2) ... (n − x + 1)

❼ Can simplify this formula to derive new counting rule

Rule. Number of Permutations of x Objects Chosen from n . The number of permutations of x objects chosen from n, Px(n), is the number of possible arrangements when x objects are to be selected from a total of n and arranged in order.  This number is

Px(n)  = n!

❼ How do we arrive at the formula for the new counting rule?

Pn     =

=

=

n(n − 1)(n − 2) ... (n − x + 1) n(n − 1)(n − 2) ... (n − x + 1)(n − x)(n − x − 1) ... (2)(1)

(n − x)(n − x − 1) ... (2)(1)

n!

(n − x)!

Examples: Permutations of x objects from n

1. Consider the set of 8 digits {1, 2, 3, 4, 5, 6, 7, 8}.

(a) How many 3-digit numbers can be constructed from this set, if sampling without re-

placement?

(b) What is the probability of selecting one of these numbers?

(c) How many of the 3-digit numbers will contain the digit 8?

(d) What is the probability that a number selected at random from the set of 3-digit numbers constructed in part (a) will include an 8?

2. How many different orderings of two letters from the set {W,X,Y,Z} are possible if no letter is allowed to appear more than once?

Combinations

❼ Sometimes we don’t care about the order in which items are selected from a set

– Drawing digits to construct a number: order is important

– Selecting people to be members of a group: order usually isn’t important

Definition. A combination is a particular selection of x objects from n, for which the ordering of the objects is irrelevant.

❼ Is there a way to calculate the number of possible combinations of objects?

Counting Combinations

❼ Let Cx(n)  be the number of combinations of x objects selected from n

❼ Consider 1 combination of x objects → second counting rule implies that there are x! different

permutations (orderings) of this combination

❼ Every combination can be ordered in x! different ways

❼ Therefore

Px(n)  = Cx(n) · x!

❼ Re-arrange to derive new rule

Combinations of x Objects Chosen from n

Rule. Number of Combinations of x Objects Chosen from n . A combination is  a particular selection of x objects from n, for which the ordering of the objects is irrelevant.  The number of possible combinations of x objects chosen from n is

n!

C =

❼ Now have enough counting rules to be able to compute probabilities relatively easily

Examples: Number of Combinations

1. How many combinations of two letters can be chosen from the set {W,X,Y,Z}, when sam- pling without replacement?

2. Question 3.31 of Newbold (2013) A student committee has six members – four under- graduate students and two graduate students.  A subcommittee of three members is to be chosen randomly, so that each possible combination of three of the six students is equally likely to be selected.  What is the probability that there would be no graduate students selected?

Examples: Number of Combinations

3. Question 3.35 of Newbold (2013) A work crew for a building project is to be made up of two craftsmen and four labourers selected from a total of five craftsmen and six labourers available.

a. How many different combinations are possible?

b. The brother of one of the craftsmen is a labourer.  If the crew is selected at random, what is the probability that both brothers will be selected?

c. What is the probability that neither brother will be selected? (Do this part as a exer- cise!)

8   Probability Rules More Probability Rules

❼ Already have a basic rule for computing probabilities, derived from the probability postu-

lates:

P(A) =

n

❼ Also have a rule for computing probability of the union of mutually exclusive events: P(A ∪ B) = P(A) + P(B)

❼ BUT: These rules only apply to relatively simple problems

❼ Need some rules for more complex cases → six new probability rules

Probability of a Complement

Rule. (1) Probability of a Complement. Let A be an event and A its complement.  Then

P(A) = 1 − P(A) .

Proof : A and A are mutually exclusive and collectively exhaustive events.  Therefore, conse- quence (iii) implies that

P(A ∪ A) = P(A) + P(A) .

Furthermore, using consequence (iv),

P(A ∪ A) = P(S) = 1 .

∴  P(A) + P(A) = 1  ⇒ P(A) = 1 − P(A) .

Probability rule (2)

Rule. (2) Let A and B be two events.  Then

P(B) = P(A ∩ B) + P(A ∩ B) .

Proof : Result (i) from set theory states that A∩B and A∩B are mutually exclusive and that

(A ∩ B) ∪ (A ∩ B) = B .

Therefore, consequence (iii) of the probability postulates applies, so that

P(B) = P(A ∩ B) + P(A ∩ B) .

Probability rule (3)

Rule. (3) Let A and B be two events.  Then

P(A ∪ B) = P(A) + P(A ∩ B) .

Proof : Result (ii) from set theory tells us that

A ∪ (A ∩ B) = A ∪ B

and that A and A ∩ B are mutually exclusive.  Therefore, consequence (iii) of the probability postulates can be applied:

P(A ∪ (A ∩ B)) = P(A) + P(A ∩ B)

= P(A ∪ B)

Addition Rule of Probabilities

Rule. (4) Addition Rule of Probabilities . Let A and B  be two events.  The probability of their union is

P(A ∪ B) = P(A) + P(B) − P(A ∩ B) .

Proof: Do it as an exercise. It follows from rules (2) and (3).