SC2000/CZ2100

Probability & Statistics

Week 3

1
Ch 4. Bivariate Data

● Introduction to Bivariate Data

● Pearson Correlation and Covariance
● Properties of Person Correlation
● Variance Sum Law II

2
 Variance Sum Law II

- Linear combination of 2 independent variables

X and Y
2
Variance of 𝑋𝑋 ± 𝑌𝑌: 𝜎𝜎𝑥𝑥±𝑦𝑦 = 𝜎𝜎𝑥𝑥2 + 𝜎𝜎𝑦𝑦2

- If the variables X and Y are correlated

2
Variance of 𝑋𝑋 ± 𝑌𝑌: 𝜎𝜎𝑥𝑥±𝑦𝑦 = 𝜎𝜎𝑥𝑥2 + 𝜎𝜎𝑦𝑦2 ± 2𝜌𝜌𝜎𝜎𝑥𝑥 𝜎𝜎𝑦𝑦

- For computation based on a sample

2
𝑠𝑠𝑥𝑥±𝑦𝑦 = 𝑠𝑠𝑥𝑥2 + 𝑠𝑠𝑦𝑦2 ± 2𝑟𝑟𝑠𝑠𝑥𝑥 𝑠𝑠𝑦𝑦
3
Eg: Students took 2 parts of a test, each worth 50 points. Part
A has a variance of 25, and Part B has a variance of 49.
The correlation between the test scores is 0.6.
(i) If the teacher adds the grades of the two parts together to
form a final test grade, what would the variance of the
final test grades be?
(ii) What would the variance of Part A - Part B be?

(i) Var (A + B) = 25 + 49 + 2*0.6*√25*√49

= 116

(ii) Var (A − B) = 25 + 49 - 2*0.6*√25*√49

= 32
4
Ch 5. Probability

● Basic Concepts – Review of Set Theory

and Venn Diagram
● Counting – Ordered, Unordered, Sampling
With and Without Replacement.
● Probability Theory
● Base Rate: Bayes’ Theorem

5
 Set Theory - Review
A set is a collection of elements.
Eg: A = {Head, Tail}, Head ∈ A, Tail ∈ A
B = {1, 2, 3, 4, 5, 6}
C = {1, 3, 5}

Set C is a subset of Set B since all elements

in C are also in B. Notation: C ⊂ B

The set of all elements in a given context is

known as an universal set 𝛀𝛀.
Null set 0 is a set with no element.
6
 Venn Diagram
universal set 𝛀𝛀

B A

Set A ⊂ 𝛀𝛀. Complement of A, denoted by A’, consists of all elements in 𝛀𝛀 that are
not in A.
Eg: 𝛀𝛀 = {1,2,3,4,5,6}, A={1,2}, A’={3,4,5,6}

Union of A and B consists of all elements in A or B. Notation: A ∪ B.

Eg: {1,2,3} ∪ {2,3,4} = {1,2,3,4}

Intersection of A and B consists of all elements in both A and B. Notation: A ∩ B.

Eg: {1,2,3} ∩ {2,3,4} = {2,3}
7
Consider rolling a die. Let the outcome set A={1,2,3}
and set B={2,4,6}. Match the following:

8
 Counting
When solving probability problems, we may
involve counting.
Eg: Calculating the probability of guessing
the correct 4 digits numeric code.
Each digit is sampled from {0,1,2,…,9}.
Different scenarios for consideration:
• sampling with replacement
• sampling without replacement
• digits drawn are ordered
• digits drawn are unordered
9
 Counting – general scenario

With replacement

Ordered
Eg:
Pick 3 numbers Two options:
(one at a time) with orWithout
without replacement.
replacement
from a set of 10
Sampling digits. After picking each number
from the set, thereplacement
Without number can
Arrangement can be replaced or no
be ordered or replacement.
unordered
Unordered

With replacement

10
 Counting – ordered sampling with replacement

Ordered sampling k elements from a set of n

elements with replacement.
No. of possible arrangements = nk
Eg: Calculate the total number of different 4-
digit numeric codes. Each digit is independently
selected from {0,1,2,…,9}.
1st digit: 10 possible numbers
2nd digit: also 10 possible numbers.
3rd and 4th digit: each also 10 possible numbers.
Total arrangements = 10 x 10 x 10 x 10 = 104
11
 Counting – ordered sampling without replacement

Ordered sampling k elements from a set of n

elements without replacement.

No. of possible arrangements =

n x (n – 1) x (n – 2) x (n – 3) x … x (n – k +1)
Number of k permutations of n elements:
nP = 𝑛𝑛!
k 𝑛𝑛−𝑘𝑘 !

12
Eg: Calculate the total number of different 4-digit
numeric codes. Each digit is sampled from
{0,1,2,…,9} without replacement.

1st digit: 10 possible numbers

2nd digit: 9 possible numbers

3rd digit: 8 possible numbers and so on.

Total arrangements = 10 x 9 x 8 x 7
10P 𝑛𝑛! 10!
i.e. there are 4 = = arrangements
𝑛𝑛−𝑘𝑘 ! 10−4 !

13
Eg: 4 boys and 2 girls sit in a row. Find the following:

(i) No. of ways of putting these 6 people in a row.

(ii) No. of ways such that each girl has a boy to her
left and to her right.

14
 Counting – unordered sampling without replacement

We choose k elements from a set of n elements

and the ordering does not matter.
𝑛𝑛
Number of arrangements equals to k Note:
𝑘𝑘
combinations of n elements =
𝑛𝑛!
𝑘𝑘! 𝑛𝑛−𝑘𝑘 !
𝑛𝑛 𝑛𝑛! 𝑛𝑛
= = =
𝑘𝑘 𝑘𝑘! 𝑛𝑛−𝑘𝑘 ! 𝑛𝑛 − 𝑘𝑘
Eg: How many combinations of two numbers
between 1 and 6 are there:
6! 6×5
Ans: = = 15
2! 6−2 ! 2 × 1
15
 Counting – unordered sampling with replacement

Pick k elements from a set of n elements, one at

a time with replacement and the ordering does
not matter.
Eg: Pick two numbers from {1, 2, 3}, we have
6 arrangements with n=3 and k=2.
i.e. (1,1), (1,2), (1,3), (2,2), (2,3) and (3,3)

In general:
𝑛𝑛 + 𝑘𝑘 − 1
No. of arrangements =
𝑘𝑘
16
Eg: A 6-bit code is made up of 4 1’s and 2 0’s.
Calculate:

(i) the no. of distinct codewords in the code?

(ii) the no. of codewords such that a ‘0’ has a ‘1’ to

its left and to its right.

17
 Probability Theory
In a random experiment, the outcome of the
experiment occurs with a certain probability.
Eg: We toss a coin. If the coin is unbiased,
then the chances of getting a Head is 50%.
Eg: We roll a fair die two times. The outcome
∈ {11, 12, 13, 14, 15, 16, 21, 22, … , 65, 66}.
Total of 36 possible outcomes.
Probability of getting a “1” followed by
another “1” is 1/36.
Probability of getting 1st no. = 2nd no. is 6/36.
18
Definitions
 A sample space is the set S of all possible
outcomes of an experiment.
 An event is a set of one or more (favorable)
outcomes in the sample space.
 Two events are mutually exclusive if they have no
outcomes in common.
 They are exhaustive if they cover all possible
outcomes.
 Two events are independent if the probability that
one occurs is not affected by whether or not the
other has occurred.
19
Ch 5. Probability

● Basic Concepts – Review of Set Theory

and Venn Diagram
● Counting – Ordered, Unordered, Sampling
With and Without Replacement.
● Probability Theory
● Base Rate: Bayes’ Theorem

20
Axioms of Probability:
(1) Probability of any outcome or event X is a non-
negative:
P(X) ≥ 0

(2) Probability of the sample space S is 1:

P(S) = 1

(3) If X1, X2, X3, … are mutually exclusive events,

then:

P(X1 or X2 or X3 …) = P(X1) + P(X2) + P(X3) + …

21
Probabilities and Events (mutually exclusive)
Eg: A company has decided that in the next 5 years,
40% of their new employees will be men, 30% will
be Singaporean, and 35% will be foreigner women.
What percentage of new employees will be
Singaporean men?

Men Women Total

Foreigner 0.35
0.3 − 0.25 0.6 − 0.35
Singaporean =0.05 =0.25
0.3
1 − 0.4
Total 0.4
=0.6
22
Probabilities and Events (non-mutually exclusive)
Eg: A certain kind of fruit is grown in 2 districts, A and
B. Both areas sometimes get fruitflies. Suppose the
probabilities are P(A)=0.1, P(B)=0.05 and P(A and
B)=0.02, what is the probability that one or other
(or both) districts are infected at a given time?
A and B

A B

P(A or B) = P(A) + P(B) – P(A and B)

= 0.1 + 0.05 – 0.02
= 0.13
23
 Probabilities and Events (independent)
Eg: A company has 2 guards. Each carries a pager
activated by sensors. Guard 1 and guard 2 respond to
pager alert 80% and 50% of the time respectively.
They independently report any alert. What is the
probability that at least one will report an alert?

24
Worked Examples considered so far:

Probabilities and Events (mutually exclusive)

Probabilities and Events (non-mutually exclusive)

Probabilities and Events (independent)

Next: Consider Conditional Probabilities

i.e. Probability of an event A given that an event B

has occurred, denoted as P(A|B)

25
Conditional Probabilities

Assuming that 10 percent of the days are rainy in a

certain city.
𝑃𝑃 rain = 0.1

The probability that it rains given that it is cloudy might

be, say,
𝑃𝑃 rain cloudy) = 0.8

This is known as conditional probability: the probability

of event A given that event B has occurred.

𝑃𝑃(𝐴𝐴 ∩ 𝐵𝐵)
𝑃𝑃 𝐴𝐴 𝐵𝐵 = , where 𝐵𝐵 > 0
𝑃𝑃(𝐵𝐵)
26
Conditional Probabilities
Given P(A)=0.5 and P(B)=0.8, determine the following.

(1) P(A|B) for Fig 1.

(2) P(A|B) for Fig 2.

(3) P(B|A) for Fig 2.

27
Conditional Probabilities
Eg: A jar contains 10 blue, 5 red, 4 green and 1 yellow
marbles. Two marbles are randomly picked. What
is the probability that one will be blue and the
other yellow?

28
Illustration using a tree diagram:

29
Bayes’ Theorem
Given two events A and B, where P(A) > 0, we have:

𝑃𝑃(𝐴𝐴 ∩ 𝐵𝐵) 𝑃𝑃 𝐴𝐴 𝐵𝐵 𝑃𝑃(𝐵𝐵)

𝑃𝑃 𝐵𝐵 𝐴𝐴 = =
𝑃𝑃(𝐴𝐴) 𝑃𝑃(𝐴𝐴)
Note: 𝑃𝑃 𝐴𝐴 = 𝑃𝑃 𝐴𝐴 ∩ 𝐵𝐵 + 𝑃𝑃(𝐴𝐴 ∩ 𝐵𝐵′ )

B A

Similarly:
𝑃𝑃(𝐴𝐴 ∩ 𝐵𝐵) 𝑃𝑃 𝐵𝐵 𝐴𝐴 𝑃𝑃(𝐴𝐴)
𝑃𝑃 𝐴𝐴 𝐵𝐵 = =
𝑃𝑃(𝐵𝐵) 𝑃𝑃(𝐵𝐵)

30
Bayes’ Theorem
Eg: A test correctly identifies a disease in 95% of people
who have it. It correctly identifies no disease in 94%
of people who do not have it. In the population, 3%
of the people have the disease. What is the
probability that one has the disease if tested positive?

31
 Summary For Probability Calculations
Are X and Y mutually exclusive?

Yes No

1. P(X and Y) = 0 because event 1. P(X and Y) > 0

(X and Y) is impossible
2. P(X or Y) = P(X) + P(Y) – P(X and Y)
2. P(X or Y) = P(X) + P(Y)

Are X and Y independent?

Yes No

1. P(X|Y) = P(X)
1. P(X|Y) = P(X and Y)
P(Y)
2. P(X and Y) = P(X) P(Y)
2. P(X and Y) = P(X|Y) P(Y)
= P(Y|X) P(X)
32