ECE 603
Probability and Random
Processes
Lessons 1-3
Chapter 1
Basic Concepts
© 2020 UMass Amherst Global. All rights reserved.
Objectives
• Review set theory
• Explore random experiments and probabilities
• Examine conditional probability
• Examine independence
• Review law of total probability
• Review Bayes’ rule
• Explore conditional independence
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
2
Rationale
• An exploration of the basic concepts of probability will provide a
foundation for discussion.
• Mathematical concepts are necessary for understanding probability
theory.
• This lesson focuses on random experiments and the axioms of probability.
• You will explore discrete and continuous probability models, before
discussing conditional probability.
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
3
Prior Learning
• A level of mathematical maturity consistent with the graduate engineering
level.
• Access to the online textbook: https://www.probabilitycourse.com/
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
4
Review of Set Theory
A set is an unordered collection of things (elements).
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
5
Review of Set Theory
• The set of natural numbers,
• The set of integers,
• The set of real number
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
6
Review of Set Theory
Set is a subset of set if every element of is also an element of
We write where " " indicates "subset ".
Example:
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
7
Review of Set Theory
if and only if and
Example:
Universal set: The set of all things that we could possibly consider in a given
context.
Universal set
Null set ;
For any set
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
8
Venn Diagrams
In a Venn diagram any set is depicted by a closed surface.
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
9
Set Operations
Union: The union of two sets and is denoted by and consist of
all objects in or
if and only if or
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
10
Set Operations
Intersection:
The intersection of two sets and is denoted by and consist of all
objects in both and
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
11
Set Operations
Complement:
The complement of a set denoted by is the set of all elements in
that are Not in
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
12
Set Operations
Difference (subtraction):
The subtraction of set from is all elements in that are not
in
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
13
Set Operations
Mutually exclusive set (disjoint):
Two sets and are mutually exclusive (or disjoint) if
are m.e. if
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
14
Set Operations
Partition:
A collection of sets is a Partition of if
a) They are disjoint .
b)
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
15
Set Operations
Theorem : De Morgan's law
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
16
Set Operations
Example:
Let and
a) b) c)
d) f) g)
The sets form a partition of
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
17
Set Operations
Theorem : Distributive law
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
18
Functions
Domain
Co-domain
Range: the set of all the possible values of (Range )
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
19
Functions
Example:
Consider the function defined as
one-to-one (invertible):
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
20
Countable and Uncountable Sets
Cardinality of a set is the number of elements in
set is finite if
set if countable if it is finite Or the elements of can be enumerated or
listed in a sequence that is,
Ex: is countable.
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
21
Countable and Uncountable Sets
Uncountable: Not countable.
e.g.,
Equivalently: set is countably infinite if it is in one-to-one correspondence
with
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
22
Countable and Uncountable Sets
Example:
(set of integers) is countable (countably infinite).
Because
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
23
Countable and Uncountable Sets
Example:
Show that a set of the form is countable.
Example:
Show that the positive rational number form a countable set:
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
24
Countable and Uncountable Sets
Example:
Show that the positive rational number form a countable set:
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
25
Countable and Uncountable Sets
But is not countable.
In fact, any interval where is not countable.
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
26
Orchestrated Conversation:
Review and Discussion
Review of Video and Exercises from Lesson 1 Video 1
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
27
Random experiment
Random experiment: A phenomenon whose outcome cannot be
predicted with certainty, such as
Random experiment:
• Roll a die
• Roll a die three times
• Flip a coin
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
28
Random experiment
Outcome:
An outcome is the result of a random experiment.
• Roll a die 3
• Roll a die 3 times (2, 3, 6)
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
29
Random experiment
Events:
An event is collection of possible outcomes.
• Roll a die (Event=E)
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
30
Random experiment
Sample Space:
The sample space is the set of all possible outcomes.
• Roll a die: random experiment
• Roll a die three times
an outcome
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
31
Random experiment
Event Set
Sample space Universal Set
Outcome Element
We say that an event occurs if the outcome of the experiment is an
element of
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
32
Random experiment
Partition:
A partition is a collectively exhaustive, and mutually exclusive set of events, i.e.,
is a Partition if
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
33
Summary of Random experiment
a) Review of set theory
b) Random experiments: Roll a die, etc.
Outcome: An outcome is a result of random experiment.
• Roll a die 3
• Roll a die three times (3,6,2)
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
34
Summary of Random experiment
Sample Space: The set of all possible outcomes
• Roll a die
Event: An event is a collection of possible outcomes.
An event is subset of
• Roll a die :
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
35
Summary of Random experiment
We also say that an event has occurred if the outcome of the experiment is an
element of
• Roll a die 2,
• Roll a die 3 times
elements.
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
36
Probability
Event Probability of
We assign a probability to every event
The portion of times event is observed in a large number of runs of
the experiment.
events
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
37
Probability
Axioms of Probability
Definition. A probability measure is a function that maps events in the
sample space to real numbers. Such that:
1) For any event
2) Probability of the sample space is
3) For any countable collection of disjoint events
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
38
Orchestrated Conversation:
Review and Discussion
Review of Video and Exercises from Lesson 1 Video 2
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
39
Probability
Roll a fair die (fair: outcomes are equally likely).
disjoint
3rd axiom:
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
40
Probability
disjoint
Equally likely outcomes:
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
41
Probability
Using the axioms:
1) what is
disjoint
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
42
Probability
2)
empty
3)
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
43
Probability
4)
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
44
Probability
5)
Use Venn diagram.
6)
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
45
Probability
Example:
Roll a die twice and observed and
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
46
Orchestrated Conversation:
Review and Discussion
Review of Video and Exercises from Lesson 1 Video 3
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
47
Whiteboard
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
48
Summary
Probability:
Finite Sample Space with equally likely outcomes:
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
49
Sample Space
Sample Space:
a) Countable:
Discrete Probability Space
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
50
Sample Space
b) Uncountable
Continuous Probability Space
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
51
Continuous Probability Space
Example: I choose a point completely at random in
a)
b)
c)
d)
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
52
Continuous Probability Space
Key point: Axioms of Probability applies to continuous probability spaces.
Example: Suppose we know that the probability that a certain machine lasts
more than or equal to years is :
Find the following sets:
a)
b)
c)
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
53
Conditional Probability
Suppose that in a certain city, 0.3 percent of the days are rainy.
The probability that it rains given that it is cloudy might be:
Conditional probability: The probability given
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
54
Conditional Probability
Example. Roll a fair die, what is the probability that the outcome is an even
number given it was less than or equal to 3, i.e.,
Note:
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
55
Conditional Probability
Definition: The Conditional Probability , the probability that
occurred given that has occurred is given by
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
56
Summary of Probability
Axioms of Probability:
a) For any event
b) Probability of the sample space is
c) If are disjoint events, then
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
57
Conditional Probability
If and are two events in a sample space then
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
58
Conditional Probability
Conditional probability satisfies the probability axioms :
a) For any event
b) Conditional probability of given is
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
59
Conditional Probability
c) If are disjoint events, then
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
60
Conditional Probability
Example. Roll two dice
: 3 dots are shown at least on one die
Find
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
61
Conditional Probability
Special cases:
1) and are disjoint:
2)
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
62
Conditional Probability
3)
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
63
Conditional Probability
Example. Roll a die, what is the probability that it is larger than or equal to 5,
given that it is an even number ?
, even number
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
64
Conditional Probability
Definition: Two events and are independent if and only if
equivalently
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
65
Conditional Probability
Warning!
Disjoint (mutually exclusive) Independent
Disjoint:
Independent:
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
66
Conditional Probability
Suppose and are disjoint:
If
If disjoint Not independent.
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
67
Conditional Probability
Example:
independent
Event that there is a connection from node to node
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
68
Conditional Probability
Remark:
1)
2) If are independent, then
a) are independent.
b) are independent.
c) are independent.
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
69
Conditional Probability
Example:
independent
Find
Note that are independent.
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
70
Conditional Probability
If and are two events in a sample space then the conditional
probability of given is defined as
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
71
Independence
Two events and are independent if and only if
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
72
Independence
Three events and are independent if all of the following conditions
hold
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
73
Independence
Example.
Two darts players throw alternately at a board and the first to score a bull wins.
On each of their throws player has probability and player of
success; the result of different throws are independent. If starts, calculate
the probability that he/she wins.
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
74
Law of Total Probability
Let be a partition of the sample space with
For any event we have
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
75
Law of Total Probability
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
76
Orchestrated Conversation:
Law of Total Probability
Example:
Three coins are in a bag:
a) Coin 1: probability of heads is 0.9.
b) Coin 2: probability of heads is 0.6.
c) Coin 3: probability of heads is 0.3.
I draw a coin at random and toss it. What is the probability of heads?
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
77
Bayes' Rule
For any two events and , where , we have
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
78
Bayes' Rule
If is a partition of the sample space , and is any event
with we have
Lesson 61
Chapter © 2020 UMass Amherst Global. All rights reserved.
79
Orchestrated Conversation:
Bayes' Rule
Example.
In the previous problem, suppose that we know the result is heads; what is the
probability that Coin 1 was chosen?
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
80
Bayes' Rule
Example. In a communication system a zero or a one is transmitted with the
probability or
respectively. Due to the noise in the channel, a zero can be received as a one,
with probability and a one can be received as a zero also with probability .
A one was observed, what is the probability that a one was transmitted?
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
81
Conditional Independence
Two events and are independent if and only if
Two events and are conditionally independent given an event if and
only if
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
82
Chain Rule for Conditional Probability
We can extend this to 3 or more events:
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
83
Summary of this Lesson
You explored the basic concepts of probability that will provide a foundation for
discussion of probability throughout this term. You also reviewed mathematical
concepts needed to understand probability theory. You had the opportunity to
examine random experiments and the axioms of probability. Additionally, you
explored discrete and continuous probability models and discussed conditional
probability.
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
84
Post-work for Lessons 1-3
• Complete homework assignment for Lessons 1-3: HW#1
Go to the online classroom for details.
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
85
To Prepare for the Next Lesson
• Read Chapter 2 in your online textbook:
https://www.probabilitycourse.com/chapter2/2_1_0_counting.php
• Complete the Pre-work for Lesson 4.
Visit the online classroom for details.
Chapter 1 © 2020 UMass Amherst Global. All rights reserved.
86
