[go: up one dir, main page]
Scribd
Upload
52 views1 page

(CS 5008) Reinforcement Learning: Assignment 3: t+1 T t+1 T 0 0 0 4

This document contains 6 questions related to Markov chains and reinforcement learning. The questions cover: 1) calculating the distribution after 4 time steps of a random walk on integers with 0.5 probability of moving left or right, 2) determining the number of states and transition matrix for a queue with maximum length 9, 3) verifying that the distribution at time t+1 is also a distribution if the distribution at time t is a distribution, 4) using Bayes' rule to calculate the probability of the initial state given observations, 5) calculating the probability of the state at time t given observations up to time k<t, and 6) working through an example of rain and umbrellas for filtering, prediction, smoothing and maximum likelihood

Uploaded by

Sparsh Jain
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
52 views1 page

(CS 5008) Reinforcement Learning: Assignment 3: t+1 T t+1 T 0 0 0 4

This document contains 6 questions related to Markov chains and reinforcement learning. The questions cover: 1) calculating the distribution after 4 time steps of a random walk on integers with 0.5 probability of moving left or right, 2) determining the number of states and transition matrix for a queue with maximum length 9, 3) verifying that the distribution at time t+1 is also a distribution if the distribution at time t is a distribution, 4) using Bayes' rule to calculate the probability of the initial state given observations, 5) calculating the probability of the state at time t given observations up to time k<t, and 6) working through an example of rain and umbrellas for filtering, prediction, smoothing and maximum likelihood

Uploaded by

Sparsh Jain
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 1

(CS 5008) Reinforcement Learning : Assignment 3

Markov Chain

Q1) Consider a random walk on the set of integer Z described as follow: P rob(st+1 = s + 1|st =
s) = 0.5 and P rob(st+1 = s−1|st = s) = 0.5. Start from various initial distributions µ0 (remember
s0 ∼ µ0 ), and find out µ4 , the distribution after 4 time steps.
Q2) Consider a single queue with maximum length n = 9. What is the total number of states? Let the
queue evolve in discrete time steps t = 0, 1, . . ., and let probability of arrival of a customer between
times t and t + 1 be p, and let the probability that a customer is serviced between t and t + 1 be q.
Also, let arrival and service be independent of each other. Describe the probability transition matrix
for this system.
Q3) We know that µ>
t+1 = µt P. Verify that µt+1 is a distribution if µt is a distribution.
Q4) Consider the filtering problem discussed in the class? Verify that P rob(s0 |o0 , s0 ∼ µ0 ) is
nothing but the Bayes rule.
Q5) Consider the filtering problem discussed in the class? How will we find P rob(st |ok , s0 ∼ µ0 ),
where k < t.
Q6) Please work out the “rain and umbrella" explained in class for i) Filtering ii) Prediction iii)
Smoothing and iv) Maximum likelihood sequence (Viterbi Algorithm). Play around with different
numbers.

You might also like