Lecture 1: Introduction to Reinforcement Learning
Lecture 1: Introduction to Reinforcement
Learning
David Silver
Lecture 1: Introduction to Reinforcement Learning
About RL
Many Faces of Reinforcement Learning
Computer Science
Engineering Neuroscience
Machine
Learning
Optimal Reward
Control System
Reinforcement
Learning
Operations Classical/Operant
Research Conditioning
Bounded
Mathematics Psychology
Rationality
Economics
Lecture 1: Introduction to Reinforcement Learning
About RL
Branches of Machine Learning
Supervised Unsupervised
Learning Learning
Machine
Learning
Reinforcement
Learning
Lecture 1: Introduction to Reinforcement Learning
About RL
Characteristics of Reinforcement Learning
What makes reinforcement learning different from other machine
learning paradigms?
There is no supervisor, only a reward signal
Feedback is delayed, not instantaneous
Time really matters (sequential, non i.i.d data)
Agent’s actions affect the subsequent data it receives
Lecture 1: Introduction to Reinforcement Learning
About RL
Examples of Reinforcement Learning
Fly stunt manoeuvres in a helicopter
Defeat the world champion at Backgammon
Manage an investment portfolio
Control a power station
Make a humanoid robot walk
Play many different Atari games better than humans
Lecture 1: Introduction to Reinforcement Learning
The RL Problem
Reward
Rewards
A reward Rt is a scalar feedback signal
Indicates how well agent is doing at step t
The agent’s job is to maximise cumulative reward
Reinforcement learning is based on the reward hypothesis
Definition (Reward Hypothesis)
All goals can be described by the maximisation of expected
cumulative reward
Do you agree with this statement?
Lecture 1: Introduction to Reinforcement Learning
The RL Problem
Reward
Examples of Rewards
Fly stunt manoeuvres in a helicopter
+ve reward for following desired trajectory
−ve reward for crashing
Defeat the world champion at Backgammon
+/−ve reward for winning/losing a game
Manage an investment portfolio
+ve reward for each $ in bank
Control a power station
+ve reward for producing power
−ve reward for exceeding safety thresholds
Make a humanoid robot walk
+ve reward for forward motion
−ve reward for falling over
Play many different Atari games better than humans
+/−ve reward for increasing/decreasing score
Lecture 1: Introduction to Reinforcement Learning
The RL Problem
Reward
Sequential Decision Making
Goal: select actions to maximise total future reward
Actions may have long term consequences
Reward may be delayed
It may be better to sacrifice immediate reward to gain more
long-term reward
Examples:
A financial investment (may take months to mature)
Refuelling a helicopter (might prevent a crash in several hours)
Blocking opponent moves (might help winning chances many
moves from now)
Lecture 1: Introduction to Reinforcement Learning
The RL Problem
Environments
Agent and Environment
observation action
Ot At At each step t the agent:
Executes action At
Receives observation Ot
reward Rt
Receives scalar reward Rt
The environment:
Receives action At
Emits observation Ot+1
Emits scalar reward Rt+1
t increments at env. step
Lecture 1: Introduction to Reinforcement Learning
Inside An RL Agent
Major Components of an RL Agent
An RL agent may include one or more of these components:
Policy: agent’s behaviour function
Value function: how good is each state and/or action
Model: agent’s representation of the environment
Lecture 1: Introduction to Reinforcement Learning
Inside An RL Agent
Policy
A policy is the agent’s behaviour
It is a map from state to action, e.g.
Deterministic policy: a = π(s)
Stochastic policy: π(a|s) = P[At = a|St = s]
Lecture 1: Introduction to Reinforcement Learning
Inside An RL Agent
Value Function
Value function is a prediction of future reward
Used to evaluate the goodness/badness of states
And therefore to select between actions, e.g.
vπ (s) = Eπ Rt+1 + γRt+2 + γ 2 Rt+3 + ... | St = s
Lecture 1: Introduction to Reinforcement Learning
Inside An RL Agent
Model
A model predicts what the environment will do next
P predicts the next state
R predicts the next (immediate) reward, e.g.
a 0
Pss 0 = P[St+1 = s | St = s, At = a]
Ras = E [Rt+1 | St = s, At = a]
Lecture 1: Introduction to Reinforcement Learning
Inside An RL Agent
Categorizing RL agents (1)
Value Based
No Policy (Implicit)
Value Function
Policy Based
Policy
No Value Function
Actor Critic
Policy
Value Function
Lecture 1: Introduction to Reinforcement Learning
Inside An RL Agent
Categorizing RL agents (2)
Model Free
Policy and/or Value Function
No Model
Model Based
Policy and/or Value Function
Model
Lecture 1: Introduction to Reinforcement Learning
Inside An RL Agent
RL Agent Taxonomy
Model-Free
Value Function Actor Policy
Critic
Value-Based Policy-Based
Model-Based
Model
Lecture 1: Introduction to Reinforcement Learning
Problems within RL
Exploration and Exploitation (1)
Reinforcement learning is like trial-and-error learning
The agent should discover a good policy
From its experiences of the environment
Without losing too much reward along the way
Lecture 1: Introduction to Reinforcement Learning
Problems within RL
Exploration and Exploitation (2)
Exploration finds more information about the environment
Exploitation exploits known information to maximise reward
It is usually important to explore as well as exploit
Lecture 1: Introduction to Reinforcement Learning
Problems within RL
Examples
Restaurant Selection
Exploitation Go to your favourite restaurant
Exploration Try a new restaurant
Online Banner Advertisements
Exploitation Show the most successful advert
Exploration Show a different advert
Oil Drilling
Exploitation Drill at the best known location
Exploration Drill at a new location
Game Playing
Exploitation Play the move you believe is best
Exploration Play an experimental move
