[go: up one dir, main page]
Scribd
Upload
246 views16 pages

AlphaGo's AI Training and Achievements

The document discusses how Go is difficult for computers to play due to the huge search space and the difficulty of evaluating positions. It describes how AlphaGo uses neural networks trained via supervised learning on human expert games and reinforcement learning from self-play to develop strong evaluation and move selection policies. AlphaGo defeated top professional players Lee Sedol and Fan Hui in 2016, showing computers can now play Go at the highest level.

Uploaded by

Daniel Andrés Crespo
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
246 views16 pages

AlphaGo's AI Training and Achievements

The document discusses how Go is difficult for computers to play due to the huge search space and the difficulty of evaluating positions. It describes how AlphaGo uses neural networks trained via supervised learning on human expert games and reinforcement learning from self-play to develop strong evaluation and move selection policies. AlphaGo defeated top professional players Lee Sedol and Fan Hui in 2016, showing computers can now play Go at the highest level.

Uploaded by

Daniel Andrés Crespo
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 16

Why is Go hard for computers to play?

Game tree complexity = bd

Brute force search intractable:

1. Search space is huge


2. Impossible for computers
to evaluate who is winning
Convolutional neural network
Value network
Evaluation

v (s)

Position
Policy network
Move probabilities

p (a|s)

Position
Neural network training pipeline

Human expert Supervised Learning Reinforcement Learning Self-play data Value network
positions policy network policy network
Supervised learning of policy networks
Policy network: 12 layer convolutional neural network

Training data: 30M positions from human expert games (KGS 5+ dan)

Training algorithm: maximise likelihood by stochastic gradient descent

Training time: 4 weeks on 50 GPUs using Google Cloud

Results: 57% accuracy on held out test data (state-of-the art was 44%)
Reinforcement learning of policy networks
Policy network: 12 layer convolutional neural network

Training data: games of self-play between policy network

Training algorithm: maximise wins z by policy gradient reinforcement learning

Training time: 1 week on 50 GPUs using Google Cloud

Results: 80% vs supervised learning. Raw network ~3 amateur dan.


Reinforcement learning of value networks
Value network: 12 layer convolutional neural network

Training data: 30 million games of self-play

Training algorithm: minimise MSE by stochastic gradient descent

Training time: 1 week on 50 GPUs using Google Cloud

Results: First strong position evaluation function - previously thought impossible


Exhaustive search
Reducing depth with value network
Reducing breadth with policy network
Professional Amateur Beginner
dan (p) dan (d) kyu (k)
Evaluating AlphaGo against computers

9d

7d

5d

3d

1d
9p
7p
5p
3p
1p

1k
3k
5k
7k

Gnu
Go
Fuego
Pachi
Zen
Crazy Stone
AlphaGo (Nature v13)
AlphaGo (Seoul v18)

0
4500

4000

3500

3000

2500

2000

1500

1000

500
Computer Programs Calibration Human Players

DeepMind challenge match Lee Sedol (9p)


AlphaGo (Mar 2016) Top player of
4-1 past decade

Beats Beats

Nature match Fan Hui (2p)


AlphaGo (Oct 2015) 3-times reigning
5-0 Euro Champion

Beats Beats

KGS Amateur
Crazy Stone and Zen
humans
Whats Next?
Demis Hassabis

You might also like