[go: up one dir, main page]
Extrapolating the spectacular performance of GPT3 into the future suggests that the answer to life, the universe and everything is just 4.398 trillion parameters.

Jun 10, 2020 · 8:26 PM UTC

67
660
80
3,765
Replying to @geoffreyhinton
could convolving dense layers across the masked-self-attention input layers (and hidden input layers) in the decoders help keep GPTNs weight count in check, while allowing for larger context windows? similarly: convolving masked self-attention across large input context windows
Replying to @geoffreyhinton
Sheesh--when did 42 become 4.398 trillion? Deep #AI hypes up everything!
3
42
Replying to @geoffreyhinton
In light of your noticing this 3 years ago, why the change of heart last month?
1
Replying to @geoffreyhinton
And all this time I thought it was just 42 parameters.
Replying to @geoffreyhinton
@geoffreyhinton we took this theme and ran with it for our write-up on GPT-3, lambdalabs.com/blog/gpt-3/. Your quote had top billing!

GPT-3 A Hitchhiker's Guide

A balanced perspective on OpenAI's GPT-3. We summarize how the A.I. research community is thinking about Open AI's new language model.

lambdalabs.com
1
Replying to @geoffreyhinton
Sounds like a comfy way to Utopia. How many training tokens, though?
1
Replying to @geoffreyhinton @renormalized
Before or after distillation?
14
Replying to @geoffreyhinton @stanfordnlp
GPT-4, it seems, will have slightly higher computational requirements than GPT-3.
1
Replying to @geoffreyhinton
my calculations estimated 4.399 trillion.. I guess we are close enough and run with 4.39.. ;)
Load more