falcon arch fix for tied output embeddings #4978
Conversation
|
One thing is strange though, not super simple to debug. That's despite the openbuddy variant being a bit smaller in total size. |
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
|
|
Thanks, I didn't think about that! Performance now is at 35tokens/sec ! |
|
Hm cool! Will wait for slaren to confirm this is fine before merging |
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
|
Does this mean that we can revive #3626? |
llama.cpp
Outdated
| } else { | ||
| model.output = ml.create_tensor(ctx_output_split, tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}); // needs to be on GPU | ||
| ml.n_tensors++; // artificial tensor | ||
| } |
There was a problem hiding this comment.
This should also work, but I would prefer decreasing ml.n_created instead of increasing ml.n_tensors.
There was a problem hiding this comment.
I was thinking that given we actually have one more tensor in use than in the model file it's better to increase that var. But we can change it to ml.n_created--;
Should I change it ? you can do it as well of course:)
There was a problem hiding this comment.
Let's change it as suggested and merge then
There was a problem hiding this comment.
changed and tested it
|
@cebtenzzre yep, this should be doable now. |
* falcon arch fix for tied output embeddings * Update llama.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update llama.cpp * Update llama.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update llama.cpp --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* falcon arch fix for tied output embeddings * Update llama.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update llama.cpp * Update llama.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * Update llama.cpp --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
The latest falcon finetune from Openbuddy is merging intput/output tensors, it was typically separate before as
lm_head.Here it is: https://huggingface.co/OpenBuddy/openbuddy-falcon-40b-v16.1-4k
This PR uses input embeddings if no output ones are available.
Tested on wizard 40 (separate) and and openbuddy 40 (shared) and both work.