8000 falcon arch fix for tied output embeddings by cmp-nct · Pull Request #4978 · ggml-org/llama.cpp · GitHub
[go: up one dir, main page]
Skip to content

falcon arch fix for tied output embeddings #4978

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jan 18, 2024
Merged

falcon arch fix for tied output embeddings #4978

Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
94f330a
falcon arch fix for tied output embeddings
cmp-nct Jan 16, 2024
196348d
Update llama.cpp
cmp-nct Jan 16, 2024
80350da
Update llama.cpp
cmp-nct Jan 16, 2024
a0dee64
Update llama.cpp
cmp-nct Jan 17, 2024
85070cf
Update llama.cpp
cmp-nct Jan 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion llama.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3435,7 +3435,12 @@ static bool llm_load_tensors(
{
model.output_norm = ml.create_tensor(ctx_output, tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd});
model.output_norm_b = ml.create_tensor(ctx_output, tn(LLM_TENSOR_OUTPUT_NORM, "bias"), {n_embd});
model.output = ml.create_tensor(ctx_output_split, tn(LLM_TENSOR_OUTPUT, "weight"), {n_embd, n_vocab});
if (gguf_find_tensor(ml.ctx_gguf, tn(LLM_TENSOR_OUTPUT, "weight").c_str()) >= 0) {
model.output = ml.create_tensor(ctx_output_split, tn(LLM_TENSOR_OUTPUT, "weight"), {n_embd, n_vocab});
} else {
model.output = ml.create_tensor(ctx_output_split, tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}); // needs to be on GPU
ml.n_tensors++; // artificial tensor
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should also work, but I would prefer decreasing ml.n_created instead of increasing ml.n_tensors.

Copy link
Contributor Author
@cmp-nct cmp-nct Jan 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking that given we actually have one more tensor in use than in the model file it's better to increase that var. But we can change it to ml.n_created--;
Should I change it ? you can do it as well of course:)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's change it as suggested and merge then

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed and tested it

}

for (int i = 0; i < n_layer; ++i) {
Expand Down
0