8000 llama : support qwen3 rerank and embeddings by ngxson · Pull Request #14029 · ggml-org/llama.cpp · GitHub
[go: up one dir, main page]
Skip to content

llama : support qwen3 rerank and embeddings #14029

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
+171 −57
Open

llama : support qwen3 rerank and embeddings #14029

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
3f3b9a2
model : support qwen3 rerank and embeddings
ngxson Jun 5, 2025
f8fd440
Merge branch 'master' into xsn/qwen3_embd_rerank
ngxson Jun 5, 2025
e0eb4b8
correct labels
ngxson Jun 5, 2025
030dc3b
correct output token position
ngxson Jun 6, 2025
f8facb3
Merge branch 'master' into xsn/qwen3_embd_rerank
ngxson Jun 6, 2025
0777cd3
rm redundant settings
ngxson Jun 6, 2025
8edd2cf
add embedding model tokenizer chkhsh
ngxson Jun 6, 2025
c02f53d
add rerank prompt
ngxson Jun 6, 2025
c2f4dc7
fix prompt
ngxson Jun 6, 2025
cbb6f20
fix style
ngxson Jun 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
correct labels
  • Loading branch information
@ngxson
ngxson committed Jun 5, 2025
commit e0eb4b8bfef8989dbb3efd3f61b416bfe9094f1c
2 changes: 1 addition & 1 deletion convert_hf_to_gguf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3095,7 +3095,7 @@ def set_gguf_parameters(self):
if is_rerank:
self.gguf_writer.add_pooling_type(gguf.PoolingType.RANK)
self.gguf_writer.add_sep_token_id(self.sep_token_id)
self.gguf_writer.add_uint32(gguf.Keys.Classifier.OUTPUT_LABELS, 2)
self.gguf_writer.add_classifier_output_labels(["yes", "no"])

def _get_cls_out_tensor(self, data_torch: Tensor) -> Tensor:
# extract "yes" and "no" tokens from the output lm_head tensor
Expand Down
2 changes: 1 addition & 1 deletion src/llama-model.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2470,7 +2470,7 @@ bool llama_model::load_tensors(llama_model_loader & ml) {
tok_embd = create_tensor(tn(LLM_TENSOR_TOKEN_EMBD, "weight"), {n_embd, n_vocab}, 0);

// output rerank
cls_out = create_tensor(tn(LLM_TENSOR_CLS_OUT, "weight"), {n_embd, 2}, TENSOR_NOT_REQUIRED);
cls_out = create_tensor(tn(LLM_TENSOR_CLS_OUT, "weight"), {n_embd, hparams.n_cls_out}, TENSOR_NOT_REQUIRED);

// output
output_norm = create_tensor(tn(LLM_TENSOR_OUTPUT_NORM, "weight"), {n_embd}, 0);
Expand Down
Loading
0