8000 model : jina-embeddings-v3 support by CISC · Pull Request #13693 · ggml-org/llama.cpp · GitHub
[go: up one dir, main page]
Skip to content

model : jina-embeddings-v3 support #13693

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open

model : jina-embeddings-v3 support #13693

8000
Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
6303ea2
initial jina-embeddings-v3 support
CISC May 21, 2025
ba51f89
initial jina-embeddings-v3 support
CISC May 21, 2025
4ac1380
initial jina-embeddings-v3 support
CISC May 21, 2025
1274c8c
fix vocab parsing with only tokenizer.json
CISC May 22, 2025
65a37fa
set mask token lstrip attribute
CISC May 22, 2025
f2d876a
additional unk_token_id fallback just in case [no ci]
CISC May 23, 2025
b17e981
revert vocab_size() change [no ci]
CISC May 26, 2025
4046701
Merge branch 'master' into cisc/jina-embeddings-v3
CISC Jun 1, 2025
f5d0305
merge tensor loading into general bert
CISC Jun 1, 2025
3862d95
rope
CISC Jun 1, 2025
8ab8d36
Merge branch 'master' into cisc/jina-embeddings-v3
CISC Jul 6, 2025
9a39ccb
add lora embedding and loading (non-functional)
CISC Jul 6, 2025
966d0e0
export separate lora ggufs instead
CISC Jul 6, 2025
bea6d06
add adapter metadata api
CISC Jul 7, 2025
9b67ea2
use std::string
CISC Jul 8, 2025
e092776
convert_hf_to_lora compatibility
CISC Jul 20, 2025
8009403
Merge branch 'master' into cisc/jina-embeddings-v3
CISC Jul 20, 2025
d98c7d9
fix assert
CISC Jul 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
additional unk_token_id fallback just in case [no ci]
  • Loading branch information
@CISC
CISC authored May 23, 2025
commit f2d876ad5ab5013d4df5848038e18e2f8b51760a
2 changes: 1 addition & 1 deletion convert_hf_to_gguf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3689,7 +3689,7 @@ def _xlmroberta_set_vocab(self) -> None:
else:
added_vocab = tokenizer.get_added_vocab()
unk_token = tokenizer_config_json.get("unk_token")
unk_token_id = added_vocab.get(unk_token, 3)
unk_token_id = added_vocab.get(unk_token, tokenizer_json["model"].get("unk_id", 3))

for token_id in range(vocab_size):
piece = tokenizer._convert_id_to_token(token_id)
Expand Down
0