8000 llama : fix tokenizer by goerch · Pull Request #2315 · ggml-org/llama.cpp · GitHub
[go: up one dir, main page]

Skip to content

llama : fix tokenizer #2315

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 27 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
ac793a2
Fix for #2023
goerch Jul 21, 2023
8c9d1e7
Fix typo
goerch Jul 21, 2023
9f055e3
Add missing include
goerch Jul 22, 2023
bf665cc
Replace VLA with std::vector
goerch Jul 22, 2023
c8ae817
Add possibly missing typename
goerch Jul 22, 2023
94a0ee1
More testing of the tokenizer
goerch Jul 22, 2023
0e74a72
Added whitespace escaping and unescaping
goerch Jul 22, 2023
e6b1a50
Fix for #2310
goerch Jul 23, 2023
dba8369
One more test case...
goerch Jul 23, 2023
b97a505
Fix C linkage for llama_token_to_str
goerch Jul 24, 2023
81fae1d
Fixing llama_token_to_str for the different sentence_piece token types
goerch Jul 24, 2023
281a4b4
Fixing tests
goerch Jul 24, 2023 8000
a0d28b2
Remove comment
goerch Jul 24, 2023
39c9a3b
Added test cases
goerch Jul 24, 2023
fe7508c
Fix review remarks.
goerch Jul 24, 2023
8253a53
Fix test
goerch Jul 24, 2023
e68580f
Remove llama.cpp.h
goerch Jul 25, 2023
3bdf106
Merge branch 'master' into fix-2023
goerch Jul 25, 2023
b4a5461
Resolve merge conflict with grammar stuff.
goerch Jul 25, 2023
de41d5e
Fix static declarations
goerch Jul 26, 2023
30a0e4c
Fixing function ordering issue
goerch Aug 6, 2023
1b54429
Fix tokenizer regression in convert.py and improve CPP interface for …
goerch Aug 6, 2023
19e950f
Adding support for Aquila (GPT2?) tokenizer.
goerch Aug 6, 2023
bb6a58d
Simplifying an expression.
goerch Aug 6, 2023
5d52192
Remove inactive code.
goerch Aug 6, 2023
38fbb74
Merge branch 'master' into fix-2023
goerch Aug 7, 2023
f1f85de
Split BPE and SentencePiece vocabularies
goerch Aug 8, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
One more test case...
  • Loading branch information
goerch committed Jul 23, 2023
commit dba8369a39bc612e4ab6e25e76e28014f0ec6f4a
3 changes: 2 additions & 1 deletion tests/test-tokenizer-0.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,10 @@ static std::string unescape_whitespace(llama_context* ctx, const llama_token* to
static const std::map<std::string, std::vector<llama_token>> & k_tests()
{
static std::map<std::string, std::vector<llama_token>> _k_tests = {
{" ", {1, 259,},},
{ " ", {1, 259, }, },
{ "\t", { 1, 29871, 12, }, },
{ "\n", { 1, 29871, 13, }, },
{ "\t\n", { 1, 29871, 12, 13, }, },
{ "Hello world", { 1, 15043, 3186, }, },
{ " Hello world", { 1, 29871, 15043, 3186, }, },
{ "Hello World", { 1, 15043, 2787, }, },
Expand Down
0