10000 server : refactor slot input data, move tokenizer to HTTP thread by ngxson · Pull Request #10023 · ggml-org/llama.cpp · GitHub
[go: up one dir, main page]
Skip to content

server : refactor slot input data, move tokenizer to HTTP thread #10023

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Oct 24, 2024
Merged

server : refactor slot input data, move tokenizer to HTTP thread #10023

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
125835b
server : refactor slot input data, move tokenizer to HTTP thread
ngxson Oct 23, 2024
5c749be
move prompt_tokens.empty() check
ngxson Oct 23, 2024
3abc339
Merge branch 'master' into xsn/refactor_server_slot_input
ngxson Oct 23, 2024
60d4194
fix incorrect if branch
ngxson Oct 23, 2024
b550011
fix infinite generation loop
ngxson Oct 23, 2024
cff97ad
bring back infill validation
ngxson Oct 24, 2024
fea5ca4
add infill test
ngxson Oct 24, 2024
07381f7
try fixing format_infill
ngxson Oct 24, 2024
c34ab08
fix test
ngxson Oct 24, 2024
575b133
remove redundant code
ngxson Oct 24, 2024
4a9f3e7
rename completion to inference
ngxson Oct 24, 2024
13ee779
update docs
ngxson Oct 24, 2024
7f7acdb
use llama_tokens everywhere
ngxson Oct 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
remove redundant code
  • Loading branch information
@ngxson
ngxson committed Oct 24, 2024
commit 575b1332ab28c1892f726a6e92a815461bbc4240
10 changes: 1 addition & 9 deletions examples/server/server.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -715,10 +715,6 @@ struct server_context {
metrics.init();
}

std::vector<llama_token> tokenize(const json & json_prompt, bool add_special, bool parse_special) const {
return tokenize_mixed(ctx, json_prompt, add_special, parse_special);
}

server_slot * get_slot_by_id(int id) {
for (server_slot & slot : slots) {
if (slot.id == id) {
Expand Down Expand Up @@ -1352,10 +1348,6 @@ struct server_context {
std::vector<server_task> create_tasks_cmpl(json data, server_task_cmpl_type cmpl_type) {
std::vector<server_task> tasks;
auto create_task = [&](json & task_data, llama_tokens & prompt_tokens) {
if (prompt_tokens.empty()) {
// TODO @ngxson : should not throw an error
throw std::runtime_error("prompt must not be empty");
}
SRV_DBG("create task, n_tokens = %d\n", (int) prompt_tokens.size());
server_task task;
task.id = queue_tasks.get_new_id();
Expand Down Expand Up @@ -2877,7 +2869,7 @@ int main(int argc, char ** argv) {
const bool add_special = json_value(body, "add_special", false);
const bool with_pieces = json_value(body, "with_pieces", false);

std::vector<llama_token> tokens = ctx_server.tokenize(body.at("content"), add_special, true);
std::vector<llama_token> tokens = tokenize_mixed(ctx_server.ctx, body.at("content"), add_special, true);

if (with_pieces) {
for (const auto& token : tokens) {
Expand Down
0