10000 llama.cpp : split llama_context_params into model and context params by slaren · Pull Request #3301 · ggml-org/llama.cpp · GitHub
[go: up one dir, main page]
Skip to content

llama.cpp : split llama_context_params into model and context params #3301

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Sep 28, 2023
Merged

llama.cpp : split llama_context_params into model and context params #3301

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
cf1f805
llama.cpp : split llama_context_params into model and context params
slaren Sep 21, 2023
39f4afa
fix metal build
slaren Sep 21, 2023
f28e495
fix freq_base/scale default to model value
slaren Sep 21, 2023
96f6dcd
llama-bench : keep the same model between tests when possible
slaren Sep 21, 2023
7f95379
move n_threads to llama_context_params, add n_threads_batch
slaren Sep 21, 2023
d41b53c
fix mpi build
slaren Sep 21, 2023
ceb18e4
remove kv_size(), cuda scratch fixes
slaren Sep 22, 2023
92fb8ab
remove low-vram option
slaren Sep 22, 2023
a6084cc
add n_threads_batch to system info, refactor to get_system_info()
slaren Sep 23, 2023
3469684
add documentation about --threads-batch to the READMEs
slaren Sep 23, 2023
e5afe42
llama-bench fix
slaren Sep 23, 2023
0e9ed7f
main : fix rope freq/scale warning
slaren Sep 23, 2023
8f5b0ea
llama.cpp : add llama_get_model
slaren Sep 26, 2023
65b83f3
Merge remote-tracking branch 'origin/master' into llama-model-params
slaren Sep 28, 2023
5659391
remove duplicated ctx/model functions
slaren Sep 28, 2023
17e841a
cuda : print total VRAM used
slaren Sep 28, 2023
c8a9658
Merge remote-tracking branch 'origin/master' into llama-model-params
slaren Sep 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix metal build
  • Loading branch information
@slaren
slaren committed Sep 21, 2023
commit 39f4afac204fedf44d4f21836203f42182aa3131
6 changes: 3 additions & 3 deletions llama.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6376,7 +6376,7 @@ struct llama_context * llama_new_context_with_model(
llama_token token = llama_token_bos(ctx); // not actually used by llama_build_graph, but required to choose between token and embedding inputs graph
ggml_cgraph * gf = llama_build_graph(*ctx, &token, NULL, n_tokens, n_past);
#ifdef GGML_USE_METAL
if (params.n_gpu_layers > 0) {
if (model->n_gpu_layers > 0) {
ctx->ctx_metal = ggml_metal_init(1);
if (!ctx->ctx_metal) {
LLAMA_LOG_ERROR("%s: ggml_metal_init() failed\n", __func__);
Expand Down Expand Up @@ -6415,13 +6415,13 @@ struct llama_context * llama_new_context_with_model(
}

#ifdef GGML_USE_METAL
if (params.n_gpu_layers > 0) {
if (model->n_gpu_layers > 0) {
// this allocates all Metal resources and memory buffers

void * data_ptr = NULL;
size_t data_size = 0;

if (params.use_mmap) {
if (ctx->model.mapping) {
data_ptr = ctx->model.mapping->addr;
data_size = ctx->model.mapping->size;
} else {
Expand Down
0