8000 kv-cache : rework kv_cell by ggerganov · Pull Request #13706 · ggml-org/llama.cpp · GitHub
[go: up one dir, main page]
Skip to content

kv-cache : rework kv_cell #13706

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
May 25, 2025
Merged

kv-cache : rework kv_cell #13706

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
be9558e
kv-cache : rework kv_cell
ggerganov May 22, 2025
71be7e5
kv-cells : use "shift" instead of "delta" consistently
ggerganov May 23, 2025
7b3f12a
llama : add llama_max_parallel_sequences()
ggerganov May 23, 2025
6221dd2
kv-cells : update comments [no ci]
ggerganov May 23, 2025
43b40d3
context : fail upon construction if sequences exceed max value
ggerganov May 23, 2025
f71e737
kv-cells : get_pos() -> pos_get() + comments
ggerganov May 23, 2025
dd394a6
kv-cells : fix tracking of "used" cells
ggerganov May 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
kv-cells : update comments [no ci]
  • Loading branch information
@ggerganov
ggerganov committed May 24, 2025
commit 6221dd292e1219133be35cba69b0ec0f88a73601
6 changes: 5 additions & 1 deletion src/llama-kv-cells.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,12 +194,15 @@ class llama_kv_cells_unified {
return shift[i];
}

// check if a cell is not empty and its position is within [p0, p1)
bool pos_in(uint32_t i, llama_pos p0, llama_pos p1) const {
assert(i < pos.size());

return pos[i] >= p0 && pos[i] < p1;
}

// set the position of an empty cell
// does not modify "has_shift"
// note: call only if the cell is empty
void pos_set(uint32_t i, llama_pos p) {
assert(i < pos.size());
Expand All @@ -210,6 +213,7 @@ class llama_kv_cells_unified {
}

// pos[i] = pos[i] + d
// sets "has_shift" to true
// note: call only if the cell is not empty
bool pos_add(uint32_t i, llama_pos d) {
assert(i < pos.size());
Expand All @@ -233,6 +237,7 @@ class llama_kv_cells_unified {
}

// pos[i] = pos[i] / d
// sets "has_shift" to true
// note: call only if the cell is not empty
void pos_div(uint32_t i, int d) {
assert(i < pos.size());
Expand Down Expand Up @@ -267,7 +272,6 @@ class llama_kv_cells_unified {
//
std::vector<llama_pos> shift;

// TODO: assert n_seq_max <= 64
std::vector<std::bitset<LLAMA_MAX_PARALLEL_SEQUENCES>> seq;
};

0