context : allow cache-less context for embeddings #13108
Conversation
58115a2 to
7e79a42
Compare
|
I'll work on rebasing and merging this next - it should be a good improvement for embedding models by reducing the allocated memory during inference. |
4f0ea9b to
c14ee72
Compare
|
Thanks for your awesome work Georgi. |
| // extract the rerank score - a single float per sequence | ||
| auto & embd_seq_out = embd_seq; | ||
|
|
||
| for (uint32_t s = 0; s < ubatch.n_seqs; ++s) { | ||
| const llama_seq_id seq_id = ubatch.seq_id[s][0]; | ||
| if (embd_seq_out.find(seq_id) != embd_seq_out.end()) { | ||
| continue; | ||
| } | ||
| embd_seq_out[seq_id].resize(1); | ||
| ggml_backend_tensor_get_async(backend_embd, t_embd, embd_seq_out[seq_id].data(), (seq_id)*sizeof(float), sizeof(float)); | ||
| } | ||
| } break; |
There was a problem hiding this comment.
@ggerganov shouldn't you include a documentation change stating that rank pooling is now supported?
There was a problem hiding this comment.
It was already supported before this change when using llama_decode(). Now llama_encode() also supports it. Not sure we need to document - do you have something in mind where to add docs about this?
* origin/master: (39 commits) server : vision support via libmtmd (ggml-org#12898) sycl : implementation of reordered Q4_0 MMVQ for Intel GPUs (ggml-org#12858) metal : optimize MoE for large batches (ggml-org#13388) CUDA: FA support for Deepseek (Ampere or newer) (ggml-org#13306) llama : do not crash if there is no CPU backend (ggml-org#13395) CUDA: fix crash on large batch size for MoE models (ggml-org#13384) imatrix : Add --parse-special for enabling parsing of special tokens in imatrix calculation (ggml-org#13389) llama-run: add support for downloading models from ModelScope (ggml-org#13370) mtmd : fix batch_view for m-rope (ggml-org#13397) llama : one-off chat template fix for Mistral-Small-2503 (ggml-org#13398) rpc : add rpc_msg_set_tensor_hash_req (ggml-org#13353) vulkan: Allow up to 4096 elements for mul_mat_id row_ids (ggml-org#13326) server : (webui) rename has_multimodal --> modalities (ggml-org#13393) ci : limit write permission to only the release step + fixes (ggml-org#13392) mtmd : Expose helper_decode_image_chunk (ggml-org#13366) server : (webui) fix a very small misalignment (ggml-org#13387) server : (webui) revamp the input area, plus many small UI improvements (ggml-org#13365) convert : support rope_scaling type and rope_type (ggml-org#13349) mtmd : fix the calculation of n_tokens for smolvlm (ggml-org#13381) context : allow cache-less context for embeddings (ggml-org#13108) ...
|
This PR breaks the ability to request embeddings for typical, decoder-only models with causal attention from the llama.cpp server (using |
|
Does it work correctly if you remove the |
Yes, it does. I wasn't aware that --embedding became optional in #10135. NaNs aside, I don't understand why an e5-mistral (arch=llama) or BiQwen2 (arch=qwen2vl) embedding model with See also #13247. |
|
The main reason is for performance and less memory usage. Non-causal models do not require to have a KV cache to be evaluated and the concept of batching into micro-batches is not relevant for them. So it is better to avoid doing these extra steps that are done in
|
@cebtenzzre Should be fixed with #13797 |
target #12799
There is no need to create a KV cache when using embedding models such as BERT. This saves memory compared to
master.API Changes
llama_encode()method is now the recommended way to compute embeddings and rerank.llama_decode()can still be used to compute embeddings as before.llama_decode()fallbacks tollama_encode()and prints a warning.In short, whenever the KV cache is not needed - use
llama_encode(). Otherwise - usellama_decode(). The changes are backwards compatible.