Open
Description
Prerequisites
Please answer the following questions for yourself before submitting an issue.
- [Y] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [Y] I carefully followed the README.md.
- [Y] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- [Y] I reviewed the Discussions, and have a new bug or useful enhancement to share.
Expected Behavior
Please provide a detailed written description of what you were trying to do, and what you expected llama-cpp-python to do.
I tried running a simple hello world embedding query to make sure llama-cpp-python was working after doing a clean install of CentOS 9 with CUDA, Python 3.11.8, VSCode and the works.
Code:
import argparse
from llama_cpp import Llama
llm = Llama(model_path="/home/jfuehne/Desktop/AI/Code/models/llama-2-13b-chat.Q5_K_M.gguf", embedding=True)
print(llm.create_embedding("Hello world!"))
Expected output (this is the output given for this code when I downgrade the llama-cpp-python package to 0.2.55):
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
llama_model_loader: loaded meta data with 19 key-value pairs and 363 tensors from [../llama-2-13b-chat.Q5_K_M.gguf](https://file+.vscode-resource.vscode-cdn.net/../models/llama-2-13b-chat.Q5_K_M.gguf) (version GGUF V2)
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = LLaMA v2
llama_model_loader: - kv 2: llama.context_length u32 = 4096
llama_model_loader: - kv 3: llama.embedding_length u32 = 5120
llama_model_loader: - kv 4: llama.block_count u32 = 40
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 13824
llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 7: llama.attention.head_count u32 = 40
llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 40
llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 10: general.file_type u32 = 17
llama_model_loader: - kv 11: tokenizer.ggml.model str = llama
llama_model_loader: - kv 12: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv 13: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 14: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv 15: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 16: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 17: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 18: general.quantization_version u32 = 2
...
llama_print_timings: sample time = 0.00 ms [/](https://file+.vscode-resource.vscode-cdn.net/) 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: prompt eval time = 386.32 ms [/](https://file+.vscode-resource.vscode-cdn.net/) 4 tokens ( 96.58 ms per token, 10.35 tokens per second)
llama_print_timings: eval time = 0.00 ms [/](https://file+.vscode-resource.vscode-cdn.net/) 1 runs ( 0.00 ms per token, inf tokens per second)
llama_print_timings: total time = 386.32 ms [/](https://file+.vscode-resource.vscode-cdn.net/) 5 tokens
Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?b1ca8dda-082b-43df-886e-52deefa61e39) or open in a [text editor](command:workbench.action.openLargeOutput?b1ca8dda-082b-43df-886e-52deefa61e39). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...
{'object': 'list', 'data': [{'object': 'embedding', 'embedding': [0.01151881321297587, ..., -0.003823969292499485], 'index': 0}], 'model': '../llama-2-13b-chat.Q5_K_M.gguf', 'usage': {'prompt_tokens': 4, 'total_tokens': 4}}
Current Behavior
Please provide a detailed written description of what llama-cpp-python did, instead.
When using 0.2.57 of llama-cpp-python (the version autoinstalled by pip):
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
Cell In[6], line 8
3 from llama_cpp import Llama
6 llm = Llama(model_path="/home/jfuehne/Desktop/AI/Code/models/llama-2-13b-chat.Q5_K_M.gguf", embedding=True)
----> 8 print(llm.create_embedding("Hello world!"))
File ~/Desktop/AI/.venv/lib/python3.11/site-packages/llama_cpp/llama.py:752, in Llama.create_embedding(self, input, model)
750 embeds: List[List[float]]
751 total_tokens: int
--> 752 embeds, total_tokens = self.embed(input, return_count=True) # type: ignore
754 # convert to CreateEmbeddingResponse
755 data: List[Embedding] = [
756 {
757 "object": "embedding",
(...)
761 for idx, emb in enumerate(embeds)
762 ]
File ~/Desktop/AI/.venv/lib/python3.11/site-packages/llama_cpp/llama.py:863, in Llama.embed(self, input, normalize, truncate, return_count)
860 p_batch += 1
862 # hanlde last batch
--> 863 decode_batch(p_batch)
865 if self.verbose:
866 llama_cpp.llama_print_timings(self._ctx.ctx)
...
--> 824 raise RuntimeError("Failed to get embeddings from sequence pooling type is not set")
825 embedding: List[float] = ptr[:n_embd]
826 if normalize:
RuntimeError: Failed to get embeddings from sequence pooling type is not set
Environment and Context
Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.
- Physical (or virtual) hardware you are using, e.g. for Linux:
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 39 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: GenuineIntel
Model name: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
CPU family: 6
Model: 158
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 13
CPU(s) scaling MHz: 16%
CPU max MHz: 5000.0000
CPU min MHz: 800.0000
BogoMIPS: 7200.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss
ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonsto
p_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid
sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpu
id_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust b
mi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm
ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp vnmi md_clear flush_l1d arch_capabilities
Virtualization features:
Virtualization: VT-x
Caches (sum of all):
L1d: 256 KiB (8 instances)
L1i: 256 KiB (8 instances)
L2: 2 MiB (8 instances)
L3: 16 MiB (1 instance)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerabilities:
Gather data sampling: Mitigation; Microcode
Itlb multihit: KVM: Mitigation: VMX disabled
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Retbleed: Mitigation; Enhanced IBRS
Spec rstack overflow: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Enhanced / Automatic IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Srbds: Mitigation; Microcode
Tsx async abort: Mitigation; TSX disabled
- Operating System, e.g. for Linux:
$ uname -a
Linux localhost.localdomain 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
- SDK version, e.g. for Linux:
$ python3.11 --version
Python 3.11.8
$ make --version
GNU Make 4.3
Built for x86_64-redhat-linux-gnu
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ g++ --version
g++ (GCC) 11.4.1 20231218 (Red Hat 11.4.1-3)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Failure Information (for bugs)
Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.
Steps to Reproduce
Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.
install python 3.11.8, with CUDA 12.4 and drivers. sudo install all the required backend modules for python such as:
sudo dnf install gcc make libffi-devel openssl-devel bzip2-devel sqlite-devel readline-devel zlib-devel xz-devel git wget curl python3-venv
Set up a venv in vscode and install required packages (in my scenario, I believe I may have initially allowed llama-cpp-python to install without specifying CMAKE args, not sure if this is the root cause, but I believe this should have only resulted in me running on CPU based default and later reinstalling, right?)
3.
Run the provided HelloWorld embedding query with llama_cpp
Code:
import argparse
from llama_cpp import Llama
llm = Llama(model_path="/home/jfuehne/Desktop/AI/Code/models/llama-2-13b-chat.Q5_K_M.gguf", embedding=True)
print(llm.create_embedding("Hello world!"))