Description
Prerequisites
Please answer the following questions for yourself before submitting an issue.
- I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- I carefully followed the README.md.
- I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- I reviewed the Discussions, and have a new bug or useful enhancement to share.
Expected Behavior
To override values by passing the kv_override parameter as {"tokenizer.ggml.pre": "llama3"}
I expected llama-cpp-python to correctly handle the dictionary with values and override the values properly.
Current Behavior
Instead, I received an error "Received error 'bytes' object does not support item assignment (type=value_error)" when passing a dictionary with string value ("llama3") in this case.
Environment and Context
Python 3.8.1
- Physical (or virtual) hardware you are using, e.g. for Linux:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 1
Core(s) per socket: 16
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6246R CPU @ 3.40GHz
Stepping: 7
CPU MHz: 3999.999
BogoMIPS: 6800.00
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 36608K
NUMA node0 CPU(s): 0-15
NUMA node1 CPU(s): 16-31
- Operating System, e.g. for Linux:
Linux emul318 4.18.0-425.19.2.el8_7.x86_64 #1 SMP Fri Mar 17 01:52:38 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
- SDK version, e.g. for Linux:
$ make --version
GNU Make 4.2.1
Built for x86_64-redhat-linux-gnu
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ g++ --version
g++ (GCC) 8.5.0 20210514 (Red Hat 8.5.0-16)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Failure Information (for bugs)
Steps to Reproduce
- Create a dictionary, e.g., kv_overrides = {"tokenizer.ggml.pre": "llama3"}
- Pass this on to the Llama constructor
Failure Logs
Received error 'bytes' object does not support item assignment (type=value_error)
Findings
From the documentation, I think the error lies in the line
self._kv_overrides_array[i].value.str_ value[:128] = v_bytes
Here, self._kv_oversrides_array[i].value.str_value is a ctypes array of bytes (ctypes.ARRAY(ctypes.c_char, 128)), and you're trying to assign a bytes object to a slice of it.
https://llama-cpp-python.readthedocs.io/en/stable/api-reference/#llama_cpp.llama_cpp.llama_model_kv_override_value
In Python, bytes objects are immutable, so you can't assign to their elements individually. You need to create a new bytes object with the desired values.