Closed
Description
https://github.com/abetlen/llama-cpp-python/blob/main/llama_cpp/llama.py#L139
If llama-cpp tries to load the model that is not currently supported, such as GPT4All-quantized, It will still initialize, but will throw error in future.
(For example, gpt4all-lora-quantized.bin)
llm = Llama(model_path=model_path)
This works normally, but the following, won't work - because model was not loaded and self.ctx is None.
llm("Q: Name the planets in the solar system? A: ", max_tokens=32, stop=["Q:", "\n"], echo=True)
AssertionError Traceback (most recent call last)
Cell In[12], line 1
----> 1 llm("Q: Name the planets in the solar system? A: ", max_tokens=32, stop=["Q:", "\n"], echo=True)
File [~/.python/current/lib/python3.10/site-packages/llama_cpp/llama.py:688](python/current/lib/python3.10/site-packages/llama_cpp/llama.py:688), in Llama.__call__(self, prompt, suffix, max_tokens, temperature, top_p, logprobs, echo, stop, repeat_penalty, top_k, stream)
652 def __call__(
653 self,
654 prompt: str,
(...)
664 stream: bool = False,
665 ) -> Union[Completion, Iterator[CompletionChunk]]:
666 """Generate text from a prompt.
667
668 Args:
(...)
686 Response object containing the generated text.
687 """
--> 688 return self.create_completion(
689 prompt=prompt,
690 suffix=suffix,
691 max_tokens=max_tokens,
692 temperature=temperature,
693 top_p=top_p,
694 logprobs=logprobs,
...
--> 410 assert self.ctx is not None
411 completion_id: str = f"cmpl-{str(uuid.uuid4())}"
412 created: int = int(time.time())Thus I suggest that self.ctx has to be checked right after initialization too.
Metadata
Metadata
Assignees
Labels
No labels