Closed
Description
Hi, thanks for these very useful bindings! It's made playing with llama.cpp much easier :-)
I noticed that the current implementation of the logprobs option in the completion API uses Llama.logit_to_logprob to convert the logits reported by llama.cpp into logprobs. This sends each logit
def logit_to_logprob(x: float) -> float:
return math.log(1.0 + math.exp(x))However, because the logits produced by LLaMA parameterize a Categorical distribution, I believe we must take their softmax to get the correct logprob: we should map each logit exp of the entire vector of logprobs, and then renormalize so they sum to 1, before taking their elementwise log.
Metadata
Metadata
Assignees
Labels
No labels