llama_cpp server: add missing top_k param to CreateChatCompletionRequest

Stonelinks · Stonelinks · commit a5aa6c1478de · 2023-05-01T15:38:19.000-07:00
`llama.create_chat_completion` definitely has a `top_k` argument, but its missing from `CreateChatCompletionRequest`. decision: add it
diff --git a/llama_cpp/server/app.py b/llama_cpp/server/app.py
@@ -169,6 +169,7 @@ class CreateChatCompletionRequest(BaseModel):
     model: str = model_field
 
     # llama.cpp specific parameters
+    top_k: int = 40,
     repeat_penalty: float = 1.1
 
     class Config:
