8000 Merge branch 'main' of github.com:abetlen/llama_cpp_python into main · zinccat/llama-cpp-python@4cefb70 · GitHub
[go: up one dir, main page]
Skip to content

Commit 4cefb70

Browse files
abetlenabetlen
committed
Merge branch 'main' of github.com:abetlen/llama_cpp_python into main
2 parents 715f98c + 6517900 commit 4cefb70

File tree

5 files changed

+41
-98
lines changed

5 files changed

+41
-98
lines changed

‎README.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ bash Miniforge3-MacOSX-arm64.sh
4141
```
4242
Otherwise, while installing it will build the llama.ccp x86 version which will be 10x slower on Apple Silicon (M1) Mac.
4343

44-
### Installation with OpenBLAS / cuBLAS / CLBlast
44+
### Installation with OpenBLAS / cuBLAS / CLBlast / Metal
4545

4646
`llama.cpp` supports multiple BLAS backends for faster processing.
4747
Use the `FORCE_CMAKE=1` environment variable to force the use of `cmake` and install the pip package for the desired BLAS backend.
@@ -64,6 +64,11 @@ To install with CLBlast, set the `LLAMA_CLBLAST=1` environment variable before i
6464
CMAKE_ARGS="-DLLAMA_CLBLAST=on" FORCE_CMAKE=1 pip install llama-cpp-python
6565
```
6666

67+
To install with Metal (MPS), set the `LLAMA_METAL=on` environment variable before installing:
68+
69+
```bash
70+
CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install llama-cpp-python
71+
```
6772

6873
## High-level API
6974

‎llama_cpp/server/__main__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,5 +46,5 @@
4646
app = create_app(settings=settings)
4747

4848
uvicorn.run(
49-
app, host=os.getenv("HOST", "localhost"), port=int(os.getenv("PORT", 8000))
49+
app, host=os.getenv("HOST", settings.host), port=int(os.getenv("PORT", settings.port))
5050
)

‎llama_cpp/server/app.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,12 @@ class Settings(BaseSettings):
7272
verbose: bool = Field(
7373
default=True, description="Whether to print debug information."
7474
)
75+
host: str = Field(
76+
default="localhost", description="Listen address"
77+
)
78+
port: int = Field(
79+
default=8000, description="Listen port"
80+
)
7581

7682

7783
router = APIRouter()

0 commit comments

Comments
 (0)
0