Description
it works for me....
You need installed and working (PATH is the main problem):
git
python (i use 3.10.11)
cuda toolkit (i use 11.8)
I have visual studio 2022 community AND Build Tools 2019 installed.
cmake (click on path during installation and restart computer)
Copy the script and save it as: yourname.ps1 into an empty folder
Right click and run it with powershell.
I use the new compiled folder to replace the installed llama_cpp folder in oobabooga.
Processing the first prompt is faster with cublas and larger (512?) prompts. Generating prompt speed is the same.
At the moment the overall cublas speed increase is small for me. The main advantage is rather that you can compile yourself in windows.
If it works for more people, feel free to use the script for the readme.
set-executionpolicy RemoteSigned -Scope CurrentUser
python -m venv venv
venv\Scripts\Activate.ps1
pip install scikit-build
git clone https://github.com/abetlen/llama-cpp-python.git
cd llama-cpp-python
cd vendor
git clone https://github.com/ggerganov/llama.cpp.git
cd ..
$Env:LLAMA_CUBLAS = "1"
$Env:FORCE_CMAKE = "1"
$Env:CMAKE_ARGS="-DLLAMA_CUBLAS=on"
python setup.py install
Write-Host "Done! The llama_cpp folder with the cublas dll is under ..\yourfolder\venv\Lib\site-packages\llama_cpp_python-XXX(vers)-py3.10-win-amd64.egg\llama_cpp "
Write-Host "You can use this folder to replace your old folder"
pause