Replies: 2 comments 4 replies
-
|
hi @rampa3 - I'm totally in favor of this and help on this would be much appreciated. If you open up the PR we can work out the details along the way. |
Beta Was this translation helpful? Give feedback.
-
|
Working on the PR. Just figuring out signing off first rn, as all commits in my experimental branch are just commits... |
Beta Was this translation helpful? Give feedback.
-
|
Not sure what is the cleanest way to handle this, since I need to do an extensive rebase if I would want to go back and sign off everything in the history, which would be somewhere around 20 commits I would need to hunt down. Maybe it will be simpler to make yet another branch and merge in all the changes using diff and sign off that. |
Beta Was this translation helpful? Give feedback.
-
|
@mudler Created draft #5990 from a fresh banch just for the PR (kept the original for active experimentation). I feel that doing a non-draft PR with non-working 2 out of 10 backends + another 2 out of the 8 remaining ones untested is not appropriate. I hope that I didn't screwed up anything. This time I kept in mind to sign off everything. |
Beta Was this translation helpful? Give feedback.
-
|
Damn, realized I screwed up the commit names. Should I reset the branch and redo them? |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for being a bit silent, start of master's is hitting harder than I thought it will. Sending this to make sure that no one thinks I just fully forgot about this! |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I have been experimenting with producing CPU-only builds of the Python LocalAI backends using CPU only version of Torch (
+cpu) from Torch's repository (--extra-index-url https://download.pytorch.org/whl/cputo prevent pulling CUDA dependences into CPU builds as Torch from PyPi does, since it is just a copy of the CUDA Torch), and in my opinion, at least some of them might be worth to have built officially. Don't want just outright try to work on a PR for something no one but me wants, so I am opening this discussion for it first.So far, I have these results at producing pure-CPU builds: (exllama2 is omitted, as it is a CUDA-only backend)
rfdetr-basemodel from gallery is not valid (has official build, but it is being built against CUDA Torch from PyPi - the image is very big for no reason for CPU usage)For the non-working ones or untested ones, I plan to return to over the time when I have a while, attempting to get them all working.
If anyone else would want to also attempt working on these, my experimental (and probably a bit messy) CPU branch is available here (might not be always up to date): https://github.com/rampa3/LocalAI/tree/python_backends_cpu_build_tweaks
I look forward to any further discussions about the possibility of CPU builds of Python backends.
Beta Was this translation helpful? Give feedback.
All reactions