8000 Introduce New Lookup-Table(LUT)-Based Matrix Multiplication Method by QingtaoLi1 · Pull Request #10181 · ggml-org/llama.cpp · GitHub
[go: up one dir, main page]
Skip to content

Introduce New Lookup-Table(LUT)-Based Matrix Multiplication Method #10181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
+1,154 −10
Open

Introduce New Lookup-Table(LUT)-Based Matrix Multiplication Method #10181

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
351e345
Integrate T-MAC kernels
QingtaoLi1 Oct 10, 2024
94502e4
Fix a Cmake variable fault.
QingtaoLi1 Oct 11, 2024
f673699
Remove is_lora in convert_hf_to_gguf, which is removed in master.
QingtaoLi1 Oct 11, 2024
dfac0c4
Remove uint8 branch in gguf_writer.
QingtaoLi1 Oct 11, 2024
f64c768
Restore n_tensor check.
QingtaoLi1 Oct 12, 2024
6bb4aca
Remove unused code.
QingtaoLi1 Oct 12, 2024
b266290
[llama.cpp] update convert_hf_to_gguf.py
kaleid-liner Oct 14, 2024
e86c69d
[Feat] Support TQ1_0 and TQ2_0 with T-MAC.
QingtaoLi1 Oct 23, 2024
080d2ec
Add run_pipeline option of rechunk.
QingtaoLi1 Oct 30, 2024
f84d25d
Limit enable_t_mac to take effect on INT_N only.
QingtaoLi1 Oct 30, 2024
55a8696
[rebase] Fix build error.
QingtaoLi1 Nov 5, 2024
3f7d85d
[fix] Put ggml_tmac_init at correct place.
QingtaoLi1 Nov 6, 2024
cbd8e12
Add back bitdistiller model support.
QingtaoLi1 Dec 25, 2024
c8c2599
[Optimize] Reduce memory usage in non- do_permutate cases.
QingtaoLi1 Dec 27, 2024
628394a
[fix] code style.
QingtaoLi1 Jan 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Limit enable_t_mac to take effect on INT_N only.
  • Loading branch information
@QingtaoLi1
QingtaoLi1 committed Nov 5, 2024
commit f84d25dd8fcf706e357b79ceda1437273d9b76ee
4 changes: 2 additions & 2 deletions convert_hf_to_gguf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1798,8 +1798,8 @@ def modify_tensors(self, data_torch: Tensor, name: str, bid: int | None) -> Iter
]):
# transform weight into 1/0/-1 (in fp32)
data_torch = self.weight_quant(data_torch)
if self.enable_t_mac:
# transform weight into T-MAC I2 format
if self.enable_t_mac and self.ftype == gguf.LlamaFileType.MOSTLY_INT_N:
# transform weight into T-MAC INT_N format
from t_mac.model_utils import preprocess_for_t_mac
data = LazyTorchTensor.to_eager(data_torch).numpy()
scale = np.max(np.abs(data))
Expand Down
0