10BC0 SOTA 2-bit quants by ikawrakow · Pull Request #4773 · ggml-org/llama.cpp · GitHub
[go: up one dir, main page]
Skip to content

SOTA 2-bit quants #4773

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ikawrakow merged 17 commits into from
Jan 8, 2024
Merged

SOTA 2-bit quants #4773

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
4af2488
iq2_xxs: basics
Jan 3, 2024
7ef6389
iq2_xxs: scalar and AVX2 dot products
Jan 3, 2024
7b72318
iq2_xxs: ARM_NEON dot product
Jan 3, 2024
d383f00
iq2_xxs: WIP Metal
Jan 3, 2024
dd29610
iq2_xxs: Metal dot product now works
Jan 3, 2024
1c96aa0
iq2_xxs: slighty faster dot product
Jan 3, 2024
e211fad
iq2_xxs: slighty faster dot product
Jan 4, 2024
065cc8c
iq2_xxs: even faster Metal dot product
Jan 4, 2024
06e6908
iq2_xxs: dequantize CUDA kernel - fix conflict with master
Jan 4, 2024
8240521
iq2_xxs: quantized CUDA dot product (MMVQ)
Jan 4, 2024
c19d0d0
iq2_xxs: slightly faster CUDA dot product
Jan 4, 2024
fd42737
iq2_xxs: add to llama ftype enum
Jan 4, 2024
47ae9b8
iq2_xxs: fix MoE on Metal
Jan 6, 2024
61c0405
Fix missing MMQ ops when on hipBLAS
Jan 8, 2024
7db967e
Fix bug in qequantize_row_iq2_xxs
Jan 8, 2024
5684d79
Fixing tests
Jan 8, 2024
bad5f7f
PR suggestion
Jan 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
iq2_xxs: slighty faster dot product 
TG-128 is now 48.4 t/s
  • Loading branch information
Iwan Kawrakow committed Jan 8, 2024
commit 1c96aa0d7fa25f892ca87d5765c86ea231f393ba
10 changes: 5 additions & 5 deletions ggml-metal.metal
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3592,8 +3592,6 @@ void kernel_mul_mv_iq2_xxs_f32_impl(
float yl[32];
float sumf[N_DST]={0.f}, all_sum;

const int step = sizeof(block_q2_K) * nb;

const int nb32 = nb * (QK_K / 32);

#if QK_K == 256
Expand All @@ -3611,11 +3609,12 @@ void kernel_mul_mv_iq2_xxs_f32_impl(
const int ib = ib32 % (QK_K / 32);

device const block_iq2_xxs * xr = x + ibl;
device const uint16_t * q2 = xr->qs + 4 * ib;
device const half * dh = &xr->d;

for (int row = 0; row < N_DST; row++) {

const float db = xr->d;
device const uint16_t * q2 = xr->qs + 4 * ib;
const float db = dh[0];
device const uint8_t * aux8 = (device const uint8_t *)q2;
const uint32_t aux32 = q2[2] | (q2[3] << 16);
const float d = db * (0.5f + (aux32 >> 28));
Expand All @@ -3630,7 +3629,8 @@ void kernel_mul_mv_iq2_xxs_f32_impl(
}
sumf[row] += d * sum;

xr += nb;
dh += nb*sizeof(block_iq2_xxs)/2;
q2 += nb*sizeof(block_iq2_xxs)/2;
}

y4 += 32 * 32;
Expand Down
0