CUDA: mul_mat_q RDNA2 tunings #2910
Conversation
|
With a 6700 XT using gfx1030 override on the models that I have and that I can fully offload:
|
|
This is with a 5700XT:
|
|
I tested on my 6600 RX. TL;DR is token generation isn't really affected, slight speed up for prompt processing. I tested with a 3B Sorry this isn't very scientific but I did run each test several times to make sure the results were stable. Click to expand3B
|
|
If you use the |
|
My results: Q4_K_M Master:
Master with overrides (LLAMA_CUDA_DMMV_X=64 LLAMA_CUDA_MMV_Y=8 LLAMA_CUDA_KQUANTS_ITER=1):
PR #2910:
Q4_0 Master:
Master with overrides (LLAMA_CUDA_DMMV_X=64 LLAMA_CUDA_MMV_Y=8 LLAMA_CUDA_KQUANTS_ITER=1):
PR #2910:
RX 6650XT |
|
@Ronny-apo Might want to mention your hardware. |
I've added it, thanks! |
|
I tried finding tunings for the RX 5700XT. Most of the best results had edit: tested 128/32 and it was faster for q2&q3
|
|
Device 0: AMD Radeon RX 7900 XTX, compute capability 11.0 master (bd33e5a (1176)):
Cherry-picked commit from this PR:
|
I noticed similar results on my 7900XTX as well. Not much difference from in the tg 128 test but improvement in pp 512 test. Note that 7900xtx has 50% more registers than RDNA2 GPUs, so there's still some scope for some tuning here. Also, I'm not sure if we're using fp16 here or if we're using WMMA yet. Need to dig into the code and see where hipBLAS is being called to I can check its ISA. EDIT: I tried the new gguf model from https://huggingface.co/TheBloke/Llama-2-7B-GGUF/blob/main/llama-2-7b.Q4_0.gguf master:
This PR:
|
6266f14 to
7fbd26a
Compare
|
Thank you for the feedback, everyone. I've integrated the tunings by @Engininja2 and set it up so that my original tunings are used for RDNA 2 and RDNA 3 while the new tunings are used for RDNA 1 and older. |
|
llama-2 7b Q4_0 after the latest changes ggml_init_cublas: found 1 ROCm devices:
build: 7fbd26a (1196) |
ggml-cuda.cu
Outdated
|
|
||
| int mmq_x, mmq_y, nwarps; | ||
| if (compute_capability >= CC_TURING) { | ||
| if (compute_capability >= CC_OFFSET_AMD) { |
There was a problem hiding this comment.
If these tunings don't match with the others set for the mul_mat_q functions earlier then nothing works correctly. Either main freezes or it produces nonsense. Something like this works
if (compute_capability >= CC_RDNA2) {
mmq_x = MMQ_X_Q4_0_RDNA2;
mmq_y = MMQ_Y_Q4_0_RDNA2;
nwarps = NWARPS_Q4_0_RDNA2;
} else if (compute_capability >= CC_OFFSET_AMD) {
mmq_x = MMQ_X_Q4_0_RDNA1;
mmq_y = MMQ_Y_Q4_0_RDNA1;
nwarps = NWARPS_Q4_0_RDNA1;
} else if (compute_capability >= CC_TURING) {Then a couple of macros near the start of the file
#define CC_RDNA2 (CC_OFFSET_AMD + 100*10 + 10*3)
#if defined(__gfx1030__) || defined(__gfx1031__) || defined(__gfx1032__) || defined(__gfx1033__) \
|| defined(__gfx1034__) || defined(__gfx1035__) || defined(__gfx1036__) || defined(__gfx1037__)
#define RDNA2
#endifand the #if for the RDNA2&3 tunings for the mul_mat_q functions could be replaced with #if defined(RDNA2) || defined(__GFX11__)
There was a problem hiding this comment.
Sorry, I forgot to adapt the CPU code. Also what resource are you using to get the GFX values? I didn't know where to look them up so I went by the ones listed in the ROCm documentation supported GPU list.
There was a problem hiding this comment.
LLVM doc should be relatively up-to-date, most of them are not officially supported by ROCm, but people can still build with these architectures.
Alternatively, you can find all of them in LLVM or Mesa code.
7fbd26a to
2b31552
Compare
2b31552 to
d0ef910
Compare
|
Any suggestions on how to best compare performance between this PR and master? I get a lot of variation (not specific to this PR) in average t/s between llama-bench runs with the same build, even if I set repetitions to 30 and control for GPU temperature (preheat it to it's maximum 100C for some time). EDIT: It looks like 40 minutes under maximum load was required to saturate cooling capacity on my laptop to get variation between llama-bench runs to <1%. |
|
On RX 6850M XT (RDNA2, navi22, gfx1031) with gfx1030 override this PR does not affect token generation time (<1% difference), and possibly improves prompt eval time.
|
Co-authored-by: Henri Vasserman <henv@hot.ee>
* CUDA: mul_mat_q RDNA2 tunings * Update ggml-cuda.cu Co-authored-by: Henri Vasserman <henv@hot.ee> --------- Co-authored-by: Henri Vasserman <henv@hot.ee>
* CUDA: mul_mat_q RDNA2 tunings * Update ggml-cuda.cu Co-authored-by: Henri Vasserman <henv@hot.ee> --------- Co-authored-by: Henri Vasserman <henv@hot.ee>
This PR adds mul_mat_q tunings for RDNA 2. They are enabled for all AMD GPUs if HIP is used because I expect AMD architectures to be more similar to each other than RDNA 2 is to NVIDIA - bug please do check whether these tunings work better than master if you have an AMD GPU other than an RX 6800. These are the numbers that I'm getting:
As part of this PR I changed the logic regarding compute capability on AMD: for AMD GPUs the compute capability is offset by 1000000 to separate it from the values that NVIDIA GPUs use (e.g. 860 for compute capability 8.6). Changing
CC_TURINGis now no longer necessary.