TinyChat support for GQA and memory efficient loading #90
Conversation
… efficient loading.
|
Is it correctly understood that AWQ now solely uses the modified GEMM kernel for processing context? Also, is the modified GEMM kernel faster than the previous one? |
|
Hi Casper, AWQ solely uses the GEMM kernel for context processing. I think its speed is on-par with the previous version, but it has been adapted to work with the latest weight packing scheme. Best, |
Hi Haotian, thank you for answering! I have tested the new GEMM kernel. It is a good improvement over GEMV. It is 2x faster than GEMV but 5-6x slower than the original GEMM kernel. Speed of context:
|
|
Interesting. Is that just for the context stage? We haven't done any formal benchmark yet, but we will definitely work on further improve the speed of this kernel. |
Yes, this is only for the context stage. Let me say this. In general, the new GEMV kernel is 20% faster at token generation than the original GEMM kernel. The only problem is that the context stage can be slow if you want to use 4-8-16k contexts. |
Add implementation from @ys-2020 that allows deployment of LLaMA-2-70B models on Orin-64G. Also add a faster context stage implementation compared with the release last week.