Revisit response compression in k8s API #112296
Description
What would you like to be added?
Based on load tests and production data gathered from thousands of production k8s clusters, we have observed that gzip compression in k8s API isn't optimal today. Here's the full doc with details and the SIG meeting recording where this was discussed.
The consensus is this (please add/correct anything I'm missing):
- [short-term] Reduce the default gzip level from 4 to 1 in kube-apiserver
- [medium-term] Add
DisableCompressionfield to kube-config, for clients to disable compression altogether from their end (and avoid building their own workarounds) - [medium-term] Add
--disable-compressionflag to kubectl for CLI users to easily turn off compression - [long-term] Make apiserver choose gzip compression level for requests dynamically based on heuristics like object-types and compression ratios achieved (@smarterclayton suggested this idea in the past too and we now have data to show its value specifically wrt secrets/configmaps)
- [long-term] Explore support for other compression implementations that are faster / less CPU-intensive. Some options are:
- zstd - https://pkg.go.dev/github.com/klauspost/compress/zstd
- pgzip - https://pkg.go.dev/github.com/klauspost/pgzip
- snappy - https://pkg.go.dev/github.com/golang/snappy
- CGO-based zlib - https://github.com/4kills/go-zlib (pointed by @simonis)
I'll own 1, 2 and 3 above. The rest need further eval - probably a KEP.
Why is this needed?
To improve API call latencies (specifically list calls) as well as CPU/memory usage on the apiserver and client side, by trading off for slightly higher network bandwidth usage. The doc linked above has more details.
/sig api-machinery
/sig scalability