bump memory limits for redis-[cache,store] to 7 GB #373
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Both redis-cache and redis-store have `maxmemory 6gb` in their redis.conf: https://github.com/sourcegraph/infrastructure/blob/main/docker-images/redis-cache/redis.conf#L4 and https://github.com/sourcegraph/infrastructure/blob/3d83a92ce64650f46754e2ac55c3accfe84ab0d7/docker-images/redis-store/redis.conf#L4. According to https://redis.io/topics/lru-cache#eviction-policies, redis will start evicting things from its cache once it hits that limit. However, because our _k8s pod_ memory limits are also set to 6 GB - k8s will kill the redis pods before they have a chance to perform that eviction process. Giving redis one more gigabyte to work with should give it enough room to do its cache eviction without being OOMKilled
daxmc99
approved these changes
May 3, 2021
Contributor
There was a problem hiding this comment.
LGTM
Do we have a CHANGELOG entry for this? And an upgrade notice on k8s and docker?
Member
Contributor
|
I think this is ready to go, besides a comment in the CHANGELOG |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Both redis-cache and redis-store have
maxmemory 6gbin their redis.conf: https://github.com/sourcegraph/infrastructure/blob/main/docker-images/redis-cache/redis.conf#L4 and https://github.com/sourcegraph/infrastructure/blob/3d83a92ce64650f46754e2ac55c3accfe84ab0d7/docker-images/redis-store/redis.conf#L4. According to https://redis.io/topics/lru-cache#eviction-policies, redis will start evicting things from its cache once it hits that limit. However, because our k8s pod memory limits are also set to 6 GB - k8s will kill the redis pods before they have a chance to perform that eviction process.Giving redis one more gigabyte to work with should give it enough room to do its cache eviction without being OOMKilled
Checklist