Can cluster slots be cached beyond request lifecycle? #2456

gskema · 2024-03-07T13:04:15Z

gskema
Mar 7, 2024

Hello,
we have ElastiCache cluster, we are using persistent: true argument in RedisCluster constructor + we have enabled redis.clusters.cache_slots = 1
What I noticed is that in every PHP request the first operation to cluster takes twice as long - I assume it's fetch slots + actual operation. The subsequent operations take half the time (only GET operation).

Is it intended that slots are cached per-requests, or can they be configured to be cached for whole PHP-FPM process pool for longer time?
Is it an issue with configuration like missing redis.pconnect.pooling_enabled (I expected this wouldn't apply to RedisCluster)
Does redis.clusters.persistent have any effect when constructor arg is used already?

Thanks

Answered by michael-grunder

Mar 7, 2024

redis.clusters.cache_slots uses PHP's persistent_list mechanism, which survives multiple requests. It will live until we invalidate it (e.g. PhpRedis receives a MOVED response from the cluster). It's important to note however, that each PHP-FPM worker will have it's own cache. So if you have 8 workers, they all need to warm up but at that point, you shouldn't see additional calls to CLUSTER SLOTS.
redis.pconnect.pooling_enabled enables persistent connection pooling. RedisCluster will use the pool if persistent connections are requested in the constructor..
redis.clusters.persistent is just used when connecting to "named" clusters which can be specified in php.ini. To my knowledge this fe…

View full answer

michael-grunder · 2024-03-07T18:33:20Z

michael-grunder
Mar 7, 2024
Maintainer

redis.clusters.cache_slots uses PHP's persistent_list mechanism, which survives multiple requests. It will live until we invalidate it (e.g. PhpRedis receives a MOVED response from the cluster). It's important to note however, that each PHP-FPM worker will have it's own cache. So if you have 8 workers, they all need to warm up but at that point, you shouldn't see additional calls to CLUSTER SLOTS.
redis.pconnect.pooling_enabled enables persistent connection pooling. RedisCluster will use the pool if persistent connections are requested in the constructor..
redis.clusters.persistent is just used when connecting to "named" clusters which can be specified in php.ini. To my knowledge this feature isn't used much.

I created a little toy script that connects to a constructor, echos a value and keeps track of how many times CLUSTER SLOTS is called.

You can use php's built-in webserver with an undocumented env variable to watch as the slots get cached in each worker.

# Start PHP's cli based webserver with 4 workers.
PHP_CLI_SERVER_WORKERS=4 php -S "localhost:9999" -t path/to/count-cluster-slots.php

# Then in another terminal.  You can also use a browser if that's more convenient.
w3m "http://localhost:9999/count-cluster-slots.php?host=localhost&port=7000" -dump

As you hit the URL you should see RedisCluster executing CLUSTER SLOTS once per PID. At that point the PID's persistent_list cache is warm and it need not call the command again.

If it continues to call CLUSTER SLOTS we'll have to investigate further.

0 replies

gskema · 2024-03-12T12:58:36Z

gskema
Mar 12, 2024
Author

Thank you for the test script. I ran it and indeed CLUSTER SLOTS is not called continuously. It is sometimes called in RedisCluster::__construct when it's not cached but I think that's ok. Sometime it's not cached for first 2-5 requests but them after that it pretty stable? However I still cannot figure out what's going on with the first operation taking twice as long:

0.000113, // RedisCluster::__construct (slot cache hit)
0.168065, // GET key1
0.096504, // GET key1
0.090341, // GET key1

0.1810691, // RedisCluster::__construct (slot cache miss, CLUSTER SLOTS called)
0.1916060, // GET key1 - still double time ?
0.0845618, // GET key1
0.0882408, // GET key1

0.00009512, // RedisCluster::__construct
0.13961195, // GET key1 - double
0.15950989, // GET key2 - double
0.08325195, // GET key2
0.07949614, // GET key2

Sometimes (like <20%) of the time it's not double, feeling like correct slot/node for random key is hit immediately. Other times (~80%) it's around double time taken, like it get MOVED response or something, but the slots are already cached judging by constructor time taken.

2 replies

michael-grunder Mar 12, 2024
Maintainer

I still cannot figure out what's going on with the first operation taking twice as long

The actual connection to the node? This would also mean that you occasionally don't see the spike when it just so happens to be receiving a key from the node used to cache the slots (done randomly).

gskema Mar 13, 2024
Author

I will mark this as answered since the slot caching was resolved, but will make a new question about these operation timings

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Can cluster slots be cached beyond request lifecycle? #2456

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Can cluster slots be cached beyond request lifecycle? #2456

Uh oh!

Uh oh!

gskema Mar 7, 2024

Replies: 2 comments · 2 replies

Uh oh!

michael-grunder Mar 7, 2024 Maintainer

Uh oh!

Uh oh!

gskema Mar 12, 2024 Author

Uh oh!

michael-grunder Mar 12, 2024 Maintainer

Uh oh!

gskema Mar 13, 2024 Author

gskema
Mar 7, 2024

Replies: 2 comments 2 replies

michael-grunder
Mar 7, 2024
Maintainer

gskema
Mar 12, 2024
Author

michael-grunder Mar 12, 2024
Maintainer

gskema Mar 13, 2024
Author