AWS Elasticache Redis Cluster (multi node) using PHPRedis MOSTLY makes connection to a single node #2316

cdeguzman · 2023-01-11T11:20:57Z

cdeguzman
Jan 11, 2023

Hi,

I working on a PHP project that uses phpredis to connect to AWS Elasticache using the configuration endpoint (single URL) provided by the cluster.

Here are the cluster details

Engine version: 6.2.6
Cluster mode: On
Configuration Endpont: xxx:6379

I did 2 types of settings to connect to the cluster:

Connection config A - no connection pooling, no persistent conections
Connection config B - connection pooling enabled, persistent connection enabled, connection pooling limit set

For the first setting I am using php redis version 3.1.6 and I am establishing a connection using the following method (Connection Config A)

            //$seeds = xxx:6379
            //$timeout = 1.5
            //$readTimeout= 1.5
            //$persistentConection = false
            $redis = new \RedisCluster(null, $seeds, $timeout, $readTimeout, false);
            $redis->setOption(\RedisCluster::OPT_SLAVE_FAILOVER, \RedisCluster::FAILOVER_DISTRIBUTE);

Then for the second setting I am using php redis 4.3.0, updated the redis.ini and establishing a connection using the following method (Connection config B)

#redis.ini
redis.clusters.seeds="wwRedisCluster[]=xxx:6379"
redis.clusters.timeout="wwRedisCluster=1.5"
redis.clusters.read_timeout="wwRedisCluster=1.5"
redis.clusters.persistent="wwRedisCluster=1"
redis.pconnect.connection_limit=30
redis.pconnect.pooling_enabled=1

            $redis = new \RedisCluster("wwRedisCluster");
            $redis->setOption(\RedisCluster::OPT_SLAVE_FAILOVER, \RedisCluster::FAILOVER_DISTRIBUTE);

Below is screenshot of the redis connection behavior and cache hits of the 2 connection settings.

As you can see in

Connection config A
- the current connections, cache hits and new connections for the 2 nodes are mostly the same
whist in
Connection config B
- new connections are down (expected since persistent connections are enabled.)
- BUT the cache hits and current connections are MOSTLY directed to 1st node while the 2nd node barely receives any activity.

I am lost why connection config B mostly directs the connections on node 1. I was under the assumption that the connections to both nodes needs to be equal like what happened to connection config A

Can you please provide some insight how to make the connections distribute to both nodes using connection pooling, persistent connection and pconnect.limit?

michael-grunder · 2023-02-02T18:12:59Z

michael-grunder
Feb 2, 2023
Maintainer

Sorry I missed this question when it was first posted.

Off the top of my head, i'm not sure why there would be such a concentration of requests to one node when you're setting Redis::FAILOVER_DISTRIBUTE. I'll try setting up a similar local config to see if I see anything like that behavior.

0 replies

lazar725 · 2023-05-24T17:50:11Z

lazar725
May 24, 2023

@michael-grunder I have a somewhat related issue. I'm also working with Elasticache Redis, and I've encountered a lot of "broken pipe" errors, that occurs when some master nodes reach the limit of 65K connections. I initialized the RedisCluster class with persistence flag set to true, timeout and readTimeout set to 0.25 (I didn't change any redis.ini settings). I noticed that the master nodes get most of the traffic while replica nodes get almost none, so I set the OPT_SLAVE_FAILOVER - FAILOVER_DISTRIBUTE option, as in the question above. I was expecting to see less connections to the master nodes, and more connections to the replica nodes. To my surprise, the number of connections to the replica nodes increased, but at the same time the number of connections to the master nodes did not decrease, in fact, it looks like it may have even made things a little worse. I'm using the latest version of phpredis. Is there anything I can do to solve it?

Thanks.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

AWS Elasticache Redis Cluster (multi node) using PHPRedis MOSTLY makes connection to a single node #2316

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

AWS Elasticache Redis Cluster (multi node) using PHPRedis MOSTLY makes connection to a single node #2316

Uh oh!

Uh oh!

cdeguzman Jan 11, 2023

Replies: 2 comments

Uh oh!

michael-grunder Feb 2, 2023 Maintainer

Uh oh!

lazar725 May 24, 2023

cdeguzman
Jan 11, 2023

michael-grunder
Feb 2, 2023
Maintainer

lazar725
May 24, 2023