8000 Timed out attempting to find data in the correct node · Issue #1385 · phpredis/phpredis · GitHub
[go: up one dir, main page]
Skip to content

Timed out attempting to find data in the correct node #1385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
2 tasks
jianqiangsong opened this issue Aug 8, 2018 · 17 comments · Fixed by #2459
Closed
2 tasks

Timed out attempting to find data in the correct node #1385

jianqiangsong opened this issue Aug 8, 2018 · 17 comments · Fixed by #2459

Comments

@jianqiangsong
Copy link

Expected behaviour

Actual behaviour

I'm seeing this behaviour on

  • OS: ubabntu 16.04 Lts
  • Redis: 3.2
  • PHP: 7.0
  • phpredis: 4.0

Steps to reproduce, backtrace or example script

we use rediscluster ;
so many problem : timed out attempting to find data in the correct node!

I've checked

  • There is no similar issue from other users
  • Issue isn't fixed in develop branch
@yatsukhnenko
Copy link
Member

@jianqiangsong could you add more details about issue?

@jianqiangsong
Copy link
Author

We use cluster mode, but the client links redis, and the following reports are wrong.
timed out attempting to find data in the correct node!
How to solve the mistake ？

@yatsukhnenko
Copy link
Member

I'm not sure I understand the problem :(
Could you provide test script?

@jianqiangsong
Copy link
Author

[2018-08-08 14:36:26 *176089.7] ERROR zm_deactivate_swoole (ERROR 503): Fatal error: Uncaught RedisClusterException: Timed out attempting to find data in the correct node! in /var/www/pt_target/Newtrade/SwooleCommand/Controller/SwooleServerController.class.php:221
Stack trace:
#0 /var/www/pt_target/Newtrade/SwooleCommand/Controller/SwooleServerController.class.php(221): RedisCluster->hgetall('ws_client_servi...')

@yatsukhnenko
Copy link
Member

@jianqiangsong #888 (comment)
Try to change timeouts

@jianqiangsong
Copy link
Author

Is there another solution？

@yatsukhnenko
Copy link
Member

@jianqiangsong could you show how does the key look like?

@hhp12360
Copy link

we also have this question

env:

  • OS: centos 7.0
  • Redis: 3.2
  • PHP: 5.6
  • phpredis: 4.1

error info:

  • exception 'RedisClusterException' with message 'Timed out attempting to find data in the correct node!'

and it's random happen,Each lasting one second

@michael-grunder
Copy link
Member

This isn't necessarily a bug. Have you tried monitoring your Redis servers to see if you're having latency spikes?

@pagottoo
Copy link
pagottoo commented Nov 26, 2018 •
edited
Loading

Same problem here...

the redis cluster has entered recovery mode and when it returned the error message began to appear on NewRelic.

However, another apps be connect and retrieves messages from the same redis-cluster, be one app is nodejs, and the other one is php7, without any error message.

The app that catch the exception is php5.4

We are running the apps (php, node) and redis cluster in docker (inside k8s)...

@kjoe
Copy link
Contributor
kjoe commented Dec 14, 2018 •
edited
Loading

We also have this issue on PHP 5.5, but in cluster_library.c is there any reason, that waitms calculated from connect timeout, rather than read timeout? (used in function cluster_send_command)

Another possible reason is that msstart does not reseted in every loop, as written in the documentation:

"The way RedisCluster handles user specified timeout values is that every time a command is sent to the cluster, we record the time at the start of the request and then again every time we have to re-issue the command to a different node (either because Redis cluster responded with MOVED/ASK or because we failed to communicate with a given node). Once we detect having been in the command loop for longer than our specified timeout, an error is raised."

@sofire
Copy link
sofire commented Nov 13, 2019

I have the same error on PHP 7.1，and PHP-FPM exited on signal 11 (SIGSEGV)

PHP Redis Version => 4.3.0
Redis cluster Version 4.0.11

@yatsukhnenko
Copy link
Member

@sofire it is recommended to use phpredis 5 with PHP 7

@astar10239
Copy link

Any solution for this, i am also getting the same issue

@saturnonfire
Copy link

🔼 UP
Any updates on this issue?

@ivan-nezhura
Copy link

Have the same issue

@tuomasva
Copy link
tuomasva commented Jan 2, 2024 •
edited
Loading

We are also seeing this in one of our testing environments but don't really know where to look for more information on how to fix this. We haven't even really be able to track this to redis itself but very likely seems a somekind of problem with the redis instance at least (network, hdd, redis, menory).

While the problem is on, still around half of the incoming request are served normally.

EDIT: The only hint that we are seeing is that the "broken" redis-instance gets its memory degradted over time, we have been seeing this output:

127.0.0.1:6379> memory doctor
Sam, I detected a few issues in this Redis instance memory implants:

 * High total RSS: This instance has a memory fragmentation and RSS overhead greater than 1.4 (this means that the Resident Set Size of the Redis process is much larger than the sum of the logical allocations Redis performed). This problem is usually due either to a large peak memory (check if there is a peak memory entry above in the report) or may result from a workload that causes the allocator to fragment memory a lot. If the problem is a large peak memory, then there is no issue. Otherwise, make sure you are using the Jemalloc allocator and not the default libc malloc. Note: The currently used allocator is "jemalloc-5.2.1".

I'm here to keep you safe, Sam. I want to help you.

and

127.0.0.1:6379> memory doctor
Sam, I detected a few issues in this Redis instance memory implants:

 * Peak memory: In the past this instance used more than 150% the memory that is currently using. The allocator is normally not able to release memory after a peak, so you can expect to see a big fragmentation ratio, however this is actually harmless and is only due to the memory peak, and if the Redis instance Resident Set Size (RSS) is currently bigger than expected, the memory will be used as soon as you fill the Redis instance with more data. If the memory peak was only occasional and you want to try to reclaim memory, please try the MEMORY PURGE command, otherwise the only other option is to shutdown and restart the instance.

 * High total RSS: This instance has a memory fragmentation and RSS overhead greater than 1.4 (this means that the Resident Set Size of the Redis process is much larger than the sum of the logical allocations Redis performed). This problem is usually due either to a large peak memory (check if there is a peak memory entry above in the report) or may result from a workload that causes the allocator to fragment memory a lot. If the problem is a large peak memory, then there is no issue. Otherwise, make sure you are using the Jemalloc allocator and not the default libc malloc. Note: The currently used allocator is "jemalloc-5.2.1".

 * High process RSS overhead: This instance has non-allocator RSS memory overhead is greater than 1.1 (this means that the Resident Set Size of the Redis process is much larger than the RSS the allocator holds). This problem may be due to Lua scripts or Modules.

I'm here to keep you safe, Sam. I want to help you.

kjoe added a commit to kjoe/phpredis that referenced this issue Mar 17, 2024
@kjoe
Fix random connection timeouts with Redis Cluster
b8e8eb0 
When a node timeout occurs, then phpredis will try to connect to another
node, whose answer probably will be MOVED redirect. After this we need
more time to accomplish the redirection, otherwise we get "Timed out
attempting to find data in the correct node" error message.

Fixes phpredis#795 phpredis#888 phpredis#1142 phpredis#1385 phpredis#1633 phpredis#1707 phpredis#1811 phpredis#2407
@kjoe kjoe mentioned this issue Mar 17, 2024
Merged
michael-grunder pushed a commit that referenced this issue Mar 25, 2024
@kjoe @michael-grunder
Fix random connection timeouts with Redis Cluster
eb7f31e 
When a node timeout occurs, then phpredis will try to connect to another
node, whose answer probably will be MOVED redirect. After this we need
more time to accomplish the redirection, otherwise we get "Timed out
attempting to find data in the correct node" error message.

Fixes #795 #888 #1142 #1385 #1633 #1707 #1811 #2407
michael-grunder pushed a commit that referenced this issue Aug 29, 2024
@kjoe @michael-grunder
Fix random connection timeouts with Redis Cluster
1af521a 
When a node timeout occurs, then phpredis will try to connect to another
node, whose answer probably will be MOVED redirect. After this we need
more time to accomplish the redirection, otherwise we get "Timed out
attempting to find data in the correct node" error message.

Fixes #795 #888 #1142 #1385 #1633 #1707 #1811 #2407
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix random connection timeouts with Redis Cluster kjoe/phpredis
0