debug read error on connection #70
Comments
|
Hello, Could you check your timeout settings, and try changing them to higher value? Nicolas On 30 oct. 2011, at 19:14, sorinvreply@reply.github.com wrote:
|
|
Hey Nicolas, Thanks for the reply. Timeout is set to 8000 5 seconds (which is supposed to be like forever for redis). Number of clients is set to unlimited. The server is very large and there are less than 200 clients connected at the same time. The redis process never maxes out a processor, so I assume it's not worked out too hard to issue timeouts. I am using PHP-FPM and persistent connects (supposedly it'll save the connect time, so a bit faster). Can this be an issue? Thanks, |
|
Related to this... I was confused about the timeout setting in connect/pconnect. I thought it pertained only to obtaining a connection to the server. But it turns out it sets the timeout for all requests. I set it to 2.5 seconds and was wondering why I kept getting the "read error on connection" exception thrown. The server also has it's own timeout which will cause the exception to be thrown when it is reached. |
|
@sorinv, I'm not sure how PHP-FPM is working internally. If it doesn't reuse processes, there's no advantage to using pconnect instead of connect. pconnect is useful with Apache in prefork mode, where processes are reused a number of times.
|
|
The PHP-FPM does reuse processes, so pconnect should be a gain (albeit small). On the php_stream_get returning null, I assume no way to know exactly what the cause is? If the server closed the connection, I would know to look at the server side, for example. |
|
@sorinv, what Btw, providing more meaningful exceptions if a big plus for debugging. |
|
@bobrik, that command shows 100000, so open files should not be an issue. |
|
+1 for this issue, we have some errors like this and have no idea what's the reason. |
|
This specific error has plagued us as well, and it is incredibly hard to track down. I decided to hack at it a bit tonight and may have found something. You can cause this error to happen by issuing a $connection->subscribe() command on a channel where you don't publish anything. After a certain amount of time it'll timeout (it's idle) and you'll get this error. This very simple program will do this for you: When I run this, I get the "read error on connection" message at 60 seconds, every time. My timeout in redis.conf is zero. @nicolasff: I just wanted to mention how awesome phpredis is. Truly fantastic! :) |
|
@michael-grunder, could you please try setting |
|
For me, setting I've been using |
|
+1 on this issue... When using a long timeout, occasionally the redis server is screwed up and the php lags trying to connect (when it should error out and go to the next server).... a short timeout starts showing up read error.... catch 22? |
|
+1 There's 2 pattern in OS X crash report :
And
Rest of the log doesn't concern redis
Using connect() instead of pconnect() seems to be a temporary fix |
|
Another +1 on this when attempting to subscribe to a channel. Setting the default_socket_timeout = 0 throws the exception immediately, -1 and the app hangs forever. All running on php 5.3 on Centos. |
|
+1 to the issue |
|
+1 |
1 similar comment
|
+1 |
|
Just adding another data point.. I ran into this when benchmarking Redis as a cache backend. It does not occur for me if connecting over TCP, only if using a unix socket. It occurs more frequently with more concurrent processes. For example, when testing with 64 concurrent clients, all but 17 had the error. When using a standalone PHP driver with all else being equal, there were no such errors. Debian 64bit with Redis 2.4.9 (dotdeb), phpredis 2.1.3 (master), PHP 5.3.10 (dotdeb) from CLI. Standalone PHP driver used with no errors is Credis_Client which connects with stream_socket_client and reads with fgets and stream_get_contents. |
|
Forgot to mention, I am not using persistent connections with either phpredis or standalone mode, |
|
If you have a BIG bunch of servers (over 100, is really enough) then this error may occur because of syncookie enabled in the kernel (freebsd and linux). Disabling syncookie fixed issue for us. |
|
setting syncookies to off should be done on redis servers or on clients' servers or both ? |
|
We disabled it everywhere, but I think server side is more important. |
|
I am only running one instance of Redis. What baffles me is that the standalone PHP driver has no read errors on the socket for the exact same benchmark tests, but phpredis does.. So, there is something different with phpredis' connection handling that makes it less stable. |
|
Same issue on Redis 2.4.10 and php-redis 2.1.3 on OS X 10.7.3 / macports. |
|
@asuth How a 8000 re you connecting to Redis. Are you setting a 5 second timeout? Also, have you checked your redis.conf timeout setting and the default_socket_timeout php.ini setting? I don't get this problem with the following script: $r = new Redis();
$r->connect('localhost',6379);
$st = microtime(true);
$r->blpop('mykey',0);
$et = microtime(true);
echo "took: " . ($et-$st) . " to die\n";I let it run for like ten minutes, and it blocked until I pushed something onto 'mykey' |
|
@michael-grunder not everyone connects to localhost ... |
|
@rookie7799 They don't? I thought the whole internet was just 'localhost'! Seriously though, I just used localhost as an example rather than put the IP address for one of our production instances. Seemed prudent. |
|
I'm just saying that the problem lies somewhere in the network layer and since you're testing on localhost you can't really use that as an argument - no? |
|
@rookie7799 For sure, there are just other variables. For instance if you connect with $r->connect($svr,$port,5), you'll get a 5 second timeout. I believe you'll also get that if your default_socket_timeout is 5 in php.ini The following script will timeout @5 seconds with a "Read error on connection" message, for example try {
$r = new Redis();
if(!$r->connect('localhost',6379,5)) {
echo "Couldn't connect\n";
die();
}
$st = microtime(true);
$val = $r->blpop('mykey',0);
$et = microtime(true);
} catch(Exception $ex) {
echo "ex: " . $ex->getMessage() . "\n";
$et = microtime(true);
}
echo "Took: " . ($et-$st) . " to get the key\n";It could totally be a bug, just good to track those bits down first is all. |
|
We too facing the same issue. @yatsukhnenko does this patch fixes this issue? |
|
@patademahesh, It should :) |
|
I'm going to rewrite this patch without |
|
@yatsukhnenko how would I be able to test it? I get this error every day for the past year, driving me insane! |
|
@viion you should apply the patch I've listed to the source code and compile extension |
|
Hey guys, suggestion? |
|
@matteomartinelli, definitely you are doing something wrong 😂 |
|
@yatsukhnenko There's no logic inside my callback, it just raises a |
|
I misunderstood the BTW, Solved. |
|
in my CLI php script was this line |
|
@yatsukhnenko Is there anyway you can test with the code in issue: comment: #70 (comment).... and confirm failure as outlined in the issue and then after, apply your your fix, and confirm that the failure goes away |
|
@virgofx, are you asking me to test my own patch? 😂 |
|
@yatsukhnenko i couldn't get your patch working, or it didn't fix my issue. Not sure how to confirm. My issue was happening every day for about 15 minutes constant I'd get the error. I modified my I can deal with it a few times a week since its self-recovery. This is on a site that does about 4000-5000 interactions with Redis every minute (about 80% get GETs). I do not know if that has any effect. |
|
@yatsukhnenko Yes :) Simply because it's probably easier for you once patch built to The script is literally self-contained in that comment. |
|
@virgofx, I've used your test script while created patch |
|
@viion |
|
hey, guys I am using 3.1.2 version and encountered this issue too, using but it's affecting the whole php process life cycle, so I did some digging, and found this and from git blame, update:
so, if I make some changes to the test script @@ -58,9 +58,6 @@
$iterationsPerFork = 10;
$timeout = 2;
-$redis = new \Redis();
-$redis->connect($host, $port, $timeout);
-
$forker = new \Forker();
function setRedisData($redis, $key, $data, $i, $idx) {
@@ -72,7 +69,9 @@
}
for ($i = 0; $i < $forks; $i++) {
- $forker->fork(function() use ($iterationsPerFork, $forks, $i, $redis, $timeout, $host, $port) {
+ $forker->fork(function() use ($iterationsPerFork, $forks, $i, $timeout, $host, $port) {
+ $redis = new \Redis();
+ $redis->connect($host, $port, $timeout);
for ($idx = 0; $idx < $iterationsPerFork; $idx++) {
$key = 'S:' . md5(uniqid($i . $idx, true));
$data = 'small-data';then everything goes perfectly |
|
I'm using predis, I solved the problem several days ago. I set the redis server timeout to a none zero value. What I found is my redis server hold some long time idle connections, when the client try to new connection, error comes out. So after I set the timeout, redis serve kill the outdated connection, and here comes peace. Hope helps :) |
|
@ itbdw you said |
|
Closing this issue as it is too old and contains a lot of unnecessary information... |
Ashamed, however, do not solve the problem 😲 We use the pconnect, because limit production environment, the number of connections,Is it a problem？:joy: |
|
very good~ |
|
for those who don't want to class MyClass
{
/** @var \Redis */
private $redis;
/** @var int php default socket timeout is 60 seconds for ubuntu */
private $maxTimeout = 0;
public function __construct(\Redis $redis)
{
$this->redis = $redis;
// the value may be -1 for some configurations
$this->maxTimeout = ini_get('default_socket_timeout');
}
private function getTimeout(int $timeout): int
{
// see https://github.com/phpredis/phpredis/issues/492
if ($timeout <= 0) {
// want to run forever, but may not be allowed to
return max(0, $this->maxTimeout - 1);
}
if ($this->maxTimeout <= 0) {
// there is no php maximum
return $timeout;
}
// run for as long as you can
return min($this->maxTimeout - 1, $timeout);
}then call eg |
|
Is this for all sockets or just tcp? |
Hi there,
I am getting a lot of "read error on connection" errors. Server is under load, but not a lot. I traced this to always happen here:
Any way I can further debug this to figure out why it happens?
Thanks,
Sorin
The text was updated successfully, but these errors were encountered: