8000 Queue:work fails. MySQL server has gone away · Issue #19005 · laravel/framework · GitHub
[go: up one dir, main page]
Skip to content

Queue:work fails. MySQL server has gone away #19005

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
iget-master opened this issue Apr 29, 2017 · 18 comments
Closed

Queue:work fails. MySQL server has gone away #19005

iget-master opened this issue Apr 29, 2017 · 18 comments

Comments

@iget-master
Copy link
  • Laravel Version: 5.3.31
  • PHP Version: 7.0.15
  • Database Driver & Version: MySQL 5.7.18

Description:

The issue looks like the same issue addressed by #15179 but on L5.3.

I have 3 Queue Workers running, monitored by supervisord.
My MySQL server run on another droplet under same private network.

If the connection is lost any reason, the queue worker doesn't try to reconnect as expected, and keep logging errors infinitely:

Apr 29 19:35:05 www domain:  production.ERROR: PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away in /home/abc/www/domain.com/releases/20170428203327/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:57 Stack trace: #0 /home/abc/www/domain.com/releases/20170428203327/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php(57): PDO->exec('SAVEPOINT trans...') #1 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Database/Connection.php(623): Doctrine\DBAL\Driver\PDOConnection->exec('SAVEPOINT trans...') #2 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php(175): Illuminate\Database\Connection->beginTransaction() #3 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(175): Illuminate\Queue\DatabaseQueue->pop('default') #4 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(145): Illuminate\Queue\Worker->getNextJob(Object(Illuminate\Queue\DatabaseQueue), 'default') #5 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(75): Illuminate\Queue\Worker->runNextJob('database', 'default', Object(Illuminate\Queue\WorkerOptions)) #6 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(100): Illuminate\Queue\Worker->daemon('database', 'default', Object(Illuminate\Queue\WorkerOptions)) #7 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(83): Illuminate\Queue\Console\WorkCommand->runWorker('database', 'default') #8 [internal function]: Illuminate\Queue\Console\WorkCommand->fire() #9 /home/abc/www/domain.com/releases/20170428203327/bootstrap/cache/compiled.php(1375): call_user_func_array(Array, Array) 

#10 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Console/Command.php(169): Illuminate\Container\Container->call(Array) 
#11 /home/abc/www/domain.com/releases/20170428203327/vendor/symfony/console/Command/Command.php(261): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) 
#12 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Console/Command.php(155): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) 
#13 /home/abc/www/domain.com/releases/20170428203327/vendor/symfony/console/Application.php(817): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) 
#14 /home/abc/www/domain.com/releases/20170428203327/vendor/symfony/console/Application.php(185): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) 
#15 /home/abc/www/domain.com/releases/20170428203327/vendor/symfony/console/Application.php(116): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) 
#16 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(121): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) 
#17 /home/abc/www/domain.com/releases/20170428203327/artisan(35): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) 
#18 {main}  Next Doctrine\DBAL\Driver\PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away in /home/abc/www/domain.com/releases/20170428203327/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:59 Stack trace: #0 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Database/Connection.php(623): Doctrine\DBAL\Driver\PDOConnection->exec('SAVEPOINT trans...') #1 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php(175): Illuminate\Database\Connection->beginTransaction() #2 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(175): Illuminate\Queue\DatabaseQueue->pop('default') #3 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(145): Illuminate\Queue\Worker->getNextJob(Object(Illuminate\Queue\DatabaseQueue), 'default') #4 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(75): Illuminate\Queue\Worker->runNextJob('database', 'default', Object(Illuminate\Queue\WorkerOptions)) #5 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(100): Illuminate\Queue\Worker->daemon('database', 'default', Object(Illuminate\Queue\WorkerOptions)) #6 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(83): Illuminate\Queue\Console\WorkCommand->runWorker('database', 'default') #7 [internal function]: Illuminate\Queue\Console\WorkCommand->fire() #8 /home/abc/www/domain.com/releases/20170428203327/bootstrap/cache/compiled.php(1375): call_user_func_array(Array, Array) #9 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Console/Command.php(169): Illuminate\Container\Container->call(Array) 
#10 /home/abc/www/domain.com/releases/20170428203327/vendor/symfony/console/Command/Command.php(261): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) 
#11 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Console/Command.php(155): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) 
#12 /home/abc/www/domain.com/releases/20170428203327/vendor/symfony/console/Application.php(817): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) 
#13 /home/abc/www/domain.com/releases/20170428203327/vendor/symfony/console/Application.php(185): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) 
#14 /home/abc/www/domain.com/releases/20170428203327/vendor/symfony/console/Application.php(116): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) 
#15 /home/abc/www/domain.com/releases/20170428203327/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(121): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) 
#16 /home/abc/www/domain.com/releases/20170428203327/artisan(35): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) 
#17 {main} [] []

Steps To Reproduce:

  1. Start a Queue Worker using command php artisan queue:work database --sleep=1 --tries=3
  2. Force or wait the connection be dropped (restart mysql server is a way to do it)
  3. You will see errors being thrown every second, and the queue worker will not work again.
@iget-master
Copy link
Author
8000 iget-master commented Apr 29, 2017 •
edited
Loading

Some things to consider:

1st: Looking at Illuminate/Database/Connection.php@beginTransaction line 611 we see a try-catch block that checks if the exception was caused by lost connection (line 614). But on the else if statement, there's no try-catch block. That's exactly the point where my code is failing.

2nd: The daemon process never die. Since the connection is lost for any reason and Laravel doesn't reconnect, the daemon process should exit with any error code, forcing the supervisord or anything handling it to restart the process.

3rd: I did tested put the entire code inside a try-catch block simillar to the line 611 one, but the method causedByLostConnection() do not catch some errors that means a lost connection, like: SQLSTATE[HY000] [2002] Connection Refused

@iget-esoares

@laurencei
Copy link
Contributor
laurencei commented Apr 30, 2017 •
edited
Loading

Have you tried this on a 5.4 application? There was some refactoring on how queues & database connections work under the hood by Taylor - so it might be fixed in 5.4?

@iget-master
Copy link
Author

@laurencei Didn't tried yet, I've spend some days migrating the application from 5.2 to 5.3 just to get this issue fixed, with no success.

@laurencei
Copy link
Contributor

I was looking at the code there for 5.4 vs 5.3 - and there is quite a big difference. Taylor did a big refactor here: 03f3c8c

Please upgrade to 5.4 and report back if still broken. 5.3 is unsupported, so it is unlikely to be fixed there even if it is a bug.

@themsaid
Copy link
Member
themsaid commented May 1, 2017

Please upgrade since 5.3 is not supported anymore.

@themsaid themsaid closed this as completed May 1, 2017
@iget-master
Copy link
Author

@themsaid There's no fix for this issue in 5.4 too...

@tpharaoh
Copy link
tpharaoh commented May 4, 2017

same problem here...

@iget-master
Copy link
Author
iget-master commented May 4, 2017 •
edited
Loading

@tpharaoh What version are you using?

@tpharaoh
Copy link
tpharaoh commented May 4, 2017

5.4
ave not updated if its part of a recent fix... its on a production forge server

@iget-master
Copy link
Author
iget-master commented May 4, 2017 •
edited
Loading

@tpharaoh I think that this changes that @themsaid said are part of any 5.4 version.

So, the problem is not fixed as I thought, this issue should be re-opened @themsaid

L5.4 doesn't change the causedByLostConnection() method, so the bug still exists

@tpharaoh
Copy link
tpharaoh commented May 4, 2017

I restarted my queue and it worked. Not sure what caused it in the first place

@iget-master
Copy link
Author

@tpharaoh Restarting the queue is just an workaround to fix the issue. Everytime that your mysql server become unavailable, your queue will stop working until you manually restart it. That's the issue

@laurencei
Copy link
Contributor

A temporary workout would be to use queue:listen - because that will always restart after each job.

@iget-master
Copy link
Author
iget-master commented May 4, 2017 •
edited
Loading

@laurencei Yes, it works but is slower and consume more resources from the server. 😔

@Broutard Broutard mentioned this issue May 5, 2017
Closed
@tpharaoh
Copy link
tpharaoh commented May 10, 2017 •
edited
Loading

@laurencei Sorry for nagging on old topic,but now I have a 5.3 with this issue, and I can't upgrade it for a month or so.
queue:listen should solve it as temporary solution? Can I do that via forge? Or I need to do via command line?

@laurencei
Copy link
Contributor

I dont know if you can run queue:listen via Forge. I think that might only run it as queue:work.

@iget-master
Copy link
Author

@tpharaoh Using queue:listen solve the issue on 5.3, but I'm not sure about how to do it on forge.
queue:work --once also works if you are using supervisord to restart the queue (forge use it, as far I know)

@kendepelchin
Copy link

Just had the same on my production server. Everything went okay for a few weeks until suddenly it started firing. Guess we're updating to 5.4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@laurencei @kendepelchin @tpharaoh @themsaid @iget-master
0