Description
- Laravel Version: 5.6.38
- PHP Version: 7.2.1
- Database Driver & Version: MySQL 5.7.21
Description:
The queue listener does not kill queue workers correctly on Linux.
Looking at the listener, it forks a worker process and calls run which will wait for the process to finish.
framework/src/Illuminate/Queue/Listener.php
Lines 189 to 201 in ce0e5df
The run method on the process starts the process then calls wait on it. The wait method will continuously check the timeout to make sure that the process has not yet exceeded the timeout.
https://github.com/symfony/process/blob/a867125524205460164c9ca95bc08199bb2da198/Process.php#L415
do {
$this->checkTimeout();
$running = '\\' === \DIRECTORY_SEPARATOR ? $this->isRunning() : $this->processPipes->areOpen();
$this->readPipes($running, '\\' !== \DIRECTORY_SEPARATOR || !$running);
} while ($running);The check timeout method will stop the process if the timeout is exceeded.
https://github.com/symfony/process/blob/a867125524205460164c9ca95bc08199bb2da198/Process.php#L1188
if (null !== $thi
6F52
s->timeout && $this->timeout < microtime(true) - $this->starttime) {
$this->stop(0);
throw new ProcessTimedOutException($this, ProcessTimedOutException::TYPE_GENERAL);
}The stop method with send a SIGTERM to the forked process to terminate it. It will then sleep for 1 millisecond, and if the process is still running it will send a SIGKILL to the process.
https://github.com/symfony/process/blob/a867125524205460164c9ca95bc08199bb2da198/Process.php#L855
$timeoutMicro = microtime(true) + $timeout;
if ($this->isRunning()) {
// given SIGTERM may not be defined and that "proc_terminate" uses the constant value and not the constant itself, we use the same here
$this->doSignal(15, false);
do {
usleep(1000);
} while ($this->isRunning() && microtime(true) < $timeoutMicro);
if ($this->isRunning()) {
// Avoid exception here: process is supposed to be running, but it might have stopped just
// after this line. In any case, let's silently discard the error, we cannot do anything.
$this->doSignal($signal ?: 9, false);
}
}So, the listener process with kill the worker process when the worker process has run longer than the timeout. However, this does not work correctly in a Linux environment.
For example, let's say we run a listener like this: php artisan queue:listen database --timeout 10
On Mac, if you run ps -ef | grep queue, you will see something like the following:
PID PPID CMD
8452 4973 php artisan queue:listen database --timeout 10
8456 8452 /usr/local/Cellar/php/7.2.1_12/bin/php artisan queue:work database --once --queue=default --delay=0 --memory=128 --sleep=3 --tries=0
On Linux, doing the same thing will give you the following
PID PPID CMD
31382 28462 php artisan queue:listen database --timeout 10
31386 31382 sh -c /usr/bin/php7.2 artisan queue:work 'database' --once --queue='default' --delay=0 --memory=128 --sleep=3 --tries=0
31387 31386 /usr/bin/php7.2 artisan queue:work database --once --queue=default --delay=0 --memory=128 --sleep=3 --tries=0
As you can see, on Linux, the listener doesn't fork the worker command. Instead, it forks off the sh -c command that ends up forking the worker command. So, when the listener sends the SIGTERM signal to the worker, on a Mac the worker process dies, but on Linux the sh -c process dies instead of the worker.
See symfony/symfony#21474 and the related issues for more information on this issue, and for another way to create the process to avoid this issue.
Steps To Reproduce:
- Run
php artisan queue:listen database --timeout 5 --sleep 10. This will cause the listener to fork a process that will stay alive longer than the timeout. - Run
ps -ef | grep "[q]ueue" - Wait for the timeout error on the listener
- Run
ps -ef | grep "[q]ueue"again.
On a Mac you will see that the worker is no longer running. On Linux you should still see the worker, but the sh -c command will have been killed.