8000 Queue listener doesn't kill the worker when the timeout is exceeded on Linux · Issue #25980 · laravel/framework · GitHub
[go: up one dir, main page]

Skip to content
8000

Queue listener doesn't kill the worker when the timeout is exceeded on Linux #25980

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jshayes opened this issue Oct 7, 2018 · 0 comments
Closed

Comments

@jshayes
Copy link
jshayes commented Oct 7, 2018
  • Laravel Version: 5.6.38
  • PHP Version: 7.2.1
  • Database Driver & Version: MySQL 5.7.21

Description:

The queue listener does not kill queue workers correctly on Linux.

Looking at the listener, it forks a worker process and calls run which will wait for the process to finish.

public function runProcess(Process $process, $memory)
{
$process->run(function ($type, $line) {
$this->handleWorkerOutput($type, $line);
});
// Once we have run the job we'll go check if the memory limit has been exceeded
// for the script. If it has, we will kill this script so the process manager
// will restart this with a clean slate of memory automatically on exiting.
if ($this->memoryExceeded($memory)) {
$this->stop();
}
}

The run method on the process starts the process then calls wait on it. The wait method will continuously check the timeout to make sure that the process has not yet exceeded the timeout.

https://github.com/symfony/process/blob/a867125524205460164c9ca95bc08199bb2da198/Process.php#L415

do {
    $this->checkTimeout();
    $running = '\\' === \DIRECTORY_SEPARATOR ? $this->isRunning() : $this->processPipes->areOpen();
    $this->readPipes($running, '\\' !== \DIRECTORY_SEPARATOR || !$running);
} while ($running);

The check timeout method will stop the process if the timeout is exceeded.
https://github.com/symfony/process/blob/a867125524205460164c9ca95bc08199bb2da198/Process.php#L1188

if (null !== $this->timeout && $this->timeout < microtime(true) - $this->starttime) {
    $this->stop(0);

    throw new ProcessTimedOutException($this, ProcessTimedOutException::TYPE_GENERAL);
}

The stop method with send a SIGTERM to the forked process to terminate it. It will then sleep for 1 millisecond, and if the process is still running it will send a SIGKILL to the process.
https://github.com/symfony/process/blob/a867125524205460164c9ca95bc08199bb2da198/Process.php#L855

$timeoutMicro = microtime(true) + $timeout;
if ($this->isRunning()) {
    // given SIGTERM may not be defined and that "proc_terminate" uses the constant value and not the constant itself, we use the same here
    $this->doSignal(15, false);
    do {
        usleep(1000);
    } while ($this->isRunning() && microtime(true) < $timeoutMicro);

    if ($this->isRunning()) {
        // Avoid exception here: process is supposed to be running, but it might have stopped just
        // after this line. In any case, let's silently discard the error, we cannot do anything.
        $this->doSignal($signal ?: 9, false);
    }
}

So, the listener process with kill the worker process when the worker process has run longer than the timeout. However, this does not work correctly in a Linux environment.

For example, let's say we run a listener like this: php artisan queue:listen database --timeout 10

On Mac, if you run ps -ef | grep queue, you will see something like the following:

PID   PPID CMD
8452  4973 php artisan queue:listen database --timeout 10
8456  8452 /usr/local/Cellar/php/7.2.1_12/bin/php artisan queue:work database --once --queue=default --delay=0 --memory=128 --sleep=3 --tries=0

On Linux, doing the same thing will give you the following

PID   PPID  CMD
31382 28462 php artisan queue:listen database --timeout 10
31386 31382 sh -c /usr/bin/php7.2 artisan queue:work 'database' --once --queue='default' --delay=0 --memory=128 --sleep=3 --tries=0
31387 31386 /usr/bin/php7.2 artisan queue:work database --once --queue=default --delay=0 --memory=128 --sleep=3 --tries=0

As you can see, on Linux, the listener doesn't fork the worker command. Instead, it forks off the sh -c command that ends up forking the worker command. So, when the listener sends the SIGTERM signal to the worker, on a Mac the worker process dies, but on Linux the sh -c process dies instead of the worker.

See symfony/symfony#21474 and the related issues for more information on this issue, and for another way to create the process to avoid this issue.

Steps To Reproduce:

  1. Run php artisan queue:listen database --timeout 5 --sleep 10. This will cause the listener to fork a process that will stay alive longer than the timeout.
  2. Run ps -ef | grep "[q]ueue"
  3. Wait for the timeout error on the listener
  4. Run ps -ef | grep "[q]ueue" again.

On a Mac you will see that the worker is no longer running. On Linux you should still see the worker, but the sh -c command will have been killed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant
0