8000 [Lock] SemaphoreStore : sem_remove() produces a warning · Issue #29952 · symfony/symfony · GitHub
[go: up one dir, main page]
Skip to content
8000

[Lock] SemaphoreStore : sem_remove() produces a warning #29952

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
fmata opened this issue Jan 22, 2019 · 11 comments
Closed

[Lock] SemaphoreStore : sem_remove() produces a warning #29952

fmata opened this issue Jan 22, 2019 · 11 comments
Labels
Bug Lock Status: Waiting feedback Unconfirmed

Comments

@fmata
Copy link
Contributor
fmata commented Jan 22, 2019

Symfony version(s) affected: 3.4.20 / PHP 7.2.5

Description
As stated in #27357 (comment), when I use the LockableTrait in command :

protected function execute(InputInterface $input, OutputInterface $output): int
{
    if (!$this->lock()) {
        $output->writeln('The command is already running in another process.');
    
        return 0;
    }

    // ...

    $this->release();
}

A warning is emitted :

Warning: sem_remove(): failed for SysV sempphore 140713294922928: Operation not permitted {"exception":"[object] (ErrorException(code: 0): Warning: sem_remove(): failed for SysV sempphore 140713294922928: Operation not permitted at /path/vendor/symfony/symfony/src/Symfony/Component/Lock/Store/SemaphoreStore.php:113

The command is executed by a cronjob under an user and Unix group different of those of webserver.

How to reproduce
Pre-prod and prod are impacted but I don't know how to reproduce exactly.

Possible Solution
As the warning seems to be non-blocking, I suggest to silence the call to sem_remove() as done with @sem_acquire().

Let me know if I can make the PR.
@jderusse
Copy link
Member

Ignoring deletion of semaphore is not ideal, as the number of semaphore is limited (see original bug #27356)

Note: this error is triggered when a user tries to remove a semaphore acquired (and not (yet) removed) by another user

I'm gonna investigate on it

@fmata
Copy link
Contributor Author
fmata commented Jan 22, 2019

Thanks for investigate but ours commands are always executed by the same user so acquire and remove are done under the same user and group.

@jderusse
Copy link
Member
jderusse commented Jan 22, 2019 •
edited
Loading

hmm. I missunderstood your comment The command is executed by a cronjob under an user and Unix group different of those of webserver

I'm not able to reproduce an Operation not permitted when calling acquire and release with the same user.

Could you please provide a reproduct-case?

@fmata
Copy link
Contributor Author
fmata commented Jan 23, 2019

I don't know how to build a reproducer except providing you the full command but I doubt it helps you :-/

@Simperfit
Copy link
Contributor

You can still provide us the command so we can test by ourself ;).

@fembuelita
Copy link
fembuelita commented Jun 18, 2019 •
edited
Loading

I've been getting similar issues over the last few weeks:

PHP Warning: sem_remove(): SysV semaphore 140711371240096 does not (any longer) exist in /var/www/[redacted]/app/vendor/symfony/lock/Store/SemaphoreStore.php on line 95

I'm uncertain if it's related, but it's also from a cronjob running a command that uses the Lockable trait. Here's the cronjob task that executes the script:

*/5 * * * * php /var/www/[redacted]/bakery orders:download -q --mode=PROD > /dev/null

This runs an SFTP script which is slow hence using Lockable so we don't open multiple connections and download files already in progress of being downloaded. It appears the semaphore is eventually released.

here's a few other recent errors:

PHP Warning:  sem_get(): failed acquiring SYSVSEM_SETVAL for key 0x38ee78: Invalid argument in /var/www/[redacted]/app/vendor/symfony/lock/Store/SemaphoreStore.php on line 68
PHP Warning:  sem_get(): failed for key 0x38ee78: Invalid argument in /var/www/[redacted]/app/vendor/symfony/lock/Store/SemaphoreStore.php on line 68
PHP Warning:  sem_get(): failed releasing SYSVSEM_SETVAL for key 0x38ee78: Invalid argument in /var/www/[redacted]/app/vendor/symfony/lock/Store/SemaphoreStore.php on line 68

We started having these errors occur on June 8th. We have no records of any software updates at this point, so I don't think the date of the June 8th has any significance for us.

It's unclear how we can reproduce these steps.

System Info

Lock v4.2.0

$ php -v
PHP 7.2.16-1+ubuntu16.04.1+deb.sury.org+1 (cli) (built: Mar  7 2019 20:23:06) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.2.16-1+ubuntu16.04.1+deb.sury.org+1, Copyright (c) 1999-2018, by Zend Technologies

@Simperfit
Copy link
Contributor

Could you please provide a reproducer ?

@fembuelita
Copy link

@Simperfit unfortunately not. It appears to be a race condition in that it only happens rarely despite that the scripts in question run every 5 minutes. Over the last 3 weeks I've only had the errors occur less than 5 times, and it appears the system recovers automatically from it.

@xabbuh
Copy link
Member
xabbuh commented Jul 2, 2019

Let's close here then. Without being able to reproduce we won't be able to fix it. Anyway, we can reopen once we are able to reproduce it.

@xabbuh xabbuh closed this as completed Jul 2, 2019
@hsibboni
Copy link
hsibboni commented Feb 3, 2020

Hello, I found a way to reproduce warnings.
TL,DR : But there is no bug, you should just update the documentation.

It happens on Linux with systemd configuration : RemoveIPC=1 and using a non system user:
RemoveIPC=
Controls whether System V and POSIX IPC objects belonging to the
user shall be removed when the user fully logs out. Takes a
boolean argument. If enabled, the user may not consume IPC
resources after the last of the user's sessions terminated. This
covers System V semaphores, shared memory and message queues, as
well as POSIX shared memory and message queues. Note that IPC
objects of the root user and other system users are excluded from
the effect of this setting. Defaults to "yes".

  1. About does not (any longer) exist

You need to start a command with the LockableTrait with a non system user on a Linux server with systemd and RemoveIPC = 1.

If the same user logs in and logs out, the semaphores will be removed by systemd(rsync with ssh, or sftp, were able to reproduce, in my case; it must open a session and close it).

When the LockableTrait ends, you will get this warning

PHP Warning: sem_remove(): SysV semaphore 140711371240096 does not (any longer) exist in /var/www/[redacted]/app/vendor/symfony/lock/Store/SemaphoreStore.php on line 95

Because the semaphore was removed.

And because the semaphore was removed, any other command could have used it : you have concurrency issues / race conditions ! @elliottpost

  1. About not permitted
    It is the same as 1) : LockableTrait starts, user logs in/logs out, semaphore is removed by systemd.

Then the web user acquires the semaphore : it does not share the same permissions with the Command starting user.

Then the Command ends : it cannot remove the semaphore, because it does not have permissions

Warning: sem_remove(): failed for SysV sempphore 140713294922928: Operation not permitted {"exception":"[object] (ErrorException(code: 0): Warning: sem_remove(): failed for SysV sempphore 140713294922928: Operation not permitted at /path/vendor/symfony/symfony/src/Symfony/Component/Lock/Store/SemaphoreStore.php:113

You have concurrency issues / race conditions ! @fmata

What can be done : nothing in the code, just use a system user or change configuration.

This should be warning when using the LockableTrait. Maybe you could add another Trait using the flockStore for users that cannot change this configuration on their server.

I hope I am not mistaken, but I was able to reproduce everything. And semaphore being removed was a big issue for postgresql users / some Linux distributions changed the default value for RemoveIPC : you can read more on the internet

I used the flockStore to avoid this issue.

@jderusse jderusse mentioned this issue Feb 3, 2020
Closed
@jderusse
Copy link
Member
jderusse commented Feb 3, 2020

Thanks so much @hsibboni for this well detailled issue. That would explain a lot.

Doc PR opened symfony/symfony-docs#13035

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug Lock Status: Waiting feedback Unconfirmed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@jderusse @fmata @xabbuh @fembuelita @Simperfit @carsonbot @hsibboni
0