[Messenger] memory-limit option inaccurate / memory_resolver not configurable #29957
Labels
Comments
This was referenced May 26, 2019
fabpot
added a commit
that referenced
this issue
May 27, 2019
…t (chalasr) This PR was merged into the 4.2 branch. Discussion ---------- [Messenger] Use "real" memory usage to honor --memory-limit | Q | A | ------------- | --- | Branch? | 4.2 | Bug fix? | yes | New feature? | no | BC breaks? | no | Deprecations? | no | Tests pass? | yes | Fixed tickets | #29957 | License | MIT | Doc PR | n/a At least it's consistent with what other daemon-based libraries do: https://github.com/php-enqueue/enqueue-dev/blob/master/pkg/enqueue/Consumption/Extension/LimitConsumerMemoryExtension.php#L58 https://github.com/M6Web/DaemonBundle/blob/master/src/M6Web/Bundle/DaemonBundle/Command/DaemonCommand.php#L493 Commits ------- fbfe2df [Messenger] Use real memory usage for --memory-limit
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Symfony version(s) affected: 4.2 with symfony/messenger v4.2.2
Description
Using the
messenger:consume-messagescommand with-m 250Mwhen memory_limit is set to 400M still crashes the worker.How to reproduce
I have (obviously) some module (like PHPSpreadsheet) with memory leaks in use and each consumed message increases the memory usage by around 20-60MB. Even with a buffer of 150M the limit option does not prevent the memory exhaustion.
Possible Solution
I don't know if
memory_get_usage(false)is accurate for other use cases than mine, else I would suggest to change the defaultmemory_resolverto set the real_usage option to true.If this is not possible because it would break other use cases it would be best to make the memory_resolver really configurable. He can be set in the
StopWhenMemoryUsageIsExceededReceiverconstructor butConsumeMessagesCommand::executeuses a hardcoded constructor without the ability to inject a memory_resolver via a service or replacing the complete StopWhenTimeLimitIsReachedReceiver with a service.Additional context
I think this is because the
memory_resolverinStopWhenMemoryUsageIsExceededReceiverusesmemory_get_usage()instead ofmemory_get_usage(true). My debug output is below, "before" is the value ofmemory_get_usage(true)when the MessageHandler starts executing, "X vs. Y" is my debug inStopWhenMemoryUsageIsExceededReceiver::receive, where X is the value ofmemory_get_usage()and Y is the value of the--memory-limitoption used to call the script. As you can seememory_get_usage(true)is much closer to the real memory limit when the script crashes:The unit tests dont use memory_get_usage at all so a real memory leak is not tested. I understand that it would be difficult to create a reproducible leak with predictable amout of memory usage where the discrepancy of memory_get_usage() and memory_get_usage(true) would show up.
The text was updated successfully, but these errors were encountered: