[4.4] Dramatic increase in memory consumption for tests after updating to 4.4 #35164
Comments
|
What's the size of the getTest_PrivateServicesLocator.php file in both cases? |
|
On 4.3: On 4.4: |
|
Maybe its related to those changes in https://github.com/symfony/symfony/compare/4.3..4.4#diff-87a8cc91fd08f09a8a3c0b37d7ddf177 |
|
Just as a confirmation; we've run into the same memory issue when upgrading (the framework bundle) from 4.3 to 4.4.2 Edit: The hint by @dmaicher was indeed the solution for us (thanks!). Before, our tests did not reset the kernel container after shutdown. With 4.4, you must reset the container. |
|
@allcode do you mean you made your tests call I tried that and it made no difference. |
|
@ndench Our functional test suite does not extend KernelTestCase, but uses the same strategy. We had to add a |
|
@ndench Does #35164 (comment) help? |
|
Sorry @allcode @xabbuh missed the response. All our non-unit tests (eg. functional, integration, etc) extend KernelTestCase, which calls
I then tried two things:
Yes I've made sure our Looking at the LiipFunctionalTestBundle's |
|
Does anyone have any ideas on where to look to solve this? |
|
@ndench in one of my larger project with lots of functional and kernel testcases I use this phpunit listener to free up memory from members in test classes after the test: class PHPUnitTestCleanUpListener implements TestListener
{
use TestListenerDefaultImplementation;
public function endTest(Test $test, float $time): void
{
if (!$test instanceof KernelTestCase) {
return;
}
self::stripProperties($test);
}
private static function stripProperties(object $target): void
{
$refl = new \ReflectionObject($target);
foreach ($refl->getProperties() as $prop) {
if (!$prop->isStatic() && 0 !== \strncmp($prop->getDeclaringClass()->getName(), 'PHPUnit_', 8)) {
$prop->setAccessible(true);
$prop->setValue($target, null);
}
}
}
}Its a bit of a hack but for me this reduced memory usage significantly and improved the performance quite a bit. Maybe give it a try. This does not really solve the problem but it hides the symptoms 😄 🙈 EDIT: source is this blog post that I stumbled across: https://tomnewby.net/posts/speed-up-phpunit-1-weird-trick/ |
|
@dmaicher funny story, Tom Newby is actually my colleague and wrote that blog post after the last time we had issues with PHPUnit and memory consumption 😂. It's served us very well for a while now, but it's not helping after the upgrade to 4.4. |
|
I just tried updating to the latest version of 4.4 again and am receiving the same issue. |
|
I would happily have a look but I'd need a reproducer for sure. |
|
Thanks @nicolas-grekas, I'm I bit stuck about where to start. I'll need to spend some more time breaking it down in our codebase but hopefully I can get a reproducer in a clean Symfony install for you. If you have any pointers or gut feels for where to start looking it would be much appreciated. |
|
I've been trying to create a reproducer for this. I though that it was caused by the LiipFunctionalTest bundle and the use of fixtures in integration tests. So I started with a base Symfony install, added a few entities, fixtures (including the LiipFunctionalTestBundle) and essentially copied over some tests from my project. But they didn't get the memory blow out when upgrading to 4. So now I have no idea where to look next. To try track it down further, I've installed the oldest version of the 4.4.0 branch that I can, --- a/composer.json
+++ b/composer.json
@@ -35,23 +35,23 @@
- "symfony/asset": "^4.0",
- "symfony/console": "^4.0",
+ "symfony/asset": "4.4.0-BETA1",
+ "symfony/console": "4.4.0-BETA1",
"symfony/debug-pack": "^1.0.4",
- "symfony/expression-language": "^4.0",
+ "symfony/expression-language": "4.4.0-BETA1",
"symfony/flex": "^1.0",
- "symfony/form": "^4.0",
- "symfony/framework-bundle": "^4.0",
+ "symfony/form": "4.4.0-BETA1",
+ "symfony/framework-bundle": "4.4.0-BETA1",
"symfony/monolog-bundle": "^3.1",
"symfony/orm-pack": "^1.0.5",
- "symfony/process": "^4.0",
- "symfony/security-bundle": "^4.0",
+ "symfony/process": "4.4.0-BETA1",
+ "symfony/security-bundle": "4.4.0-BETA1",
"symfony/serializer-pack": "^1.0.1",
"symfony/swiftmailer-bundle": "^3.1",
- "symfony/validator": "^4.0",
- "symfony/web-link": "^4.0",
- "symfony/workflow": "^4.0",
- "symfony/yaml": "^4.0"
+ "symfony/validator": "4.4.0-BETA1",
+ "symfony/web-link": "4.4.0-BETA1",
+ "symfony/workflow": "4.4.0-BETA1",
+ "symfony/yaml": "4.4.0-BETA1"
},
"require-dev": {
@@ -72,8 +72,8 @@
- "symfony/browser-kit": "^4.0",
- "symfony/css-selector": "^4.0",
+ "symfony/browser-kit": "4.4.0-BETA1",
+ "symfony/css-selector": "4.4.0-BETA1",
"symfony/dotenv": "^4.2.0",
"symfony/maker-bundle": "^1.0",
"symfony/profiler-pack": "^1.0.3",
@@ -123,7 +123,7 @@
"symfony": {
"id": "01C557KGKVJGP76W8CRRW45KYC",
"allow-contrib": false,
- "require": "^4.3"
+ "require": "4.4.0-BETA1"
}and the issue still exists. However the issue does not exist on the latest version of the Is there a way I can install from a given commit hash prior to that tag? Or another way I can track down this issue and create a good reproducer? |
Checkout the symfony/symfony repository via git. It contains a script called |
|
@derrabus that's exactly what I was after, thanks for pointing that out 👍 |
|
Hi there, we've used blackfire to create traces on a cut down test suite that only runs two tests. This is the minimum to replicate the issue with memory usage jumping from 134MB to 204MB after upgrading to 4.4.0-BETA1 (using the beta to minimise the diff). Trace from 4.3.11 We're not 100% sure on how best to interpret the blackfire results, but observations:
In 4.3.11, the second test run has very low memory usage compared to the first test (as expected), in comparison to 4.4.0-BETA1 that shows the second test using the same amount of memory. This would then scale out across all our tests to blow up.
When we do a compare, we're not sure how best to interpret the results. It appears like it's perhaps an issue with the TraceableEventDispatcher, but that said, there doesn't seem to have been many significant changes from 4.3.11 to 4.4.0-BETA1 that would indicate this would be where to start looking. Does anyone have any suggestion on how to interpret this? Desperate for any ideas at this point 😅 |
|
Could you share the application and the tests behind these profiles? |
|
Can you check again now that #36460 is merged ? It fixed a memory leak (but it looks like it also affected other versions, so it might not be the one relevant here) |
|
@xabbuh I'm happy to add you (and anyone else) as a collaborator onto the project, and to give a quick rundown of how to reproduce the error. Unfortunately the project isn't open source so I can't open it up to everyone. The project is contained in a publicly available vagrant box, so it will be easy to get set up. Let me know if you'd like to do this? @stof thanks for pointing out #36460, I gave it a shot by pulling the latest 4.4 branch locally and using the Additionally, after running a |
|
@ndench as written via Symfony Slack I would be interested to help debug/profile this further 😊 Sounds like a challenge 😄 |
|
@ndench I had a look at your tests and (like you also found out) its related to the The Also the container that is reset inside So this call does not reset anything and I suspect there are some cyclic references or so and PHP's garbage collection cannot clean-up everything? Some simple fix for your use Liip\FunctionalTestBundle\Test\WebTestCase;
use Symfony\Component\DependencyInjection\ContainerInterface;
use Symfony\Component\HttpKernel\KernelInterface;
abstract class BaseWebTestCase extends WebTestCase
{
/**
* @var KernelInterface
*/
private static $localKernel;
protected function tearDown(): void
{
if (self::$localKernel !== null) {
self::$localKernel->shutdown();
}
parent::tearDown();
}
/**
* override from Liip\FunctionalTestBundle\Test\WebTestCase to allow garbage collection
*/
protected function getContainer(): ContainerInterface
{
if (self::$localKernel === null) {
self::$localKernel = self::createKernel();
}
self::$localKernel->boot();
return self::$localKernel->getContainer()->get('test.service_container');
}
}this brings down memory consumption to roughly the same level as with Symfony 4.3 and on my machine this even speeds up your functional tests by more than 1 minute 😄 This should be reported on |
|
Thanks @dmaicher, really appreciate your help! I've submitted a PR to the TestFixturesBundle (liip/LiipTestFixturesBundle#62). And we're now successfully running Symfony 4.4 which halved the time our tests take to run 🎉 |
Uh oh!
There was an error while loading. Please reload this page.
Symfony version(s) affected: 4.4.2
Description
When we update to 4.4, my tests go from this:
To this:
At first, we thought that #34567 would fix it, but that was released a while back and it did not make a difference. Another possibly related issue would be #25964.
I've been unable to track down the actual issue, and can't seem to break down the problem into a reproducer on a blank Symfony installation. However, here is everything I've been able to figure out:
How to reproduce
We can't get a bug reproducer working, however these are the steps I've taken:
composer update symfony/*php vendor/bin/phpunitAdditional context
extra.symfony.require: "4.3.*"in composer.json then ran a composer update. Then withextra.symfony.require: "^4.2.0"rancomposer update symfony/*setUpfunctionsThe text was updated successfully, but these errors were encountered: