8000 Remove randomness from dumped containers by nicolas-grekas · Pull Request #25671 · symfony/symfony · GitHub
[go: up one dir, main page]

Skip to content

Remove randomness from dumped containers #25671

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 4, 2018

Conversation

nicolas-grekas
Copy link
Member
@nicolas-grekas nicolas-grekas commented Jan 3, 2018
Q A
Branch? 3.4
Bug fix? yes
New feature? no
BC breaks? no
Deprecations? no
Tests pass? yes
Fixed tickets -
License MIT
Doc PR -

With this PR, the generated container is immutable by cache clearing: it doesn't contain any random string anymore (well, third party bundles can add random things back, but at least core doesn't).

Since the class+file name of the container is based on a hash of its content, it means that they are now stable also. This should help fix some edge cases/race conditions during cache clears/rebuilds.

(fabbot failure is false positive)

@nicolas-grekas nicolas-grekas added this to the 3.4 milestone Jan 3, 2018
@nicolas-grekas nicolas-grekas force-pushed the stable-container branch 2 times, most recently from cb7046b to 23cb152 Compare January 3, 2018 21:18
$version = (new Definition(\ReflectionClass::class))->addArgument(new Reference('service_container'));
$version = (new Definition())->setFactory(array($version, 'getFileName'));
$version = (new Definition())->setFactory('implode')->addArgument(array(
(new Definition())->setFactory('filemtime')->addArgument($version),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hum actually this incurs unwanted IO, I need to look for something else

@nicolas-grekas nicolas-grekas force-pushed the stable-container branch 4 times, most recently from b8553e3 to bd4161f Compare January 4, 2018 10:59
@nicolas-grekas
Copy link
Member Author
nicolas-grekas commented Jan 4, 2018

PR ready (fabbot failure is false positive.)
This adds build metadata to dumped containers, that then can be and is referenced by bundles that need build time related entropy.

@nicolas-grekas
Copy link
Member Author

(failure caused by bug in Composer, see composer/composer#6973)

@sroze
Copy link
Contributor
sroze commented Jan 4, 2018

Could be retried now with the new release then :)

@nicolas-grekas
Copy link
Member Author

PR green (deps=high failure will be fixed by merging to upper branches)

@@ -20,7 +20,7 @@
"ext-xml": "*",
"symfony/cache": "~3.4|~4.0",
"symfony/class-loader": "~3.2",
"symfony/dependency-injection": "~3.4|~4.0",
"symfony/dependency-injection": "~3.4.3|~4.0.3",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shoudn't it be ^3.4.3|^4.0.3

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oups, fixed

@fabpot
Copy link
Member
fabpot commented Jan 4, 2018

Thank you @nicolas-grekas.

1 similar comment
@fabpot
Copy link
Member
fabpot commented Jan 4, 2018

Thank you @nicolas-grekas.

@fabpot fabpot merged commit 14dd5d1 into symfony:3.4 Jan 4, 2018
fabpot added a commit that referenced this pull request Jan 4, 2018
This PR was merged into the 3.4 branch.

Discussion
----------

Remove randomness from dumped containers

| Q             | A
| ------------- | ---
| Branch?       | 3.4
| Bug fix?      | yes
| New feature?  | no
| BC breaks?    | no
| Deprecations? | no
| Tests pass?   | yes
| Fixed tickets | -
| License       | MIT
| Doc PR        | -

With this PR, the generated container is immutable by cache clearing: it doesn't contain any random string anymore (well, third party bundles can add random things back, but at least core doesn't).

Since the class+file name of the container is based on a hash of its content, it means that they are now stable also. This should help fix some edge cases/race conditions during cache clears/rebuilds.

(fabbot failure is false positive)

Commits
-------

14dd5d1 Remove randomness from dumped containers
@nicolas-grekas nicolas-grekas deleted the stable-container branch January 4, 2018 16:11
This was referenced Jan 5, 2018
@mcnilz
Copy link
mcnilz commented Jan 5, 2018

After updating to symfony 3.4.3 I get the following error:

In ParameterBag.php line 102:

  You have requested a non-existent parameter "container.build_id".

@nicolas-grekas
Copy link
Member Author

@mcnilz can you share a reproducer please? does this happen if you clear the cache manually?

@mcnilz
Copy link
mcnilz commented Jan 5, 2018

@nicolas-grekas yes on cache:clear and each request.
The stack trace is:


Fatal error: Uncaught Symfony\Component\DependencyInjection\Exception\ParameterNotFoundException: You have requested a non-existent parameter "container.build_id". in 
/app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ParameterBag/ParameterBag.php:102 
Stack trace: 
#0 /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ParameterBag/EnvPlaceholderParameterBag.php(57): Symfony\Component\DependencyInjection\ParameterBag\ParameterBag->get('container.build...')
#1 /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/Container.php(133): Symfony\Component\DependencyInjection\ParameterBag\EnvPlaceholderParameterBag->get('container.build...') 
#2 /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php(1243): Symfony\Component\DependencyInjection\Container->getParameter('container.build...') 
#3 /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php(1196): Symfony\Component\DependencyInjection\ContainerBuilder-> in /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ParameterBag/ParameterBag.php on line 102```

@nicolas-grekas
Copy link
Member Author

A service is used at compile time... Can you share the full stack trace please?

@mcnilz
Copy link
mcnilz commented Jan 5, 2018
In ParameterBag.php line 102:
                                                                                
  [Symfony\Component\DependencyInjection\Exception\ParameterNotFoundException]  
  You have requested a non-existent parameter "container.build_id".             
                                                                                
Exception trace:
 Symfony\Component\DependencyInjection\ParameterBag\ParameterBag->get() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ParameterBag/EnvPlaceholderParameterBag.php:57
 Symfony\Component\DependencyInjection\ParameterBag\EnvPlaceholderParameterBag->get() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/Container.php:133
 Symfony\Component\DependencyInjection\Container->getParameter() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1243
 Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1196
 Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1109
 Symfony\Component\DependencyInjection\ContainerBuilder->createService() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1241
 Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1196
 Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1109
 Symfony\Component\DependencyInjection\ContainerBuilder->createService() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:598
 Symfony\Component\DependencyInjection\ContainerBuilder->doGet() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1239
 Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1196
 Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1109
 Symfony\Component\DependencyInjection\ContainerBuilder->createService() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1241
 Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1196
 Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices() at /app/vendor/symfony/symfony/src/Symfony/Component/De
8000
pendencyInjection/ContainerBuilder.php:1109
 Symfony\Component\DependencyInjection\ContainerBuilder->createService() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1241
 Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1196
 Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1109
 Symfony\Component\DependencyInjection\ContainerBuilder->createService() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:598
 Symfony\Component\DependencyInjection\ContainerBuilder->doGet() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1239
 Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1196
 Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1109
 Symfony\Component\DependencyInjection\ContainerBuilder->createService() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1241
 Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1196
 Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1196
 Symfony\Component\DependencyInjection\ContainerBuilder->doResolveServices() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:1109
 Symfony\Component\DependencyInjection\ContainerBuilder->createService() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:598
 Symfony\Component\DependencyInjection\ContainerBuilder->doGet() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:563
 Symfony\Component\DependencyInjection\ContainerBuilder->get() at /app/vendor/jms/metadata/src/Metadata/Driver/LazyLoadingDriver.php:23
 Metadata\Driver\LazyLoadingDriver->loadMetadataForClass() at /app/vendor/jms/metadata/src/Metadata/MetadataFactory.php:103
 Metadata\MetadataFactory->getMetadataForClass() at /app/vendor/jms/security-extra-bundle/Security/Authorization/Interception/SecurityPointcut.php:84
 JMS\SecurityExtraBundle\Security\Authorization\Interception\SecurityPointcut->matchesMethod() at /app/vendor/jms/aop-bundle/DependencyInjection/Compiler/PointcutMatchingPass.php:149
 JMS\AopBundle\DependencyInjection\Compiler\PointcutMatchingPass->processDefinition() at /app/vendor/jms/aop-bundle/DependencyInjection/Compiler/PointcutMatchingPass.php:65
 JMS\AopBundle\DependencyInjection\Compiler\PointcutMatchingPass->process() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/Compiler/Compiler.php:141
 Symfony\Component\DependencyInjection\Compiler\Compiler->compile() at /app/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:753
 Symfony\Component\DependencyInjection\ContainerBuilder->compile() at /app/vendor/symfony/symfony/src/Symfony/Component/HttpKernel/Kernel.php:634
 Symfony\Component\HttpKernel\Kernel->initializeContainer() at /app/vendor/symfony/symfony/src/Symfony/Component/HttpKernel/Kernel.php:137
 Symfony\Component\HttpKernel\Kernel->boot() at /app/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:63
 Symfony\Bundle\FrameworkBundle\Console\Application->doRun() at /app/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:148
 Symfony\Component\Console\Application->run() at /app/bin/console:28

maybe the JMS stuff

@mcnilz
Copy link
mcnilz commented Jan 5, 2018

Yes. If I disable JMSSecurityExtraBundle (just for testing) it works again.

@nicolas-grekas
Copy link
Member Author

ok, I can reproduce it.
@mcnilz can you please open a dedicated issue? comments on closed PRs cannot be tracked correctly.

@Tobion
Copy link
Contributor
Tobion commented Jan 21, 2018

@nicolas-grekas the proxy-manager uses deterministic proxy properties since Ocramius/ProxyManager#385 but the proxy class name is still random between cache clears. This is because https://github.com/symfony/symfony/blob/master/src/Symfony/Bridge/ProxyManager/LazyProxy/PhpDumper/ProxyDumper.php#L105 uses spl_object_hash which does not sta 6D40 y the same between runs. I tried to replace it with \Symfony\Component\DependencyInjection\ContainerBuilder::hash which indeed makes the container hash consistent. But something is still strange. The cache warmer get executed twice. So doctrine/DoctrineBundle#763 might actually be a problem.

@Tobion Tobion mentioned this pull request Jan 29, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants
0