8000 Performance comparison of 3.3.9 vs 4.0 beta2 · Issue #24808 · symfony/symfony · GitHub
[go: up one dir, main page]
Skip to content

Performance comparison of 3.3.9 vs 4.0 beta2 #24808

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
javiereguiluz opened this issue Nov 3, 2017 · 31 comments
Closed

Performance comparison of 3.3.9 vs 4.0 beta2 #24808

javiereguiluz opened this issue Nov 3, 2017 · 31 comments
Labels
Performance

Comments

@javiereguiluz
Copy link
Member
javiereguiluz commented Nov 3, 2017 •
edited
Loading
Q A
Bug report? no
Feature request? no
BC Break report? no
RFC? no
Symfony version 4.0

We're so close to 4.0 release that I wanted to analyze its performance in comparison with 3.3. If my profiles are correct, then I'm disappointed because Symfony 4.0 is significantly slower, both at dev and prod.

Context

The analyzed application was Symfony Demo. I used master branch for Symfony 3.3.9 and symfony/demo#673 for Symfony 4.0 beta2.

I executed rm -fr var/cache/* and composer dump -o and restarted my web server before each benchmark (3.3.9 dev, 3.3.9 prod, 4.0 beta2 dev 4.0 beta2 prod).

The environment was PHP 7.1, Apache, SSD disk, OPCache enabled, Xdebug disabled, native PHP (no Docker or VM).

Results for PROD environment

(differences are from 3.3 to 4.0; positive values means worse performance)

Page URL Time difference Memory difference Comparison profile
Home / +17% +4% Details
Blog index /en/blog/ +8% +0% Details
Blog show /en/blog/posts/lorem-ipsum-dolor-sit-amet-consectetur-adipiscing-elit +12% +1% Details
Backend index /en/admin/post/ +16% +0% Details
Backend edit /en/admin/post/1/edit +10% +0% Details

Results for DEV environment

(differences are from 3.3 to 4.0; positive values means worse performance)

Page URL Time difference Memory difference Comparison profile
Home / +24% +10% Details
Blog index /en/blog/ +15% +6% Details
Blog show /en/blog/posts/lorem-ipsum-dolor-sit-amet-consectetur-adipiscing-elit +16% +7% Details
Backend index /en/admin/post/ +20% +8% Details
Backend edit /en/admin/post/1/edit +17% +6% Details

In the dev environment, there's an explosion of new function calls. Example:

function-calls
@sstok
Copy link
Contributor
sstok commented Nov 3, 2017

Was this test conducted at an SSD based system or harddisk? And was opcache enabled?

@xabbuh
Copy link
Member
xabbuh commented Nov 3, 2017

Is there a special reason you didn't profile the application with a warmed up cache?

@theofidry
Copy link
Contributor
theofidry commented Nov 3, 2017 •
edited
Loading

@javiereguiluz I think it's important you clarify what kind of performances you are looking for: dev & prod performances.

Obviously dev performances are not as important, although you don't want it to be too slow either.

For production performances, I think the basic optimization steps should be followed otherwise benchmarking them is useless:

  • composer dump-autoload --classmap-authoritative (it's better than optimized)
  • warm up the cache beforehand
  • enable opcache with an acceptable expiration time and cache (cf. this article)

I would also be curious to have the actual blackfire profile :)

@javiereguiluz
Copy link
Member Author

I've updated the description with more information:

The environment was PHP 7.1, Apache, SSD disk, OPCache enabled, Xdebug disabled, native PHP (no Docker or VM).

@xabbuh not sure what do you mean. I cleared the cache and fully regenerated it before performing each benchmark. In any case, if I did it wrong, I did the same for all benchmarks, so it doesn't matter.

@theofidry prod is more important, but dev is super important. "dev" is what you see all day every day as a developer. It can't be slow.

@wouterj
Copy link
Member
wouterj commented Nov 3, 2017

@xabbuh not sure what do you mean. I cleared the cache and fully regenerated it before performing each benchmark. In any case, if I did it wrong, I did the same for all benchmarks, so it doesn't matter.

Your PR description only says you cleared the cache, not fully regenerated it. It's better to always start with a freshly warmed-up cache, as that's the case with your prod app normally as well.

@cordoval
Copy link
Contributor
cordoval commented Nov 3, 2017

don't show percentages, show ms response time for both columns and amount of RAM, percentages are sometimes deceiving

@theofidry
Copy link
Contributor

@cordoval a percentage can be consistent from a machine to another whereas the time can greatly differ

@cordoval
Copy link
Contributor
cordoval commented Nov 3, 2017

you need always a scale, just as a reference, when you take a look at google maps you have a horizontal bar of 1 mile or 1 kilometer to give you the idea of the scale, that is.

@dmaicher
Copy link
Contributor
dmaicher commented Nov 4, 2017

I have some similar results on my biggest app when comparing 3.3.10 vs 3.4.0-BETA2.

I used ab -n 10000 -c 1 ... and:

  • prod env + debug=false
  • composer dump --classmap-authoritative
  • warmed up cache
  • opcache enabled
  • restarted php 7.1 fpm before each run

The only package changed is symfony/symfony.

I tried each version multiple times and the results are consistent to +- 0.5ms.

page mean time per request 3.3.10 mean time per request 3.4.0-BETA2 diff
html page with heavy twig template rendering 40.697 ms 49.621 ms +21,9%
json api endpoint 24.992 ms 29.555 ms +18,2%

@cordoval
@nicolas-grekas
Copy link
Member

As always, reproducer welcomed.

@dmaicher
Copy link
Contributor
dmaicher commented Nov 4, 2017

I'm currently doing binary search to find the commit 😄 Went back quite a few commits on 3.4 and now its the same performance as 3.3. Will let you know once I found out more.

@dmaicher
Copy link
Contributor
dmaicher commented Nov 4, 2017

This is a part of the git history of 3.4:

commit a442e378e18d399e942bcffe0c2e960fcdc0b53f
Merge: 537c496 fa62e50
Author: Fabien Potencier <fabien.potencier@gmail.com>
Date:   Thu Sep 28 16:21:02 2017 -0700

    feature #24362 [HttpKernel] Deprecate some compiler passes in favor of tagged iterator args (nicolas-grekas)

    ...

commit 537c496dfee1d04aab832f8441f8b6cfe7dc2af5
Merge: c7f664c 95358ac
Author: Fabien Potencier <fabien.potencier@gmail.com>
Date:   Thu Sep 28 16:19:46 2017 -0700

    minor #24366 [Lock] Use cache connection factories in lock (jderusse)
    
    ...

The last commit where 3.4 is equally fast as 3.3 is 537c496dfee1d04aab832f8441f8b6cfe7dc2af5. The commit after (a442e378e18d399e942bcffe0c2e960fcdc0b53f) slows it down for me.

537c496...a442e37

@nicolas-grekas any idea?

@nicolas-grekas
Copy link
Member

No idea, if anything, this should impact compile time only, but the bench are about runtime diffs?
A Blackfire comparison could maybe help?

@dmaicher
Copy link
Contributor
dmaicher commented Nov 5, 2017

Ok after some profiling, debugging and slack discussions with @nicolas-grekas I think we found the issue:

537c496...a442e37#diff-687bdbb38a4dc672ca2a79f23e764892L127

Replacing this compiler pass with a "tagged argument" is now also done for debug=false whereas before it was only added for debug=true. Means now the ResourceCheckerConfigCache constantly iterates over all resourcecheckers and checks if the cache is still fresh which obviously takes time.

I will work on a PR to fix the issue.

@dmaicher dmaicher mentioned this issue Nov 5, 2017
Merged
@dmaicher
Copy link
Contributor
dmaicher commented Nov 5, 2017

@javiereguiluz can you benchmark the symfony demo again with this change?

https://github.com/symfony/symfony/pull/24824/files#diff-0e793081ceb720201745c982a568903fR364

For me the performance is now pretty much the same again as on 3.3.

fabpot added a commit that referenced this issue Nov 5, 2017
@fabpot
bug #24824 [FrameworkBundle][Config] fix: do not add resource checker…
dc25f9a 
…s for no-debug (dmaicher)

This PR was merged into the 3.4 branch.

Discussion
----------

[FrameworkBundle][Config] fix: do not add resource checkers for no-debug

| Q             | A
| ------------- | ---
| Branch?       | 3.4
| Bug fix?      | yes
| New feature?  | no
| BC breaks?    | no
| Deprecations? | no
| Tests pass?   | yes
| Fixed tickets | #24808
| License       | MIT
| Doc PR        | -

As mentioned within #24808 replacing the `ConfigCachePass` here 537c496...a442e37#diff-687bdbb38a4dc672ca2a79f23e764892L127 with a tagged iterator argument resulted in resource checkers being added even if debug=false.

This resulted in a performance drop as on every request all the checkers have been checked.

This restores the previous behavior and does not add any checkers if debug=false.

Commits
-------

645f712 [FrameworkBundle][Config] fix: do not add resource checkers for debug=false
@fabpot fabpot closed this as completed Nov 5, 2017
@javiereguiluz
Copy link
Member Author

@dmaicher I've tested your patch and it improves things a lot. Instead of ~15% worse performance, now I see just ~4% worse performance.

But your patch is only for prod, so dev is still ~20% slower. Maybe somewhere we could optimize things to not make so many new function calls:

function-calls

@dmaicher
Copy link
Contributor
dmaicher commented Nov 5, 2017

@javiereguiluz ok thanks for confirming. My patch does not have any impact on dev environment indeed.

Maybe we should reopen here? the +20% for dev are indeed a bit too much I guess 😢

@steevanb
Copy link
steevanb commented Nov 6, 2017 •
edited
Loading

I have similar results here : phpbenchmarks.com

Each Symfony version drop down query per seconds :(

@nicolas-grekas
Copy link
Member

@steevanb interesting! Can you show me where cache warmups happens for Symfony? I don't see it in https://github.com/phpbenchmarks/symfony-3-4 but maybe I missed it?

@javiereguiluz javiereguiluz mentioned this issue Nov 6, 2017
Closed
@steevanb
Copy link
steevanb commented Nov 6, 2017 •
edited
Loading

You can find benchmark protocol here :
http://www.phpbenchmarks.com/en/benchmark-protocol

#1 Include init_benchmark.sh, and call init() ( https://github.com/phpbenchmarks/symfony-3-4/blob/master/init_benchmark.sh for Symfony 3.4).
#2 Restart php-fpm service, to reset OPCache etc.
#3 Cache warmup isn't enough, it do not generate all caches. So, i make a first "non saved" benchmark, with 1,000 calls (instead of 50,000 for the real one), concurrency 1, to be sure all caches and OPCache are OK.
#4 Benchmarks with 50,000 calls, for concurrencies 1, 5, 10 and 20. For now, only Hello World and Rest API are benchmarked, i'm on 10 news list page benchmark.

@nicolas-grekas
Copy link
Member

@steevanb there is one step missing: calling bin/console cache:warmup, this one is required to generate some caches that will not be created by just hitting the first 1000 times.

Copy link
Contributor
dmaicher commented Nov 6, 2017 •
edited
Loading

Just FYI: I benchmarked again two pages (html page and a json api endpoint mentioned above) with 3.3.10 vs 3.4.0-BETA3 using ab -n 10000 -c 1 ... and prod env and I cannot see any relevant difference between both. Equal performance in my benchmarks.

@sroze
Copy link
Contributor
sroze commented Nov 6, 2017

calling bin/console cache:warmup, this one is required to generate some caches that will not be created by just hitting the first 1000 times

@nicolas-grekas interesting. Which ones?

@nicolas-grekas
Copy link
Member

eg the annotations' cache

@steevanb
Copy link
steevanb commented Nov 6, 2017

Humm, didn't know that ... Are you sure cache:warmup do something that calling url will never do ?

If it's true, for now i'm lucky, i don't use annotation. But that's weird.

@nicolas-grekas
Copy link
Member

Yes I am :)

@steevanb
Copy link
steevanb commented Nov 6, 2017

Ok, good information ! I don't think it will change my results for now, but i will add it for news benchmark.

Thanks !

@theofidry
Copy link
Contributor

Well it's unlikely to be an issue unless for benchmarks and read-only environments which is why the issue can easily get unnoticed

@sroze
Copy link
Contributor
sroze commented Nov 6, 2017

@nicolas-grekas is there any reason for that? Shouldn't we generate the caches at the first request?

@nicolas-grekas
Copy link
Member
nicolas-grekas commented Nov 6, 2017 •
edited
Loading

That's always been the case in fact: cache warmers are heavy processes. Eg parsing ALL Twig templates on the first request, or ALL annotations, etc. would mean paying a high price in dev (or prod).
Then, having ALL annotations (there are others) at hand still allows to generated an even more optimized cache pool (a single PHP file that OPCache can cache.)
I don't think we can improve this process, as it's logical. But as @theofidry said, that's not bad anyway, you just won't have the extra perf benefit you could achieve. Not a big issue usually.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Performance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
0