APC vs Zend Optimizer+ Benchmarks with Symfony2

If you are following some of the biggest names in the PHP scene you may have seen that Zend Optimizer+ is now open-source on Github at https://github.com/zend-dev/ZendOptimizerPlus and it seems that will be included in PHP 5.5 distribution.

This is awesome news as we will finally get a PHP Accelerator bundled with the main language distribution and there should not be (let’s hope) weird issues like the ones happening in early versions of PHP 5.4 with APC that has prevented massive adoption of the new stable version of our favorite language.

There are some really promising performance tests available here but at the time of writing this post there are neither Symfony2 nor Zend Framework 2 benchmarks so I decided to grab the extension and do my own performance tests.

So… let’s see the results

Hardware and software used

What is tested? Basically the simplest application you can develop with Symfony2 (Version 2.2.0).
Composer was run with -o flag which, as you should know, optimizes autoload creating a classmap (except the test which is said that it war ran without it)

Machine used is my laptop, basically a Macbook Air with 1.8 Ghz i5 processor, 8Gb RAM and SSD HD

PHP version is 5.4.12, APC is 3.1.13 and OS is MacOSx Mountain Lion

And the tests are run with Apache Benchmark against a Macports default Apache installation with these settings:

ab -n 5000 -c 10 <symfony2_url>

A Symfony2 Hello world

I’ve seen some “Hello world” performance tests around the Internet using Twig but I guess the fairest test for a Symfony2 minimal footprint is some controller like this one:

<?php
 
namespace RicardClau\Bundle\BenchmarkBundle\Controller;
 
use Symfony\Bundle\FrameworkBundle\Controller\Controller;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Route;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Template;
use Symfony\Component\HttpFoundation\Response;
 
class DefaultController extends Controller
{
    /**
     * @Route("/benchmark", name="benchmark")
     * @return \Symfony\Component\HttpFoundation\Response
     */
    public function benchmarkAction()
    {
        return new Response('APC vs ZO+');
    }
}

Results with Xdebug

PHP 5.4.12 without APC, but with composer optimization

~ 37 req / s (really poor performance)

PHP 5.4.12 with APC 3.1.13 no custom, but without composer autoload optimization

~ 155 req / s

PHP 5.4.12 with APC 3.1.13 no custom

~ 180 req / s (nice boost with composer autoload optimization!)

PHP 5.4.12 with APC 3.1.13 and apc.stat = Off

~ 195 req / s

PHP 5.4.12 with APC 3.1.13, APC Universal ClassLoader and apc.stat = Off

~ 195 req / s (almost same performance if you use composer autoload optimizer and stat is set to off)

PHP 5.4.12 with Zend Optimizer+ no custom

~ 211 req / s

PHP 5.4.12 with Zend Optimizer+ recommended settings

zend_optimizerplus.memory_consumption=128
zend_optimizerplus.interned_strings_buffer=8
zend_optimizerplus.max_accelerated_files=4000
zend_optimizerplus.revalidate_freq=60
zend_optimizerplus.fast_shutdown=1
zend_optimizerplus.enable_cli=1

~ 220 req / s

PHP 5.4.12 with Zend Optimizer+ extreme settings

All of the above plus
zend_optimizerplus.save_comments=0
zend_optimizerplus.enable_file_override=1

~ 202 req / s (which is really weird to be lower actually but I’ve tested it many times and seems to happen in my setup)

Please note that to make this last test work properly, you should have your Symfony2 cache warmed before the test if you are using annotations. Otherwise these settings will make your annotations get ignored!

Results WITHOUT Xdebug

(thx to @pborreli suggestion)

PHP 5.4.12 without APC, but with composer optimization

~ 40 req / s (really poor performance)

PHP 5.4.12 with APC 3.1.13 no custom, but without composer autoload optimization

~ 249 req / s

PHP 5.4.12 with APC 3.1.13 no custom

~ 260 req / s (nice boost with composer autoload optimization but less impressive than before!)

PHP 5.4.12 with APC 3.1.13 and apc.stat = Off

~ 280 req / s (still nice boost)

PHP 5.4.12 with APC 3.1.13, APC Universal ClassLoader and apc.stat = Off

~ 280 req / s (almost same performance again if you use composer autoload optimizer and stat is set to off)

PHP 5.4.12 with Zend Optimizer+ no custom

~ 307 req / s

PHP 5.4.12 with Zend Optimizer+ recommended settings

zend_optimizerplus.memory_consumption=128
zend_optimizerplus.interned_strings_buffer=8
zend_optimizerplus.max_accelerated_files=4000
zend_optimizerplus.revalidate_freq=60
zend_optimizerplus.fast_shutdown=1
zend_optimizerplus.enable_cli=1

~ 337 req / s

PHP 5.4.12 with Zend Optimizer+ extreme settings

All of the above plus
zend_optimizerplus.save_comments=0
zend_optimizerplus.enable_file_override=1

~ 312 req / s (again slower)

Some Excel graphics

APC vs Zend Optimizer Plus in Symfony2

Conclusions

First of all, I would like to say that these results must be taken with a pinch of salt as they were made in my personal laptop which is NOT a production server. Operating System is MacOSx so they might differ a little bit on a Linux machine prepared for production.

Having said that, there are some interesting facts:

- With APC, Symfony2 performance is dramatically boosted!. What would we do without PHP Accelerators? Thus, the importance of having an accelerator bundled with the standard distribution

- With composer autoload optimization we might get a benefit of ~20-25% even with a small test like this. Don’t forget it in your production environments!. However, disabling xDebug made this benefit get reduced to ~10%.

- With apc.stat set to Off, we get an extra boost of ~8-10% even in such a small test. This gets bigger when the amount of files opened increases. Don’t forget to consider it. However, remember that whenever a change is uploaded you should reset APC so that your changes are noticed.

- Xdebug make things SLOW. However, most of the % are similar, except maybe the performance boost with composer autoloader optimizer. Please check that you have not it enabled in your production servers!

- Zend Optimizer+ is faster than APC, even with the default installation. With recommended settings and in the best scenario we get our requests to be 10 times faster than PHP with no Accelerator!

- With recommended settings we get a performance boost of ~11% with xDebug and of ~15% without it, if we compare with APC best performance, but there seems to be something wrong with the extreme settings which were getting worse performance. This was consistent in 5 test I did in different moments, so it seems we can discard that there might have been some background process consuming CPU.

- Is it worth to switch? Well, it depends, at least at the moment of writing these lines. If you are not using APC as a local storage cache, it seems that you will get an extra boost for free, which is always good. Also, if it is bundled with the standard distribution, there is one less PECL package to get into account for automated machine deploys. However, if you are using APC as local storage in your application, I don’t think the performance boost is worth if we’re losing the easy caching that APC provides.

- As Lukas Smith states in his comment, there is some work being done with ApcU (which can be seen at https://github.com/krakjoe/apcu) which will be APC minus the byte choice cache. If they make it, with the launch of PHP 5.5 we will have an optimizer bundled with PHP and we will still be able to install ApcU. This will allow us to benefit of the performance boost in ZO+ while not losing APC storage and upload files hook. It definitely seems PHP 5.5 will be another amazing version of our favorite language with Generators and Zend Optimizer+ being the biggest improvements!

Please, feel free to add any comments to these tests!

21 Comentarios.

  1. there is work being done on APCu which will be APC minus the byte choice cache (ie. user cache, upload progress hook)

  2. Ricard Clau

    Good to know! I wish I knew some C to contribute :???:

    I’ve added this in the blog entry, thanks Lukas for sharing it!

  3. Thanks for doing these benchmarks. Interesting results!

  4. Thanks for the benchmarks, really interesting.ZO seems to be really efficient.

  5. Thank you for your benchmarking. That’s awesome

  6. I constantly get

    ab -n 5000 -c 10 -r http://google.com/
    This is ApacheBench, Version 2.3
    Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
    Licensed to The Apache Software Foundation, http://www.apache.org/

    Benchmarking google.com (be patient)

    Test aborted after 10 failures

    apr_socket_connect(): Operation already in progress (37)
    Total of 3 requests completed

    Any ideas?

  7. Ricard Clau

    Hi @Zyava

    It may happen that Google detects Apache Benchmarks somehow and bans temporarily your connections

    • I tried on my website and my dev VM and it doesn’t work too. Google is just as example.

      • Ricard Clau

        Oh ok, sorry, then the only thing that comes into my mind is there is some problem in your local Apache installation.

        No further ideas, ab has always worked for me both in Mac OSx and Ubuntu :cry:

  8. Ricard Clau

    ulimit is for your local open resources, having ab with -c 10 parameter would only create 10 concurrent connections on your side.

    My guess is that Google bans the connection

  9. ab just randomly fails. It can fail on 2 request or on 590 request, it is random. And I’m not the only one who have this problem: http://syshen.cc/post/10922563476/performance-testing-to-node-js-server-on-mac-lion
    Seems like this bug was introduced in lion and still exists.

  10. I found that I had some old ab version in
    /Applications/MAMP/Library/bin/ab which ab which was used instead of default one :twisted: . Deleted it and with system-wide one, compiled with help of http://superuser.com/questions/323840/apache-bench-test-erroron-os-x-apr-socket-recv-connection-reset-by-peer-54
    it works. At last!

  11. Ricard Clau

    Happy to read this! Sometimes MacOSx makes weird things like this :)

  12. I believe for real applications testing -H “Accept-Encoding: gzip,deflate” option should be added.

  13. Krzysiek Zalasa

    Hi Ricard,

    It’s very helpful article, thanks! Can You attach your APC config from your test?

    Regards,
    Krzysiek

  14. Ricard Clau

    Hi Krzysiek

    APC version was 3.1.13 and all values were default
    The only exception was apc.stat value that was changed in some tests to show the performance difference between On and Off.

    Regards!

  15. Thomas Lallement

    Could it be possible to have a benchmark with a little more complex code? (create and render a simple form with twig for example)

  16. Great post Ricard, and quite helpful. Thanks :)