Drupalcon Session: Performance testing The Economist Online using The Grinder

Performance testing The Economist Online using The Grinder

Presented by David Strauss of Four Kitchens

The only way to get close to accurate load testing for a real site is to use a replica of production servers and infrastructure.  You must have a solid testing framework that includes hardware, software & tools, and scripts and configurations specific to your site's needs. Load testing should not be done on production "live" sites or servers.

Hardware that must be added in addition to standard web servers, database servers, etc. is a load testing controller and load injectors.  The load injectors send the web page requests to the dev servers, and the testing controller manages the load injectors.  You need multiple load injectors in order to come close to simulating a real-world scenario.

The point of testing is to find out where the breaks or bottlenecks happen, to find see where problems will be happening and be able to measure their severity. Your next step after testing is to track down the cause of the problem, develop a solution, implement it and run the performance tests again to see if the problem is improved.

Tools

The Grinder

• Load testing software
• Written in Jython

Using The Grinder

• Write some unit tests as sanity checks
• Test actions performed by logged-in users (voting, viewing, editing)
• Test variety of data, some was behind a paywall
• Test hard enough that you can actually break the site. (This is done on development machines, not the live site.)
• Write tests that simulate a live load
• Some Issues that can be discovered
  • Menu Rebuild Stampede
  • Variable Cache Rebuild Stampede

Innotop

• MySQL monitoring tool

VarnishHist

• Included when install Varnish (a reverse proxy server, usually installed to improve performance)
• Histogram view of your Varnish cache
• See when it is having to send lengthy requests back to Apache

HTTP Traffic Proportion Analyzer

• Safe to run on live sites.
• Analyze traffic patterns on live sites. Use this data to help you develop good tests based on actual traffic, not just guesses.
• Can group into buckets (usually based on section of the site, but can be grouped by many custom rules.)

Further Notes

They spent about ten times the amount of time developing and running the tests, compared to fixing the problems that were discovered in running the tests.  Working out the fixes was fairly quick, once the actual causes were known.

More Grinder tips on the Four Kitchens wiki.