CiviCRM / CiviEvent handled thousands of registrations in one day

Every year in June, around the 10th day, a commemorative event happens in Akron, Ohio - the annual celebration of the founding of Alcoholics Anonymous. Hosted at the University of Akron, over 10,000 participants from around the world gather to celebrate the founding of this wonderful fellowship. In recent years, registration for this all weekend event has moved from mail-in forms to an online registration process. Online registration challenges from the past few years had made us seek a solution to that could handle a surge of registrations in the first 24 hours, and collect all the necessary information required by the University. In past years, Zen Cart could handle our registrations - it may have even been able to handle it this year, except we needed to collect more infomation than in years' past, and we wanted to still be able to allow multiple registrations per transaction. One serious issue from last year was the server became overloaded as thousands of registants came to the site to register at midnight. We had 42 days to pick a solution, and implement. First, we decided to move the site to either a dedicated server or VPS. We tested various VPS solutions. One issue we encountered was the dreaded fopen error. In the end, we chose VPS.net as our hosting provider mainly because they did not have the fopen issue we had at other providers AND we had the ability to beef up the server for a few days, then return the server to normal. Now we needed software. We used virtualmin GPL for a hosting control panel, but we moved from suexec to mpm-itk. No server overload issues that way. Now, we have a server and a control panel (we need this for the technically challenged). We found CiviCRM via a google search, and did some research for a few days. OK, we liked it, but could we install it, test it, customize it, and move it into production in 39 days time? We did, with some technical assistance via IRC - thanks Lobo. We installed Joomla 1.5.15, easier for our people to work with than Drupal. Then we installed and configured CiviCRM 3.04, and upgraded as we went along to 3.13. The final product is http://foundersdayregistration.akronaa.org. During testing, we uncovered some performance issues. We added php5-memcache and memcached to our installation, added an sql cache plugin to Joomla, and turned on system caching in Joomla. This solved those issues. We also had to customize our apache config files and my.cnf for our system. Having tested again, performance was no longer an issue. So, with the server properly configured and tuned, we tested some more. We also created some views and installed a plugin to Joomla to extract on a daily basis all the information the treasurer and secretary would need. The first day of registration was March 1st. Registration began at 9 A.M. EST. The final system confuration was as follows: Debian Lenny Virtualmin GPL Joomla 1.5.15 CiviCRM 3.1.13 php5-memcache, apache2-mpm-itk (these are the basically the only deviations from a standard Virtualmin install) sql cache and memcache in Joomla Apache2 configured with mpm-itk max clients as memory permitted and mysql configured for the memory we have (4 gigs on day one, 768 after day 4) So, 42 days from the beginning of installation, we went live. In the first hour, we handled 800 registrations at 25% cpu and memory usage. 9 days later, that's today, we have handled thousands of registrations online, with no issues whatsoever related to CiviCRM or the server. We back up the database hourly to S3. An email of the days transaction is sent at 1 AM to the appropriate people, and we just sit back and say Thank you to all who made this possible.

Comments

Thanks for sharing.

Did you keep stats on the number of simultaneous visitors at peak time ?

X+

Great case study -- thanks for sharing. Glad to hear it was such a great success!

I do have the exact number of transactions at peak - peak was within the first 10 minutes of registation. Somewhere in the neighborhood of 450 within 10 minutes was what actually occurred. I'd have to pull from the database to be positive, but that's a good rough estimate.

Great write up - thanks for posting this - would be interesting to have some comparisons of how others are managing peak flows - or if they aren't coping, what their specs are/were.

Thanks for the write-up but now I wonder if our VPS solution will suffer from it with higher transaction volume ...

It's a php error that occurs on some VPSs with Joomla during an install of a component, module, etc. Hence, the reason there's an alt install of CiviCRM for Joomla. I don't remember the root cause of the issue, it's something to do with not being able to uncompress the files and folders into the proper directory structure. Probably a missing module or misconfiguration of php.ini, but I just didn't feel like figuring it out. The alt install worked for me.

In the Drupal world that would be considered a traffic load far too small to require the use of Memcache given the beefy hardware that you've got.

That may be so. The biggest bottleneck I saw was that in my Joomla configuration, the registration page hits the database for every visitor to get a session id.

Thanks for sharing a really clear case study that helps in assessing peak server load requirements.

It is also interesting to see how people are influenced between Joomla! and Drupal as a host CMS. Our clients tend to have the same feeling as your team, that Joomla! is more user friendly.

Until the Joomla! install of CiviCRM becomes a "front end" component and/or Joomla! introduces greater access control, the access control issues around using Joomla! for CiviCRM mean that many of our clients end up wanting Joomla! for managing their website and Drupal for CiviCRM - so we end up with Joomla! CMS and a Drupal/CiviCRM CRM and a bunch of custom Joomla! components talking to Drupal/CiviCRM. This is currently not easily workable if clients wish to run lots of different events through their Joomla! website (you usually end up with a Drupal/CiviCRM registration/payments site).

As different CMS continue to emerge and develop (eg WordPress), increasingly there may be a need to standardise the CiviCRM admin interface around a single framework/CMS, but with the underlying objects being more extensively accessed through web services from a range of other CMS so that generalised extensions can be built for those CMS that will talk to CiviCRM, for example through an event ID and key over SSL, perhaps with a username and password as well.

good info thanks

I would be interested to hear more on which Joomla plugin you used for sql caching.
I loooked on the site and found som errors here: Content Encoding Error
http://foundersdayregistration.akronaa.org/schedule-of-events/friday-events.html

hope you can resolve quickly

Query Cache is the component we use for sql caching. As for the encoding error, that's an old page, no longer there, and we have decided not to fix those issues not directly related to registration until after registration has closed on May 15. At the moment, a functioning registration system is more important than fixing technical issues that do not actually affect registration.