CiviMail Performance ..

Earlier today Fen Labalme from CivicActions sent an email to the dev list regarding CiviMail performance and the not-so-great number that they've seen on their servers. His complete email and the thread is here. The results were not great and I wanted to verify and check the results on my local powerbook. I ran a few tests with mock data and came back with significantly better results.

• We really have not spent a lot of time optimizing the speed and throughput of CiivCRM. I suspect if we spend a week or so, we should be able to squeeze out a 2-3x improvement. All numbers approximate. Would be awesome for someone from the community to step up and either lead the charge or sponsor the effort or both. Ideally our goal should be to send at least 100K - 250K mails/hour on a 2 or 3 machine architecture on modern hardware. Web server on one machine, db server on the other, email server on the third (or share with db). This basically comes down to 1666 - 4166 emails / minute
• I did a few test email on my local Powerbook with 1G ram and a 2.16 GHz Intel Core Duo. Running php5.2.2 mysql5.0.37 (and on trunk). I also have zend platform installed (they have a free developer version and comes highly recommended!)
• I generated sample data with first name / last name and email. My mail messages had a few tokens in there in addition to the required tokens. all the email addresses ended in example.com, i.e. the local smtp server could not deliver them. I also sent a mime message, i.e. both html and text messages were encoded in the mail body
• sending a 1000 email via civimail took between 55-65 seconds. I also tested this with a 10000 email blast. so we are pretty close to 1000 mails / min which is not bad for a labtop :). I also have mysql logging all queries and not optimized for production at all.
• I wanted to check how much time we were spending handing the mail off to the smtp server. We are spending pretty close to 50% of the time in the smtp transfer. We could optimize this quite a bit by having multiple processes crunch the mail queue at the same time since half the processes are waiting on the smtp server.

Note that the CiviMail architecture has evolved from Groundspring's EmailNow product which currently delivers more than 10 million messages a month So i think we are in a pretty good place to begin with :) I think we can really make civimail industrial strength and rock solid if we invest the time and resources on this. If you or your organization can contribute significantly to this please let us know. I think the community will benefit greatly with more changes and optimization :)

Comments

I'm glad of the up side on CiviMail's sending performance. This is also just the information I need to sell groups on putting resources into an open source solution.

More important than sending efficiency to many organizations starting with CiviMail and having smaller lists is the performance of CiviMail versus spam filters.

One newsletter sent with CiviMail regularly is put into junk mail by Thunderbird, though I've unjunked it more than once.

Could this be related to the token in the reply address? I'm very interested in others experiences and any numbers. Or if you want me to subsribe to your newsletter to see what happens!

ben, Agaric Design Collective