Trellon has been working on a fairly large implementation of CiviCRM for an advocacy group based in DC that was seeking to implement a CRM system that will help them manage their online fundraising and email marketing program. Email marketing is a large part of the organization's outreach strategy, so having a mail system that can deliver their 50,000+ emails in a short amount of time was a necessity.
We explored methods for optimizing CiviMail and decided to follow recommendations made by Lobo on the CiviCRM wiki. We set up the client mail and database server on a dedicated server with an Intel 2 Quad Core 2Ghz processor, 16GB of RAM, and 70 GB of disk space. We wrote some custom scripts to batch emails over to the mail server on cron to increase delivery speeds as well.
We have been able to get some really good times on delivery. I ran a test with the required tokens plus two additional tokens sent to 101,270 sample email addresses, which took 42 minutes. This is roughly 2411 per minute, or 144,671 emails per hour. This is a 50% greater delivery speed that we had set as our optimal goal.
I think with some additional optimization tweaks we can get up to 200,000 or 250,000 per hour. We are going to be exploring other methods to optimize delivery for this client over time as their email list grows. We will post additional tips and code samples for scripts we used once the site goes live around June 15th.
If you have recommendations for ways to optimize CiviMail delivery or benchmarks for CiviMail configurations you have set up, please post them in the comments.
Comments
Sending to 250,000 people in an hour is one thing. But if 5% of those people come visit your site immediately, can you handle that too?
That's more the issue that I'm dealing with right now. Though the architecture that I'm working with is a little different. The mail sending lives on a separate server on a different platform entirely. DB & Web live on the same server.
If all those people came to the site, we could handle it easily. We optimize Drupal to perform under heavy loads, so this is not a huge concern for us. It was more a worry of being able to get emails in people's in boxes
A couple ways we have optimized this site are using the Boost module to generate HTML pages for anonymous users, installing memcache and xcache, and the server comes with load balancing.
The link to Lobo's post on optimizing CiviCRM is broken, any chance someone has the info on steps taken to optimize CiviMail?
Thanks!
check:
http://osdir.com/ml/crm.civicrm.devel/2007-05/msg00037.html
recently fen did report that civimail was sending mail faster than yahoo/aol likes to accept, so he was looking for ways to slow things down!. performance has been improved significantly since 2007!
lobo
We've found significant speed increases in mail delivery if you can sort the outbound messages by domain. Depending upon your configuration, this usually allows the outbound SMTP server to pipeline multiple messages in a single connection to the target host.
Totally great, however, just as a word of warning to all who read this blog post. even after all of these optimizations, rapid delivery could still be affected. The problem here is the receiving mail servers. The time for delivery could be all over the map. in some cases it could be days. The blog post mentions that the test emails were sent to to sample email addresses; in the real world, you will send only to real email addresses. If an org ends up flooding a domain with thousands of emails in a short time, in all likelihood, the org's sending mail server will be greylisted or told to defer emails for a specified time, which in most cases means emails will not be delivered for several hours. also, depending on the number of mail agents that are sending mails, queues could get backed up and mails dropped, so you will need to configure mail queues to deal with very large blasts. (the upper limit on the active queue is 20000 in postfix, which could pose a problem to the scenario in this blog post).
so while these recommendations are much appreciated and thank you very much for the info, the fact is, there is more to rapid delivery than fast mail servers and batching. an org needs to get a good reputation as a sender and even then *guaranteeing* delivery in a short amount of time is very difficult and something I would recommend that nobody guarantees to a client, ever. it's just safer that way.
peace,
-Shane