Integrating Sendgrid into CiviMail

Sendgrid is a relatively new cloud-based service that takes care of mailing list SMTP delivery and attempts to get the highest possible deliverability rate. We didn't want to have to manage our own SMTP server here at the Participatory Politics Foundation, so when we migrated to CiviCRM recently, we decided to take the sendgrid route for SMTP. Integration with CiviMail is pretty straightforward: all you really have to do is change your SMTP settings in CIviMail to match your sengrid credentials. But sendgrid is different from a typical SMTP server in the way it handles bounces and spam reports. First of all, sendgrid uses a reputation score to keep spammers away -- the gist of it is that 80% or more of your messages must deliver without failures. If you go below 80%, your sendgrid account can be suspended. Furthermore, they don't send any bounce or spam report messages back to you, the sender. And they don't attempt redelivery. Instead, they record the e-mail addresses that bounced and prevent further messages to those addresses from being delivered -- so your score will never drop if you try to send further messages to bounced addresses. So, we could actually stop once we change the SMTP settings, but then we'd keep trying to mail to all of those bounced addresses, and that's kind of annoying even if it doesn't impact our sendgrid reputation. It would be better to keep the bounce and spam report lists on sendgrid in sync with CiviCRM, and mark CiviCRM contacts as "on hold" when they bounce. Sendgrid has a simple Event API that will push notifications of bounces and spam reports to a URL on your server. We wrote a PHP endpoint to handle these events. You can grab our code here. You will want to change the civicrm path and unique key, which you'll provide to sendgrid in your post notification URL. Make this script publicly accessible and test it out like this: $ wget -O - -S --post-data 'email=bouncing-contact@hotmail.com&event=bounce' 'http://yoursite.org/sendgrid-notifier.php?key=CHANGE_THIS_TO_YOUR_UNIQUE_KEY_FROM_ABOVE' Once it's working to your satisfaction, you can point sendgrid to your URL, and away you go! Some changes we'd love to see in the future:

• Turn this into an official module/plug-in?
• Sendgrid also sends a bounce reason message and a bounce status code in the push notification -- it would be nice to record these somehow.
• ... and, for some temporary failures, we could tell sendgrid to remove the address from their bounce list, via their web api.

I welcome your feedback and suggestions for other improvements. PS. The sendgrid Event API also notifies you of opens, clicks, and unsubscribes, but we turned all of those off in our sendgrid filter settings, since CiviMail already provides all of these for us.

Comments

HI, seems that sendgrid tracks as well the open rate. Have you compared it with civimail internal tracking ? Do you get the same % and could you share what it is with us?

We've disabled all of that tracking on their end and are just going with CiviCRM's stats -- but if we ever do turn on open tracking, I'll let you know how it goes.

Hi,

It'd be nice you try to enable the opening rate tracking once or twice to confirm if that's the same as civi (it should, obviously).

X+

Looks great. Would be great to have an option beyond CiviSMTP for email deliverability.

This is great work, guys! I'm about to start using it with a client.

I made some changes to your code to use a CIVICRM_SENDGRID_KEY constant defined in the civicrm.settings.php file rather than hard-coding the shared secret. I have it up on github here: http://github.com/cap10morgan/civicrm_sendgrid

I'm also discussing ways we can make this more secure with the SendGrid folks. I'd rather not have to have the shared secret go across the wire in cleartext, right there in the URL. They say they're discussing my suggestions and will get back to me. I'll update this code if/when they provide a more secure way to verify that the post events are really coming from them.

Once that's done, it will probably be easiest to just use the CIVICRM_SITE_KEY as the shared secret. But since the shared secret gets exposed in the URL currently, I didn't want to use site key just yet.

It strikes me that there should be a link to this module on github https://github.com/emotive/CiviCRM-sendgrid-integration

Is there any other info on this project? It looks great but there is zero docs as far as i can see.

It has been a few months since using this how to and script (big thanks by the way!).
Seems the pricing has changed and the Event API is no longer included.
http://sendgrid.com/pricing.html

Still a great service though.

I know this is way after the fact but for anyone who is interested the Event API is still available, but you have to be on the Silver package.  Any with a $70 a month difference between bronze and silver I'm guessing that's not great news for most people.

The good news is that there is a way of obtaining bounce information.   View this page for the full details:

http://sendgrid.com/docs/API%20Reference/Web%20API/bounces.html

Cheers