Recently we noticed the Make-It-Happen on CiviCRM.org which proposed to integrate CiviCRM with Clickatell and other SMS gateways. I approached dlobo on IRC and asked if he would be interested in seeing the code we had developed that would do some of what the MIH proposed. He asked why we hadn't mentioned it before and my sheepish response was that, even though it worked, it felt like it was a duct tape solution not ready for public consumption. However, at his request, I offer it up for your collective use.
The Problem:
A year or two ago a client approached us and asked if we could integrate SMS into his CiviCRM site. He was working with college bound teenagers who, apparently, don't read email anymore. (Kids these days...) What he needed was a way to communicate with them over SMS with the following features:
- All inbound and outbound messages must be tracked
- "Real" SMS should be used as opposed to utilizing the 123456789@carrier.com email address that is usually available.
- The teenager should be able to opt out, if desired, with little or no interaction from the site staff.
- He needed the ability to respond while on the road using his smart phone.
- Requirement 4 and requirement 1 had to coexist. (IE anything he sent back via his phone still had to be tracked)
- The solution needed to be reasonably cheap. He was willing to pay, but it had to be reasonable.
We added a seventh requirement of NOT hacking CiviCRM Core (if at all possible)
For those that are curious, we chose not to use the 123456789@carrier.com method because while it is free to send messages they are rarely identifiable as coming from a particular source and some non-smart phones make it very difficult to reply to them in any meaningful fashion. Additionally, and ultimately more importantly, some carriers seem to "lose" email-to-sms messages with surprising regularity.
With these requirements in mind we started looking.
The Tools:
What we finally settled on was to utilize Clickatell's SMS services (http://www.clickatell.com). In our case we were very interested in their small business packages. They offer a Central API package which, as the name suggests, allows a developer to access both inbound and outbound SMS via a remote API. Be careful when setting up your account as they offer other packages which only allow you to administer your SMS messages via their web site. We made this mistake during development and while the Clickatell staff was EXTREMELY helpful it was an unneeded annoyance. The Central API package we chose costs $9.95 a month and provides 500 outbound credits, free incoming messages, a free two-way "phone" number, and automatic "STOP/REPLY" management (satisfying requirement 3 for us). Our account IS limited to US numbers only but there are other packages that offer international only or a mix. You can also get more credits per month if you prepay for longer time frames. As an aside, each SMS costs a certain number of credits but at least US phone number to US phone number it appears to always be 1 credit.
The Implementation:
With the tools and the requirements in place we decided to implement the SMS message sending in a method similar to sending email when NOT using CiviMail. That is to say, you can send a batch message to the results of a select in CiviCRM. Each SMS shows up as an activity on the contact record as type "Text Message (SMS)" with a "subject" of either "SMS Sent" or "SMS Received". Unfortunately we didn't have the time then to enable the ability to enter an activity on the contact directly and have it send out the SMS. There is LIMITED support for "tokens" in the outbound message. We only needed First and Last name replacement for things like "Dear {{firstname}}..."
The Clickatell API supports delayed sending (with caveats), invalid after timestamps (with caveats) and status updates. Status updates and inbound messages are handled via a callback from Clickatell to our system. We also support the three default priority channels that Clickatell provides meaning that in theory, a high priority message received by Clickatell after some medium priority messages would be sent before those messages assuming the medium priority messages were still queued for delivery. In practice, we've never filled their queue sufficiently to need this.
So far, this satisfies all but requirement 4 from the client. For that last part we had to think outside the box a little and came up with the following method. When a message is received inbound from Clickatell the activity is created recording the message and then an email is sent to an administrator with a link to a "tiny" web page. This page shows the name of the contact the message is from, the date the message was received and the message itself. A small textarea is provided to allow a response to be composed and sent. When submitted the message is then sent using the existing Clickatell code. The reason this method was chosen is that the administrator (in our case) always had EITHER their email open when sitting at their desk OR their smart phone with them when out of the office but they rarely had both. Since they received their email on their phone AND their computer they could delete the notifications as they processed them and because they were using IMAP it would "disappear" from the other email client.
The Caveats:
As mentioned above delayed sending is available in the Clickatell API but ultimately that is limited to when the message is handed off to the carrier or gateway. If I request that Clickatell deliver the message after 1pm on the 1st that is when Clickatell will deliver the message to the carrier. The carrier could fail to deliver the message for any reason at any point. Similarly, Clickatell supports an Invalid After timestamp. This could be used to stop attempting to deliver the message "Don't forget the meeting is at 6pm" after we actually get to 6pm. As with the delayed sending feature Clickatell can only stop attempting delivery if it has not handed the message off to the carrier. Related to this, the Clickatell Status is fundamentally the status of delivery to the carrier or gateway and NOT to the phone at the other end.
CiviCRM by default doesn't support custom fields against the SMS activity type so we had to make a database change to support that. Trivial but it was a bit scary. Ultimately by making this change we were able to add fields to support tracking the Clickatell ID (useful for searching their system, and potentially a manually entered SMS by its absence), the Clickatell Status (Sent, Failed, etc), the Clickatell Time, and the cost in credits.
CiviCRM does have the option of a "Primary" phone number but for our client that implied the number to call not necessarily the number to send an SMS to. Because our client has a fairly small and controlled database we chose to "cascade" through locations to find a valid phone number marked as Mobile. In other words, our configuration file has an array that lists the location number and we simply loop over that. This means we only have a granularity of Work before Home or similar but if there are multiple Work numbers tagged as Mobile we don't have fine grained control over which one will be used.
We also assume that a cell phone number is tied to one, and only one, contact. This was the easiest way to match an incoming message to a contact. In our client's case this meant, at worst, that a parent or sibling might get tied to a message and that was an acceptable risk. We also assume US numbers when we format the messages for sending. For example, we strip off leading ones on a phone number.
Finally, we hard code into our configuration file the ID of the "user" that the activity that holds inbound messages is associated with and we also hard code into the configuration files the email address to send notifications too. In both cases this was acceptable because this is a one man shop. I could easily see a need to look at the contact to determine, for example, who the contact's assigned "case manager" was or to use some other dynamic criteria such as geographical area and assigned salesperson or anything else. You CAN enable SMS notification of the responses however please keep in mind that each SMS sent costs credits even if it is to yourself.
The Wish List and Things to Extend:
If someone were to implement this directly in CiviCRM I would want to see the following:
The biggest missing feature is a clean way to indicate the primary number for a given phone type. This extends beyond just our SMS use case. I might have a primary home number; if I work multiple jobs I might have a primary work number; I may have a work cell and a personal cell, etc. Right now we don't have a way to distinguish the difference but I really can't think of a clean way to implement this either. I suppose one way would be to add Phone Types "Primary Phone", "Primary Mobile", "Primary Fax" and so on but as far as I know this would currently require end user training to make sure that only one phone number had the "Primary Phone" tag.
Right now you can only send bulk messages (even if it's just to one person). In theory, a cron job should be able to watch the activity table for SMS activities with no Clickatell ID and send it through. We didn't NEED that feature so we reserved manual SMS activity entry for errant messages sent outside of this system and made it a "training" issue.
From a code standpoint, this was tested against a CiviCRM 3.4.8 install under Joomla 1.5.25 and uses the v2 CiviCRM API. I assume it will need some "love and care" prior to being used with any of the CiviCRM 4 branches or the v3 API. I also have not tested the code with Drupal or WordPress but seeing that it utilizes mostly backend code, which I believe is mostly shared, and the CiviCRM API it would seem to me to be a safe proposition. Your mileage, however, may vary. When this is tested against Joomla 1.7, Drupal, WordPress or CiviCRM 4.x, which I keep meaning to get to, I will certainly post the results here if someone doesn't beat me to it.
The Files:
Anyone who would like to get a copy of the files, such as they are, can download them from our website at http://www.thimbleweedconsulting.com/projectfiles/ProtoCiviSMS-1.0.zip. They are free for you to use. We ask only that you send any improvements and bug fixes back our way.
Included with the files is an Installation Instructions.txt that also includes some information on how to set up the Clickatell callbacks.
Matt Neimeyer
matt@thimbleweedconsulting.com
Thimbleweed Consulting
Comments
Thanks for sharing. I think people are often reluctant to share code when it's not of the quality they would like it to be, or when it has a lot of hard-coding in it, or some rough edges. It's a shame because we miss out on the 80% they've done a really good job of because they don't think the 20% if polished enough yet.
From me, thanks for sharing too! I wholeheartedly agree with Eileen that our own quality standards often keep us from sharing worthwhile stuff....I know I am always much more aware of my shortcomings if I share any code :-)
From me, thanks for sharing too! I wholeheartedly agree with Eileen that our own quality standards often keep us from sharing worthwhile stuff....I know I am always much more aware of my shortcomings if I share any code :-)
This is great, any chance this is going to lower the amount of money needed for the CiviSMS MIH? I'm lobbying my boss to make a contribution & and momentum helps!
Thanks for sharing! This is really great!