Hi! I am Utkarsh Sharma, a third year undergraduate student studying Electrical Engineering from IIT Bombay. This the final blog post regarding my GSoC project Email Preview Cluster. Tim Otten is my mentor for this project.
My midterm blog post about the project can be seen at
The Composer GUI
I have added a new block to the Mailing Page for requesting an Email Preview Cluster. This block sits at the bottom of the page just below the Preview Block. When a new mailing page is opened, then initially the Request Preview Option is disabled. This option is only enabled after all the required fields are filled and at least one Email Client has been selected from the checkboxes.
When the "Request Preview" button is clicked, a Preview Batch is POSTed by the composer GUI to the Preview Manager containing the selected renderers (email clients), the body_html, the body_text and the subject in the the form of a JSON object. The composer then keeps checking the status of the batch and shows a faded icon for the respective email clients until the screenshot (or an error) has been returned.
Clicking on the faded icon would generate a CRM alert requesting the user to wait, while clicking on a clear icon would open the returned screenshot in a new window (or display the error received in a CRM alert).
When an old mailing page is opened, the composer checks if there's an on-going preview batch which might have been requested earlier from the same mailing page. In case there is a on-going preview batch, the composer checks it's status and manages the display inside the block accordingly. Only when the user clicks the "Cancel" button is the initial checkbox screen shown. Clicking the cancel button deletes any preview batch that was POSTed on the Preview Manager from the current mailing page.
The Preview Manager
The Preview Manager is a batch manager for email previews. It is based on Loopback which is an open-source Node.js framework for creating end-to-end REST APIs. The preview manager provides a RESTful CRUD API which may be used for two types of agents:
Composer - The mail user-agent (MUA) in which a user prepares a mailing. Generally, the Composer submits a PreviewBatch and polls to track its progress.
Renderer - The backend service which prepares a screenshot of how an email would appear when read in different MUAs. Generally, a Renderer polls for a pending PreviewTask record, prepares a screenshot, and then marks it as completed.
When a composer posts a "Preview Batch" at the Manager, the Manager breaks it down into "Preview Tasks", each preview task having a specific renderer type (e.g. gmail, yahoo). When a composer tries to read the status of it's preview batch, the Preivew Manager checks if a renderer has picked/returned a screenshot for all of the preview tasks related to the particular preview batch and relays back the status/screenshot/error message to the composer. When a composer cancels a preview batch, the preview manager deletes the preview batch and all the preview tasks related to it.
When a renderer tries to claim a preview task of it's type (e.g. A gmail renderer would look for a preview task of gmail renderer type), the Preview Manager hands over any pending preview tasks to the renderer and marks that task as "taken". When the same renderer reports back with a screenshot/error, the Preview Manager updates the preview task's result and marks the task as "complete". This result then gets relayed over to the composer.
The Renderer
The renderer is Node.js based code which polls the Preview Manager periodically to claim a pending preview task of it's type, prepare a screenshot for that task and POST it back to the preview manager. Currently, the webmail-renderer package is installed as a dependency of the Preview Manager package, but when scaling out, renderers are designed be placed on remote locations and still be able to claim and post the screenshots back to the Preview Manager.
A renderer has a particular type, which is the MUA/email client it prepares the screenshot for. Right now, we have two working renderers, one for Gmail and one for YahooMail. Each renderer looks for a pending (not "taken") task for it's own type at the Preview Manager and works on it.
The renderer uses picks up the Email content from the Preview Task, sends an email using NodeMailer, a Node.js tool, to a user configured email-ID (from another user configured email ID) and then uses WebdriverIO, another Node.js tool, to automate the browser to login to the receiver email account, open the email, produce the screenshot. It then POSTs this screenshot back to the preview task. In case the renderer encounters an error, it reports the error back to the preview task instead of the screenshot.
A more detailed description of the renderer can be found on the Midterm Blogpost.
Authentication
Creating, Reading, Updating and Deleting a preview batch would require some permissions. For that an authentication system has been set in place at the Preview Manager. When a new mailing page is opened, this block is only shown if the composer has successfully fetched an access token from the Preview Manager. An access token can be fetched from the preview manager only by an already existing user, which, at the moment, can only be created manually. In case the access token couldn't be fetched, this Email Preview Cluster block is completely hidden from the GUI.
To create a user (and subsequently fetch an access token) the user needs to configure the Preview Manager settings (namely the username, password and the preview-manager-url) on CiviCRM and then manually create a user on the preview manager REST API (on <preview-manager-url>/
Setting Up Email Preview Cluster
Instructions for setting up email preview cluster on your CiviCRM can be found on the Preview Manager Github Repository.
Conclusion
The Email Preview Cluster service, despite having reached this point, can be improved further in the following ways:
- Scaling out to other email and even non-webmail clients like Thunderbird, MacMail, Live, Hotmail, etc.
- Increasing security on the Preview Manager: Providing a sysAdmin, restricting user creation permisions, adding authorization along with authentication for making CRUD calls to Preview Batches and Preview Tasks.
- Increasing security on the renderer: Saving sender and receiver email credentials in a more secure database than a json file.
- Making the setup/installation easier. Making it's integration with CiviCRM easier for the user.
I'd love to be a part of the further work done on this project.
I can't thank my mentor Tim Otten enough for all the guidance and help (and a framework) he's provided me with during these three months. I'd also like to thank Kurund Jalmi, and my friends Pranay Yadav, Siddhant Rajagopalan and Vishal Agarwal for their help.
All in all, it was amazing working on this project. It was a great learning experience. I'd like to thank CiviCRM for providing me with this wonderful opportunity.
Important Links
The code for this project is divided over three repositories and can be found on
https://github.com/
https://github.com/civicrm/
https://github.com/
The forum thread I started for this project can be found on
http://forum.civicrm.org/
And here you can find the Wiki page Tim created for Email Preview Cluster
http://wiki.civicrm.org/
Comments
Thanks, Utkarsh. I know that you came into this project without a whole lot of software development experience, and it's been great to see you progress through each of the tiers -- working with different languages and programming models, and getting each to talk to the others.
Someone sent me an email asking, "When can we get this in the hands of users?" (paraphrased) Perhaps the best milestone is deploying to http://dmaster.demo.civicrm.org/ . There are two issues on the critical path:
After that, the main thing will be getting some testing/feedback from users with a fresh perspective, esp. regarding: