Hi! I am Utkarsh Sharma, a third year undergraduate student studying Electrical Engineering from IIT Bombay. This the final blog post regarding my GSoC project Email Preview Cluster. Tim Otten is my mentor for this project.
My midterm blog post about the project can be seen at
I have added a new block to the Mailing Page for requesting an Email Preview Cluster. This block sits at the bottom of the page just below the Preview Block. When a new mailing page is opened, then initially the Request Preview Option is disabled. This option is only enabled after all the required fields are filled and at least one Email Client has been selected from the checkboxes.
When the "Request Preview" button is clicked, a Preview Batch is POSTed by the composer GUI to the Preview Manager containing the selected renderers (email clients), the body_html, the body_text and the subject in the the form of a JSON object. The composer then keeps checking the status of the batch and shows a faded icon for the respective email clients until the screenshot (or an error) has been returned.
Clicking on the faded icon would generate a CRM alert requesting the user to wait, while clicking on a clear icon would open the returned screenshot in a new window (or display the error received in a CRM alert).
When an old mailing page is opened, the composer checks if there's an on-going preview batch which might have been requested earlier from the same mailing page. In case there is a on-going preview batch, the composer checks it's status and manages the display inside the block accordingly. Only when the user clicks the "Cancel" button is the initial checkbox screen shown. Clicking the cancel button deletes any preview batch that was POSTed on the Preview Manager from the current mailing page.
The Preview Manager is a batch manager for email previews. It is based on Loopback which is an open-source Node.js framework for creating end-to-end REST APIs. The preview manager provides a RESTful CRUD API which may be used for two types of agents:
Composer - The mail user-agent (MUA) in which a user prepares a mailing. Generally, the Composer submits a PreviewBatch and polls to track its progress.
Renderer - The backend service which prepares a screenshot of how an email would appear when read in different MUAs. Generally, a Renderer polls for a pending PreviewTask record, prepares a screenshot, and then marks it as completed.
When a composer posts a "Preview Batch" at the Manager, the Manager breaks it down into "Preview Tasks", each preview task having a specific renderer type (e.g. gmail, yahoo). When a composer tries to read the status of it's preview batch, the Preivew Manager checks if a renderer has picked/returned a screenshot for all of the preview tasks related to the particular preview batch and relays back the status/screenshot/error message to the composer. When a composer cancels a preview batch, the preview manager deletes the preview batch and all the preview tasks related to it.
When a renderer tries to claim a preview task of it's type (e.g. A gmail renderer would look for a preview task of gmail renderer type), the Preview Manager hands over any pending preview tasks to the renderer and marks that task as "taken". When the same renderer reports back with a screenshot/error, the Preview Manager updates the preview task's result and marks the task as "complete". This result then gets relayed over to the composer.
The renderer is Node.js based code which polls the Preview Manager periodically to claim a pending preview task of it's type, prepare a screenshot for that task and POST it back to the preview manager. Currently, the webmail-renderer package is installed as a dependency of the Preview Manager package, but when scaling out, renderers are designed be placed on remote locations and still be able to claim and post the screenshots back to the Preview Manager.
A renderer has a particular type, which is the MUA/email client it prepares the screenshot for. Right now, we have two working renderers, one for Gmail and one for YahooMail. Each renderer looks for a pending (not "taken") task for it's own type at the Preview Manager and works on it.
The renderer uses picks up the Email content from the Preview Task, sends an email using NodeMailer, a Node.js tool, to a user configured email-ID (from another user configured email ID) and then uses WebdriverIO, another Node.js tool, to automate the browser to login to the receiver email account, open the email, produce the screenshot. It then POSTs this screenshot back to the preview task. In case the renderer encounters an error, it reports the error back to the preview task instead of the screenshot.
A more detailed description of the renderer can be found on the Midterm Blogpost.
Creating, Reading, Updating and Deleting a preview batch would require some permissions. For that an authentication system has been set in place at the Preview Manager. When a new mailing page is opened, this block is only shown if the composer has successfully fetched an access token from the Preview Manager. An access token can be fetched from the preview manager only by an already existing user, which, at the moment, can only be created manually. In case the access token couldn't be fetched, this Email Preview Cluster block is completely hidden from the GUI.
To create a user (and subsequently fetch an access token) the user needs to configure the Preview Manager settings (namely the username, password and the preview-manager-url) on CiviCRM and then manually create a user on the preview manager REST API (on <preview-manager-url>/explorer). Once the user is created on the Preview Manger, the composer automatically logs in, fetches the access token and enables the email preview cluster service.
Instructions for setting up email preview cluster on your CiviCRM can be found on the Preview Manager Github Repository.
The Email Preview Cluster service, despite having reached this point, can be improved further in the following ways:
I'd love to be a part of the further work done on this project.
I can't thank my mentor Tim Otten enough for all the guidance and help (and a framework) he's provided me with during these three months. I'd also like to thank Kurund Jalmi, and my friends Pranay Yadav, Siddhant Rajagopalan and Vishal Agarwal for their help.
All in all, it was amazing working on this project. It was a great learning experience. I'd like to thank CiviCRM for providing me with this wonderful opportunity.
The code for this project is divided over three repositories and can be found on
https://github.com/utkarshsharma/civicrm-core - The Email Preview Cluster GUI
https://github.com/civicrm/prevem - The Preview Manager
https://github.com/utkarshsharma/webmail-renderer - The Webmail Renderer
The forum thread I started for this project can be found on
And here you can find the Wiki page Tim created for Email Preview Cluster