Hi! I am Utkarsh Sharma, a third year undergraduate student studying Electrical Engineering from IIT Bombay. I am working on the project Email Preview Cluster for CiviCRM and this blogpost talks about the same. I’m being mentored by Tim Otten for this project.
The forum thread I started for this project can be found on
http://forum.civicrm.org/
And here you can find the Wiki page Tim created for Email Preview Cluster
http://wiki.civicrm.org/
Aim of the Project:
All webmail services such as Gmail, Yahoo Mail, Live, etc. have their own way of rendering the HTML Emails. Our aim is to provide a service that will give the user a peek into how those emails that will be sent as campaigns, newsletters, etc. will appear to the receivers using these various webmail services. There are many paid services for this purpose, but we aim to provide this feature under a free, open license.
Overview and Dataflow:
For the Email Preview system, we are using a three agent system: The Composer, The Preview Manager and The Renderer.
This three agent system gives us the freedom to extend the Preview System to more webmail and even non-webmail services in the future. We can have different kinds of renderers for different types of email services working with the same manager. Further, just one or a handful of Managers can cater to a number of Composers and Renders.
Composer:
Any work has not been started on the Composer and we’ll come to it after we’re done with the Preview Manager and the Renderer.
The Preview Manager:
When the Preview Manager receives a preview request from any of the Composers, it hands the task over to an idle Renderer (the third agent) and returns the screenshots generated by the Renderer to the Composer who requested for it.
The Preview Manager is the next thing we’re going to work upon and more can be read about it on the Wiki page.
The Renderer:
The Renderer, which is a Javascript code, uses a web-browser automation tool called WebdriverIO to log into email accounts and take screenshots, and another tool called Nodemailer to send emails. Both WebdriverIO and Nodemailer are available as NodeJS packages.
A renderer periodically polls its manager to check if there’s a pending task on the Manager’s side. When the task is eventually handed over to the renderer, it picks up the Email content from the JSON object and sends it to predefined Email Ids on various webmail services. It then logs into all those webmail accounts for which a preview for the email has been requested, opens the particular email, takes the screenshots and returns them to the Manager.
I’ll go into more details of the Renderer as this is the part I’ve worked on.
Sending Email
The Renderer polls the manager periodically to check if the Manager has got a task for it. If there is a pending task, the Renderer picks up the email from a remote JSON file and sends it to various webmail accounts using Nodemailer.
Saving Screenshots
Once the Email has been sent, WebdriverIO is put into action. WebdriverIO picks the email addresses from a local JSON Config file and logs into them one by one, opens the email and saves the screenshot to send it back to the Manager as response.
When WebdriverIO logs into an Email account, it uses the from-address of the email to uniquely identify one email from the others. We plan to assign a unique from-address to every task, thus making it easier for WebdriverIO to open the correct email.
WebdriverIO uses the Selenium standalone server to communicate with the browser. A PhantomJS server, which uses the headless PhantomJS browser, can also be used instead of Selenium. The initial testing is being done on Selenium with Firefox, but we plan to move to PhantomJS as it has some advantages over Selenium (1. Easier installation- PhantomJS is available as a NodeJS package 2. Selenium is unstable in the long run).
A lot of the work on the renderer has been completed:
1. Polling the manager (a remote JSON file, for now)
2. Sending the mail using Nodemailer
3. Gmail and Yahoo are working on Selenium+Firefox
That is, we have successfully managed to locally save screenshots of the received email for Gmail and YahooMail.
4. Gmail is working on PhantomJS. Yahoo is posing some problems though.
The prototype of the Renderer is available at https://github.com/
Here are the screenshots taken from Gmail and YahooMail using Selenium+Firefox.
Here is the screenshot taken from Gmail using PhantomJS:
Key Deliverables
1. 24 June
Midterm Submission: Model for Renderer
Blog post about the Midterm Submission
2. 28 July
Beta Version
Blog post about the Beta version
3. 18 Aug
Code will be made available on Github
Final blogpost about road maps to incorporating the project with CiviCRM
Detailed Documentation of the project
Timeline for the remaining duration:
24 June:
- First Blog Post describing the details of the Midterm Submission and status of the project
- Midterm Submission: Renderer
24 June - 06 July: Preview Manager- Week 1 and 2
- Start working on Preview Manager
- Introduce improvements in the Renderer model, based on feedback
07 July - 20 July: Composer- Week 1 and 2
- Start working on the Composer- Fetching and displaying the received screenshots on the CiviMail UI
- Continue work on Preview Manager
21 July - 27 July: Other webmail clients
- Extend to other webmail clients
- Perform tests to ensure working of Beta Version before presenting
28th July: Beta Version
- Present Beta version based on feedback, incorporating other webmail clients
- Second blog post describing details of the Beta Version and status of the project
28 July - 03 Aug: Testing and Bug fixing Week
- Start working towards making a detailed documentation
04 Aug - 10 Aug: Wrap up
- Buffer Week
- Final Documentation
11 Aug - 17 Aug: Wrap up
- Wrap up and Buffer Week
- Merging code with CiviCRM-master
- Final Blog Post describing takeaways, accomplishments and findings on what needs to be done before the project can go live on CiviCRM
18 Aug: Final submission
Required Feedback and Suggestions:
- Feedback on the Renderer model.
- Suggestions on how we can display the retrieved screenshot(s) on the CiviMail UI
Comments
Thanks, Utkarsh. This is a great summary.