Hi I am Aditya Nambiar, and I am currently working on Visualization for Mailing Stats and A/B Testing which is my GSOC 2014 project. I am at the midterm stage of my project and would like to share the progress I have currently made. Please do give your valuable feedback and suggestions. Here is the link to my previous blog https://civicrm.org/blogs/adityanambiar/ui-mockup-ab-testing-civimail
Currently the entire A/B testing has a tabular UI which we have created using AngularJS. The user has to follow the sequence of tabs and cannot jump ahead since the other tabs are disabled and are enabled only when the User has successfully entered all the form data required in previous tabs. However once the tabs are enabled the user can jump from any enabled tab to another to make the changes he wants to.
The first tab is “Campaign” which provides options for AB testing
Depending on the selection made a corresponding tab – Compose opens on clicking “Next “ button.
1) If “From Names” is selected
2) If “Subject Lines” is selected
3) If “Two different Emails” are selected
On clicking “Next” it opens the Recipients and Winner Criteria tab.
This tab contains all the important information required to run the A/B testing such as
- Groups Selected.
- Percentage of subscribers to whom the test mails( mail A and mail B) should be sent to.
- How the winner should be decided.
- For how long should the A/B test run for.
Based on the feedback to my earlier post, we have removed the automation option.
On clicking “Next”, “Preview and Send” tab opens which display all the information in a table so that user can verify them and make changes if necessary.
The user can also click on the “Preview Mailing” Link to verify the mailing and send test mails.
The next option is the when to start option. The user can either begin the test now or choose to start it at a later date.
Clicking on the Submit & Send button, it open the results page where the two mailing will be compared and their performance can be seen as graph .
Also the various statistics about how they are performing will be available as a table. The User has the option to end the A/B testing whenever he feels like and send mailing A or B to the remaining subscribers.
Things to do:
- Store A/B testing information in database and add new API’s for it
- Work on result page
- Fix know issues and UI cleanup
Comments
Are there any plans to use unsubscribe rate/optout rate as the criteria for selecting winners? This would allow organizations that don't track their emails to still A/B test.
The when to start now /later doesn't seem quite right. Presumably if they check later then js will deliver some more fields. In which case it might make sense to default to now & have a message like 'your mail is scheduled to go straight away - click here (or some button) to schedule to sent later.
If you don't want to do that you should still default to now I think.
This is a FANTASTIC project and fills a HUGE hole in Civi's marketing capabilities.
Is this going into the core product or will the code be an extension?
When will the code be made available to the community or will it be distributed via a partner?
Thanks in advnace!
James Cope