CiviEngage is a Drupal module that (in the next release of CiviCRM) automatically configures CiviCampaign to make it easier to get up and running by setting many of the configuration tasks with sensible defaults.
Over the last few weeks I've been getting intimate with both CiviCampaign and CiviEngage in an effort to either merge CiviEnage's features in CiviCampaign or think about how to package them as an extension. In the process, I made some discoveries concerning surveys and petitions that made me pause and want to reach out for direction before proceeding. In this blog post I'm only mentioned surveys, but the same holds for petitions which use essentially the same data model.
When creating a survey in CiviCampaign (without CiviEngage), your first task is to either create a new custom data group and add custom data fields for each question you want to ask in your survey. Or, you pick an existing custom data group and add custom data fields for each question. These custom data fields are extensions of the Activity entity. This approach very elegantly allows each response to the survey be recorded as an activity, with each custom field showing the question and answer for the particular contact.
Then, you create a profile that includes the particular questions you want for your survey. This use of profile is also very convenient - for petitions it means you can easily expose the petition on a web site and get responses directly into your database.
Lastly, you create a survey. One of your options is to choose, from a drop down list, the profile containing the questions you want to use and you are all set.
At the moment, I believe there are two places for improvement.
The current data model works fine for an organization that wants to conduct one or two surveys or petitions, or perhaps an organization that wants to have 50 to 100 surveys or petitions. But what if a group wants to have a new 20 question survey every week for 5 years?
With the current model, they can either create a new custom data group for each survey that holds each new question, or they can have a single custom data group that stores all questions. The resulting database structure is either a new database table every week (after five years: 260 new tables in the database) or a single table with 5200 fields (and an immense drop down list when choosing your field when creating your profile). Either way, we're adding data horizontally via new fields rather than vertically via new rows. Both approaches will complicate any activities query that has to string together all the activities custom data tables. This problem is not critical now and, if nobody creates a lot of surveys or petitions, will never be critical. However, it seems like a design flaw that has the potential to bite us down the road.
CiviEngage solves this problem by by adding a custom data group called Survey Questions that extends the Survey component and then adding four fields called Q1, Q2, Q3, and Q4. CiviEngage adds another Custom data group that extends Activities (called Survey Responses) and adds four fields called Q1 Response, Q2 Response, Q3 Response, and Q4 Response. A profile is also automatically created called Survey Response that contains the Survey Response fields.
When you create a new survey, you simply select the Survey Response profile. Since the Survey questions data group extends the Survey component, you get four new fields at the bottom of the Survey form where you can add your questions for the given survey.
Now, no need for a new custom data group or custom fields every time you add a new survey - and we have an improved user interface.
Unfortunately, this data model and user interface has different, but no less problematic limitations. The user interface forces every survey to be a four question survey. And, each custom data field for answers uses a preset option list (yes, not, maybe, undecided) which effectively limits you to yes/no questions. This is not the solution to the user interface problem.
As for the data model - it cleanly solves the scalability problem since we don't have to add new tables and new fields every time we have new questions. But it introduces a database normalization problem. There's no normalized relationship between the fields holding the question and the fields holding the answers.
We can guess that the third field in the question table corresponds to the third field in the answer table, but there's no way to write a reliable SQL statement that embeds that logic.
Unfortunately, this is no solution to the data model problem either :(. And worse, the CiviEngage data model breaks the logic of the CiviCampaign user interface for entering survey responses and the CiviCampaign reports. Instead of displaying the survey questions, it displays the un-helpful label: Q1, Q2, etc.
At the Progressive Technology Project retreat we devoted a good chunk of time to addressing this problem and came up with a proposed strategy.
Given the analysis so far, and assuming I'm not missing anything (a big assumption :), we propose:
The Progressive Technology Project is committed to making these changes happen - but want to ensure we're moving in the right direction before we start. Please help with feedback!