Published
Tuesday, November 3, 2009 - 20:57
Written by

Introduction: The Need

Progressive Technology Project (PTP) supports grassroots community organizing groups nationwide that engage in social, economic and environmental justice efforts to empower low-income people and communities of color. These groups conduct phone banking and field canvassing as an integral part of their base-building and voter engagement strategies to engage existing and new constituents. PTP hired DharmaTech to help them build PowerBase: a customized CiviCRM for use by community organizing groups.

PowerBase helps groups manage the following data related to phonebanking and field canvassing work:

  • voter info, history, and demographics
  • reason why the phonebank or canvass is taking place
  • responses to questions asked during the phonebank or canvass
  • phonebank or canvass status
  • primary language spoken


CiviCRM falls short of generating phonebank and canvass lists, because these features were missing:

  • phonebank and canvass list report templates based on voter info, reason for canvass, responses, etc.
  • a report filter to search on specific criteria such as language spoken and smart groups
  • ability to display non-European alphabet (e.g. Chinese characters) in the PDF format of the report
  • address parsing (a critical feature for creating canvass lists, a.k.a. "walk lists")
  • ability to sort lists without affecting householded contacts

DharmaTech's Challenge

We used CiviReports to implement phonebank (call list) and canvass lists (walk list) for PTP's groups.  The goal was to generate a PDF file of information about selected contacts that could be used by people phonebanking and canvassing.

To do this we had to overcome a number of challenges. First, we drafted workflows for phonebanking and canvassing and reviewed them with PTP.  Second, we tackled the reporting mechanism.  The screen that the user uses to generate a call or walk list exposes data fields and provides filters to select data values from tables.  These fields and filters are defined by an array whose structure is not documented.  Therefore, we had to read the code to understand how to produce the needed screens.

Some of the logic for field definitions depends on the table Data Access Object (DAO).  The DAO for a table is generated when a CiviCRM distribution is created, and so no DAO exists for a custom table.  After lengthy study of the code, it became apparent that the need could be met by using the DAO of that table which was extended by the custom table.

The available filter operators are very limited, particularly for select and multiselect operations.  In particular, the operator for a single select is configurable, which is only occasionally useful, and there is no way to replace the operator with the static word "is".  The operator for the multiselect, on the other hand, is not configurable.  We needed to allow the user to choose among interpretations of the multiple selections.  For this reason, we provided alternative implementations of the select and multiselect operators.  The CRM_Report_Form class defines filter operators with class constants and a case statement, which provides no method of adding or overriding operators in a subclass.  Therefore it was necessary to modify the code of the distributed CRM_Report_Form class to define the needed filters.

The walk list required pages separated by street name and odd/even house numbers, ordered by house number.  The columns in the civicrm_address table for street_number, street_number_suffix, street_number_predirectional, street_name, street_type, street_number_postdirectional and street_unit are not populated in the database.  It was necessary to parse this information from the street_address and supplemental_address_* columns.  Then a temporary DB table was built to permit selection of the needed information.

A requirement was that the PDF show names in the contact's native language.  The CiviCRM database is internally UTF-8 Unicode, so storing and retrieving characters in any language is not a problem.  The challenge was to output characters outside the western European alphabet to the PDF file.  The basic problem is that PDF is not designed to support an alphabet with more than 255 characters, so direct support for Chinese, Japanese, Korean and Vietnamese character sets is not possible by the usual PDF mechanisms.  dompdf, the PDF generation software distributed with CiviCRM, in particular doesn't support Unicode.  However, there are various work-around methods used by certain software packages.  Firefox can display Unicode characters and output them to a PDF file, with certain restrictions.  This is a work-around until a Unicode PDF facility is implemented.


To see a demonstration of what was discussed:

http://ptp.dharmatech.org

username: demo
password: demo

You can view the work-around described by logging in - click on canvass list - select filter "group",  "is equal to", "Mandarin" - print preview. You'll see Firefox display Unicode characters and can save that html to pdf format.


Want to contribute to this project?

  1. A related feature needed in conjunction with producing phonebank and canvass lists, is the ability to input responses gathered from the phonebank or canvass, through a "batch" data entry screen that looks similar to the reports. A user can:
    • pull up the list of contacts for the phonebank or canvass, view a batch data entry screen that resembles the report, and begin entering the responses, or
    • have a search view that resembles 1 record of the report, where the user can either enter the contact id number or the barcoded representation of the contact id (barcodes are a type of font read by a barcode reader device) to find the contact, and then record the responses manually or use barcoded representations of the responses.
  2. Implement a PDF output that supports Unicode characters. Note: After researching, one approach to achieve this is using a free Java library called iText and a package called xhtmlrenderer which uses iText as an exact drop-in replacement for dompdf.

Comments

I'm really excited that CiviCrm is now being customized for organizing purposes but ran into the question whether the development work done for PowerBase is being fed back into the main branch of CiviCRM or if PowerBase represents a permanent fork. If so, how is sustainability being guaranteed?

I'm working with some local folks in San Antonio who could really use this newly evolving functionality but am afraid to talk to them about solutions that may not be sustained and continuously developed, especially if PowerBase becomes a permanent fork.

Other than that, I'm really psyched about this!

Sebastian

We hope this isn't a fork which is why we are blogging about it.

Throughout our development with PTP, we offered advice along the way to help prevent this from being a fork. We want the community to reuse the effort - it's a value of ours - but we're not quite sure how, seeing that:

  1. this isn't a simple patch.
  2. the code is based on a custom database configuration for community organizers.

When we began to wrap up the project, we asked the core team how we should contribute our code back to trunk. The response was "blog about it." So we're doing that, hoping this blog fosters a discussion that provides an approach for incorporating all of our work back into core.

The code is in our svn repository. It references custom tables in the PTP configuration so you would need to either create the same custom tables or modify the code to not reference them.

Thanks for your thoughts!

I think we will have quite a few custom reports which work under the assumption that certain custom fields are present. We wanted this work to be blogged about, so other folks can be made aware and we can start thinking about how to potentially support specific distributions with various configurations

Another area that we might be able to benefit from the drupal experience and see if we can leverage some of the existing work and allow folks to package "custom configurations"

We plan on discussing and figuring out how to incorporate some/all of these features in the upcoming civicrm code/test sprint (Nov 16th/17th), http://civicrm.org/civicrm/event/info?reset=1&id=26

lobo

Just a quick thought: It might be nice to be able to turn on or off features like this in a way similar to how components are enabled. In this case it would involve creating custom fields and parsing addresses into the detailed address fields, as well as enabling the reports. Perhaps the distinction in Drupal between disabling a module and uninstalling it would be useful in this case: the former hides the functionality, the latter removes database changes and can involve destroying data. I would be comfortable seeing a first release with support for disable/uninstall.

Thanks for taking the time to comment Joe.

Having functionality broken into modules could also:

  1. Allow a module, say CiviReport, to improve at it's own pace instead of the pace of a CiviCRM release.
  2. Possibly making it easier for the community to contribute code/documentation by being exposed to a subset of issues rather than CiviCRM as a whole (i.e. Bug bingo for CiviMail would be easier to tackle for the casual contributor than bug bingo for CiviCRM (and all components).
  3. Should allow for better testing... easier to segment the tests and interfacing with core and the modules with mock objects.
  4. We could all finally see the dream of the CiviUnicornsAndRainbows module. :)

Maybe the "componentizing extensions" discussion should be a new blog post to see if there's any traction.

Yes, componentizing is worth its own post. Seems like lobo and others are moving along there and have good ideas for occasions to involve the community.

I had meant to write, a first release _without_ install/disable.

Would be good for folks to step up and help make this a reality. Figure out what needs to be done, what changes need to be made to our exisiting process and how we can introduce these changes

I think we should start a "personal battle plan" for each release where community members basically state what they are going to work on for a specific release. Now is a good time to come up with a battle plan for 3.2

From a code perspective the components are fairly well seperated. There is still a significant amount of "config" data that gets shipped with each component that will be a bit thorny to unwind

Has any progress been made here to include parts of Powerbase (specifically the address field parsing) to CiviReports or otherwise?

it is part of the civicrm_engage module. this will be included by default in 3.2.1

the address parsing code is now part of civicrm core. please direct all future questions to the forums

thanx

lobo

I think many CiviCRM users could make use of Phonebank and Walklist reports, and I'm glad to see them on the 3.1 release feature list. I'll give you or CiviCRM a contribution or payment of $500 as a token of support for implementing 1.

I would encourage searching for other PDF libraries that support UTF-8. This issue is separate and more general than its use for these specific reports.

Well, I'm really new to Drupal AND CiviCRM, but is there any updates at all to this?   is there a Phonebank solution out there?