As part of the statistics project for CiviCRM, we are calculating a number of metrics that give us a good visibility on CiviCRM end-users. Some of these statistics have already been exposed and discussed, so this blog post will focus on how better insight into our user base can translate into positive outcomes for the project.
One of the statistics we produce is a cohort analysis. Per Wikipedia, a cohort analysis classifies users into related groups sharing a common characteristics for analysis. For CiviCRM, we grouped our users per the month they started using CiviCRM and looked at how many of these were still using CiviCRM over time, thereby calculating the attrition rate for this group.
Here is what this cohort analysis looked like for CiviCRM in early August 2015:
By looking at this graph we can see that around 65% of new CiviCRM users abandon the software during the first month of usage, then this attrition rate significantly slows down and around 10% of new users keep the software over the long term (9 months or more after initial install). Of course these numbers need to be put in perspective since there are most probably quite a lot of test sites and/or developmennt instances in these numbers, and these are meant to be kept only for a very limited amount of time. Also, the software being free, many users just 'give it a spin' out of curiosity without really being serious about using it.
But nevertheless, the question is: How can we keep more of these users over the long term?
After a heated discussion on the partners mailing list, the consensus was that adoption lagged primarely because the learning curve is too steep: beginner users do not know where to start of what to do. There does exists a wide body of documentation and helpful resources available, but these are not easilly accessible.
Cividesk decided to tackle this challenge and created:
- the Getting Started dashlet, which provides an 'orientation' for new users right from the dashboard on the home page,
- the Helptab extension, which provides contextual help by linking to helpful resources directly related to the current page.
We will revisit this blog post in a few months to see if this cohort analysis is improving over time ...
Comments
Another factor which may be artificially inflating the drop-off rate is if an admin changes the site-key or the site moves to a new url, which would cause the anonymized identifier of the site to change and our stats collection to think that an existing site had quit and a new one had started. Not sure how often that happens.
We used to do installs for tiny organizations where we would create a staging url, get it set up, then convert the url to the production one. So that kind of process would cause a big drop in first short while.
So a site which starts at dev.example.org and moves to www.example.org will appear here as an install on dev, followed by an abandonment and an install on www?
I think the technical debate of "how much the % of dropoff is affected by URL changes" is missing the point of this excellent post. If we want CiviCRM to be the "#1 default crm for nonprofits" what can we do as a community to enourage and facilitate people to continue using CiviCRM? The "getting started widget" is a great thing. Thanks Nicolas! What's next, community?
A small organization moving from a spreasheet and with no history, no instant need for specific reports, etc. likely has no issue. They can grow into CiviCRM slowly, and it will look like a dream compared to what they were using.
But I am Communications chair for a 3000 member 501c(3) organization with a long history, lots of existing data, and not much money. We are moving to CiviCRM because we have specific needs, and other CRM systems either don't support them or are not within our budget. We also cannot afford to pay for consultants. I am a volunteer. I am an experienced php programmer, but I don't have unlimited time.
We are probably in CiviCrM month 3 and we are not yet fully functional. Some of it is our fault (I cleaned massive amounts of dirty data), but a significant part is CiviCRM. I have some specific comments ... these are not intended as complaints ... they are intended to list specific areas that are in dire need of improvement for anyone heading toward CiviCRM. Also for reference, I have a long programming history including a long history of using php. I think that once we get over what looks like a 3 month hurdle, Civi will be pleasant to use.
1. It is very difficult to know what documentation is correct and what is ancient. Getting rid of the ancient would help a newbie quite a bit.
2. The documentation is often lacking. For instance, I need to use the REST API. It took an entire day to find enough documentation to actually attempt it. There are great examples (working) of individual rest calls, but determining exactly how to call it seems to be completely missing. It took ages to figure out that it expects a CUrL call, and I then had to actually make a call to determine that I would really get XML back. It was a huge relief to find that there is actually a class that can be copied to the remote site and used...but I had to learn the existance of that class from a user blog article and then apply other documentation, such as setting up the API user and key. I made a simple test call, no problem. Then I went to make the call I wanted only to find that the conversion of the input parameters did not work if they include a sub-array (which many do including the one I needed). I had to eradicate a bug in the CIVICRM API class before it would work (since fixing that, I'm in heaven). But having to debug stuff in order to get a needed facility working is absolutely daunting to a new user.
3. The data import is horrid. I used it to import our contacts and then resorted to MySQL to get almost everything else in (financials and memberships). The fact that one faces timeouts for imports of any significant size leads to having to do dozens of imports, each of which is a new opportunity for error. People are quitting before they get started. Acknowledging that I was dealing with dirty data, that import took a month and a half of nearly full time work (I am a volunteer with a full time job, and this is not it).
4. We are having difficulty with a number of reports. We are a membership organization with chapters and households. As an example, we need a breakdown of membership types by chapter (there are several ways to do this for the overall organizatin). I can do it for all members (individuals, businesses, households) where households easily have three entries -- using a cross-tab report. But there is no filter included that will let me get this down to one primary membership for a household. Creating a new report template, and I doubt it is a simple template, is daunting. Not just because one is modifying a class that one did not create, but one also has to find the parent class and study it since there is zero documentation in the closest report that I will need to copy and modify. And enabling a new report template, well, hardly obvious in the documentation that I have found. And again, what examples there are, assume that the basics are understood. Many membership reports don't allow the use of smart groups in the filters. Some needed fields (such as our chapter field) are not always available. And grouping and summary variable use is limited. Again, new users are being put in very hard places.
5. My principle user has a pair of exports that she sends to a mailhouse to do a mail merge. Guess what, for households, I need to provide the household members and to provide the numeric link to the household record. Guess what is not available -- Civi insists on giving me the name of the household. Sigh. Again, a new user is stuck trying to get around an apparent limitation that would seem to be highly artificial. I'm currently patching it together by going straight to MySQL and copying the value I need into a variable that won't be tampered with. That now means that I need either to learn to create a Cron job (scheduled job) with almost zero documentation, of how to do it, or create a Cron job that uses MySQL calls. I just found the CiviRules extension -- I'm hoping that it might give me an easier path.
6. I have come to doubt that our principal user will every be able to create some reports and exports and that it will always take a fairly advanced programmer to do anything new.
7. And before this gets called done, I need to get membership forms that can handle data for households implemented and functional with IATS. I've done this at least 3 times now for other credit card processors. This is the first time that will have to do it using two DRUPAL extensions (our website is a highly customized MVC-based CMS but not a DRUPAL site) -- I'm seriously hoping I may be able to do the forms from our website using the API and then call the IATS extension with it needing only to take the credit card info....but again, I don't see anything simple for a new CiviCMS user.
I could keep going. The point is not that these things can't be done, but rather that they take a lot of knowledge and effort to do. And that is extremely hard on new users. It is impossible for organizations where there is no programmer available.
I am persistent enough to get this done, but .... there is no doubt in my mind why people give up during the first month.
When I get my head above water, I may try to help with documentation.
BTW -- we are one of the "download and abandons" -- I put a copy on my XAMPP server so that I could look at and play with it before we made any commitment. We are now using one of the CiviCRM partner firms as our host and coach/support.
Hey ecotypes,
First of all, congratulations on getting this far! I remember the first CiviCRM installation that I did and it was not pretty :) so I can empathise with you. I agree with a lot of the things that you'd said that can be improved and want to thank you for sharing your thoughts. Feedback from first time users is invaluble in helping us improve CiviCRM.
Here are a few thoughts that are provoked by what your account so far...
I always advise people who are starting out on their first CiviCRM project to get as much advice and help as possible from people who have been there before. It is easy to assume that because CivICRM is free to download, it is free (i.e. easy) to set up and use effectively. Once you've done a few installations, that may well be the case, but it doesn't often work like that for first timers. Also (please don't take this the wrong way but) being an experienced PHP programmer is no substitute for experience with CiviCRM projects. Indeed many of our best CiviCRM implementors know very little or nothing about PHP. CiviCRM is a complex beast with lots of quirks and if you have not worked with it before, you should be prepared for some fairly big bumps along the way. In our defence, I think you'll find this is the same for a lot of projects of similar complexity.
It is great that you are now working with a partner for hosting and support. I'm not sure how much you have been chatting and getting support from the community but there are multiple ways you can reach out to people. For example, you can ask questions here: http://civicrm.stackexchange.com, and file bug reports here: http://issues.civicrm.org/.http://issues.civicrm.org.
Your post also made me reflect on our vision, that "all organizations – regardless of their size, budget, or focus – have access to an amazing CRM to engage their contacts and achieve their missions... ". The 'regardless of their budget' part seems pertinent here and for me, it is one of the most challenging parts of the project. It is amazing that organiastions with little or no bduget have access to volunteers like yourself who have the time to help them out. And we want to make it as easy as possible for you to do so. You have pointed to a few areas where we can improve. I agree that cleaning up our documentation and removing the old stuff is important. And I know you are not alone in thinking that our import and report tools could do with some love.
One of the things I appreciate most about CiviCRM is the way in which our less well funded users can piggy back on the work of their better funded cousins and this is the crux of our particular flavour of open source development. Countless people have made significant contributions that the rest of our community is now benefiting from. As your account illustrates, we still have a lot more to do before we can say that those with no budget have access to an amazing CiviCRM but we are definitely on the way, and so long as people continue to contribute back, we will get there.
Thanks for sharing your experiences and the specific pain points that you have. I hope that you continue working with CiviCRM and encourage you to contine reaching out to others in our community that can help with the specific issues that you mention above. And I also hope that in time you might get to a point where you can contribute towards fixing stuff for those people that come after you.
@ecotypes put all of this so susinctly. It has been a painful journey just getting close to importing data into the real deal. Starting from scratch has been pretty good, but trying to pull in data hurts for so many different reasons. Things like the CLI import routines are seriously undocumented, leaving the well designed but poorly operating web import routines a frustrating better choice. The additional usage of external tools like Kettle Data Integration is barely a mention in a forum post somewhere. The Getting Started applet sounds like it will be a wonderful thing for our users once they get to use it. However, we have to get there. I agree with @ecotypes that more time must be put into getting to where the Getting Started applet will be applicable. Almost nobody is going to be putting their non-profit together with CiviCRM being their original charter member. The need for this type of management happens down the road. And unfortunately, we can't debug and build while we are also trying to learn and import. Time just doesn't work that way.
@ecotypes, @gharris. While I agree with the spirit of your comments and the need for an easier start with CiviCRM (hence all the efforts I put into the Getting Started and other similar initiatives), one must not forget that a CRM & database is several levels of magnitude more complex than an Excel spreadsheet: one must not expect to click on an install buttton and be able to use it straight away. And that is going to be true whatever we do to simplify usage: a car has just a sterring wheel, 2 pedals and a stick, yet one must still attend driving courses, practice for 50 hours and pass a test before being allowed to drive one!
So, point well taken, maybe we should better manage these expectations and put a big warning/disclaimer on the download page advising people to get help from an experienced CiviCRM person rather than try implementing on their own. I will submit this idea upstream and see if it catches-up.
In the meantime, please use the resources that are already available to you if you have not done so already: the user and administrator guide is a gem (http://book.civicrm.org), we also have a wide catalog of very reasonably-priced instructor-led courses (https://civicrm.org/events), and all resources Michael already pointed to in his reply above.
For your data import needs, I suggest you contact one of the following experts: Peter @ Skvare, Jon @ Palante Tech, Youg-Jin @ Emphanos or myself (Nicolas @ Cividesk). We each have developed our own toolkits to do data imports because these are often too complex to handle with any packaged process/software - the import needs to be automated and replayable, and we almost always need to program in new data transforms/matching/merging. So this has to be scripted in a programmatic language - whether Kettle, PHP, SQL or other.
I've implemented a few Civi sites as a volunteer, and applaud the idea of the getting started dashlet. Some very visible & obvious pointers to
documentation at book.civicrm.org and the civicrm stackexchange website would help in understanding Civi and knowing where to go for questions.
Also mentions for civiteacher and extensions would help. Some simple-ish explanations of cron and scheduled jobs (arcane for a beginner) would help.
Bigger tasks would be better organisation of extensions - where to go for the latest version? and of course overhauling imports.