CiviCon London is only 1 week away which means it's time to draft a "State of the Project" presentation for the opening session. It's a good time to look up from our computer screens so we can think about and discuss goals for the next few years.
Our number one goal is to build a strong vibrant community which can sustain the project for a long time to come. We think this means folks from across the community participating in ALL aspects of the project, and the project becoming self-sustaining from a financial point of view. We are super-excited to see folks in the community taking on important tasks in lots of areas …
- Organizing CiviCon and UK Code / Book sprints
- API v3 team
- Accounting system integration
- Case studies / website redesign
- Floss manuals book
- CRM Idol marketing and coordination
- Number of consulting providers and Civi-ASP's on the rise
- Number of people submitting improvements and bug reports WITH patches increasing significantly
Looking to the future, we envision a completely self-sustaining project with:
- Make It Happen campaigns driving a large set of functionality
- Large organizations scratching their own itch individually or collaboratively
- Teams of developers and users taking ownership of Quality Assurance, Documentation, Support, Usability and other key areas
- A good stream of reproducible bug reports, tests and patches
- GREAT person-to-person support on the forum AND IRC (be the Zappos.com of open source projects!)
- Library of excellent online tutorials and training guides
- Improved user and developer documentation
- … and more
We'd love to hear your thoughts. Where would you like to see the project go in the next 6 / 12 / 18 months? What's your vision and how can we get there together? Although many of you will be tempted to post feature requests - we're hoping to hear more big picture thoughts for this dialog :-)
Comments
Feature-wise, the project could gain greater adoption with a more robust membership model that includes solid reporting; the current data model and reporting makes it very difficult to tease out useful information, like month-over-month or year-over-year statistics on new/renewed/upgraded/downgraded memberships. I currently write SQL views and stored procedures to get information out for clients (and my own org) in a meaningful way. The concept of copying an entire membership to multiple contacts (instead of a linking table to tie contacts to a single, top-level membership record) adds to the difficulty of this.
Confluence is tricky because there are multiple versions of pages and it's hard to find information. For example, USPS integration took the better part of a day to set up for our org. There are few video tutorials. Ideally there would be a tool to crowdsource needs of what docs and tutorials are missing, use voting to decide priority, perform peer review, and cross them off the list as they're considered complete. The concept would be similar to StackExchange, OSQA, or Question 2 Answer. This concept could be used for features as well (but in a separate silo), so the most desired features float to the top and then become Make It Happen campaigns. Forums just aren't an ideal tool for this kind of info.
Finally, I think there are many in the community like me who know a lot about nonprofits and effective use of CRM systems to advance the mission, but are only amateur PHP coders. We can get by from reading and tweaking examples, but there is little sample code for integrations, custom theming, etc. A sample library with concrete examples would be helpful here.
I would simply like the ability to create custom reports, like a page that shows the website link for donor organizations by date range of contribution. I would also like to have the list of contact types either show up in alphabetical order, or allow you to order it manually. Small things like this would help me tremendously without having to try and hack the code to create handwritten reports.
Hi,
My major concerns are how to make and fund hard to do, useful and not "end user visible" changes, ie. paying back the technical debt.
1) Change some of the underlying big components, eg quickform, the db layer or ORM are obsolete and haven't been improved for a while.
2) refactoring so lots of actions that are done on the form level are moved to the BAO
and as we have seen quite regularly, funding by features mean a lot of things that become overtime redondant (ie. serve the same need) but as they have slighlty different histories and features, you keep both and only lucky and experienced users are able to choose the right one. Plus a lot of features that ought to be shared and that are duplicates.
Don't want to go into arguing if tags and groups are really identical, or why you can't schedule a mass mailing from a list of participants, or that you can only add to groups from a report...or if send mail and send mass mailing are really different, or ...
Just that consolidating some of the features (eg mass actions on a list of contact) ought to be sharing the same code, and that some features that become more and more similar ought to be merged.
These are complicated and long tasks, with very few end user visibility. They should be done, and paid for, and I don't know by whom nor how to convince that it's needed to people able to invest the money.
X+
I would put moving off of SVN to Git on the list
Another one on the infrastrcture: the wiki. The namespacing by version and how that's badly done means that all the links go to a page on an obsolete version without any link to the page in the latest version, and therefore that google puts the obsolete content first.
X+
+1 on @xavier's wiki point
As another downstream developer, I also think that the major challenge is the accumulation of technical debts, inconsistencies, and tight-couplings/closed-interfaces, and I share your concern about how to pitch/fund the necessary refactoring. We need to start by describing why this technical debt matters -- e.g. how does it prevent us from achieving the vision of a self-sustaining project?
To my mind, technical debt and certain architectural issues make it harder to attract contributors who are ready to experiment with new features and ideas. From the perspective of a new, would-be contributor, technical debts manifest as "unexpected obstacles and inconsistencies" which require "lots of research -- code reading, forum discussions, googling." (This is not to say that all obstacles represent debts or that research is bad -- just that debts drive up the perceived obstacles.) If a would-be contributor sees a pattern of obstacles/debts, he'll grow frustrated and walk away. (Or, more likely, he'll accept some cowboy solution to his immediate problem -- and then avoid CiviCRM in future projects. The result is the same -- the community loses a potential contributor.) I suspect that the only contributors who make it through the learning-curve are ones with unusually high dedication, stubbornness, talent, or experience.
In other words, it feels like architectural improvements (consolidation, simplification, openness) will drive adoption by developers -- thereby expanding the talent-pool, outside contributions, and risk-taking -- which, in turn, creates a more self-sustaining community. Of course, this is only a notion, and it's only part of fostering a community. However, to support this notion, let me end with some interesting observations about other successful open-source projects:
1. For D7, Drupal core poached Drupal contrib, moving some of the most successful modules into core. This was possible because, with D5/D6, core provided on a relatively clean, open architecture with which outside developers experimented and innovated.
2. With Linux 1.x, there was some famous debate about achieving modularity. With Linux 2.x, that community adopted a compromise based on aggressive use of loadable modules -- a compromise which has endured during 15 years of heavy growth and contribution.
Both at OpenACS and ]project-open[ we were (are?) facing the same questions. We have a lot of legacy technical issues that make it really hard for new developers to come on board. Additionally, with the database changing over time, old queries won't work on newer versions of the database anymore (did I mention legacy....).
At OpenACS we were quite successful in having a core team make suggestions for core changes and have an additional club of highly engaged users (paying companies) fund these changes collaboratively.
With regards to ]project-open[ the philosophy seems to be that you change the way of doing things and adopt code once you touch it. Therefore, if a client pays you to work on Feature XYZ, you actually factor in the time that you need to bring this part of the product up to date with the latest coding standards.
Maybe one or both approaches would help CiviCRM as well.
To many of Dave's points but especially "Teams of developers and users taking ownership of Quality Assurance, Documentation, Support, Usability and other key areas" how about putting people in charge of different key areas (ala Linux's lieutenants structure) whereby those people are responsible for making sure these things are complete (ie bug free, documanted and feature complete) for each release cycle in their area. This may be better facilitated by moving to Git. Most importantly this may mean moving release cycles back until things are truly ready to release into the wild and giving up some contorl to the lieutenants to say soemthing just isn't ready for the wild yet.
But we've been able to get one in your list managed from a group outside of the core (API) without major problems.
Don't think putting the unit tests as a separate thing is a good idea, that's one that needs to be shared and owned IMO by all the "modules". As for CiviGrant, that's one of the almost duplicate module (with civicase) that I'd like to see gone.
@xavier What why don't you like lieuteniut? Just think people could salute Dave and Lobo as they walk by them in the hall... Okay I see you point :)
I could easily see the indivual lieuteniuts (for lack of a better name at this point, and I truely have no invesment in that label) owning unit testing and likley performance and scalability etc... But I would still argue that someone need to be, as @totten put it the "unit test nazi" that holds the lieuteniut's feet to the fire on Civi* (or API's the "portable application framework").
Really quite sure I don't like your naming ideas on that one ;)
as for the test, the focus is probably more on QA than unit test, but see your point
Drupal 8 calls it: "initiaitive owners". We have this to a small extent:
API v3 - Xavier / Eileen / Erik H et al
Views * - Jalama
Book - Michael
Accounting System - Joe / Dave / Andrew
Joomla - Erin / Brian S
So would be good if we can get / recruit a few more initiative owners. Some of the ones that we definitely would like to kickstart include:
Migration to DBTNG
Drupal
Technical Documentation
Wiki manager
Infrastructure / Demo machines
Migration to GIT
from the top of my head.
lobo
Not psyched about initiative maintanier title for most of these. That term lends itself to thinking of these things in a temporary sense. Few of the examples above are short term initiatives, Migrations and CiviAccounts aside. I was thinking something more like maintainers, in Drupal terms.
That's a very interesting idea, and I suspect we could find several examples of successful communities using that model (e.g. Linux).
Just to make sure I'm interpreting correctly, the bottom four non-feature areas (API, unit tests, documentation, usability, and bug fixes) are basically orthogonal to the top ten feature areas (Civi*). The idea here is that one person might be the "unit test nazi" who monitors development of other components and ensures that they all include suitable unit-tests. Other individuals might monitor APIs, documentation, UI, etc.
As an side, it might be useful to pull out the "portable application framework" -- i.e. the basic infrastructure that allows components to find each other and put up pages. I think that there's a very different set of issues involved in, say, the "View/Edit Contact" UI's vs "Manipulate DB schema." This might lead to a component chart like this.
@totten yes by no means was my list exhaustive. I had ACL and Custom fields (which I think should be the master fields controller :)) etc.. in there at one point but once I got to 14 bullet points I thought maybe I had over done it.
Nice chart. It brings up a good point would there be one portable frameowrk lieuteniut that "oversees" the different ieteniuts under them, ie administration, explorer? I shoud l have thought of that one, asit makes good sense.
The other point to make would be these people, especially at the higher level, would be as much organizer as coder in their roles with CiviCRM (though I'm assuming they would have coding chops of course).
btw, I changed the link to the chart to be target _blank :)
I think we should be a bit realistic that it might not be feasible for consultants supporting primarily D6 sites to engage much during the period when development is all on D7 but they are on D6. Certainly I'm expecting to have less engagement in the second half of this year because of our customer base.
At least I can say for myself, having only started to get involved with actual development of CiviCRM (after using it for over three years and working with somewhat capable developers) I would not jump to CiviCRM 4.0 with our clients sites primarily because this means a lot of modules need to be D7 enabled.
Our main concern there is in having to charge the clients for upgrading the theme and things like OpenAtrium.
My assumption is that once the dust has settled a little bit we can start convincing clients to pay for the upgrade (with everything this involves). Yet this also means that new code and fixes I can contribute will be 3.4 based, which may or may not work in a 4.0 environment (though I try to primarily use API v3).
I'm not sure about the make it happen campaigns, but my feeling is that most would be developed against 3.5 first as this is what the payers are still running, but I might be wrong.
While I agree that the list is ambitious and I agree that given CiviCRM's niche it's mostly the consultants who are gonna be driving things I would still advocate for casting a wide net. I mean people like Angie and Drupal or Ryan and Joomla bubbe up because there was a strong community that had lots of places to get involved and let them really scratch their itch. Who knows if CiviCRM keeps growing there may be companies who have the leeway to give their employees time to focus on the project (ie Angie @ Lullabot, all the guys at Development Seed, Palintir, etc... whoc give tons to Drupal while still accomplishing their days jobs as consultants)
I agree that if we do all development on D7 while there are still lots of sites on D6, we can expect a temporary decrease in community development. Not so much because the community is not willing to step in but because it will require additional effort if only CiviCRM requires D7?
In light of the comments in this thread, having attended CiviCon in London, and having worked with CiviCRM for a good few years now as an implementer (not a coder), I would say that the strategic focus needs to be on:
Hi,
We are at the sprint and on a verge of a break through that would allow to share much more the codebase between D6 and D7.
However and to make it possible, we will need more people helping more actively for support. Would you be prepared to commit time to participate more actively in the forum and help foiks having issues with D* ? We are already trying to cover too many positions and would need more hands to be able to make it happen.
I'm doing the training this coming week in London as part of my game plan to improve my skills further. I'd like to contribute a bit more, subject to other commitments of course, and I am aiming to do so. Had I not had to leave Civicon early last week I would have liked to have a conversation about working with other UK based CiviCRM people to help build greater awareness of the product in the UK market. Perhaps we can have that conversation through the forum?
Hey, you should have introduced yourself! Glad you enjoyed CiviCon!