At CiviCon London 2014 the topic of extensions came up in a couple of conversations. Should some extensions be part of core, how do we deal with really good extensions and really bad ones, should we show how many times an extension is downloaded etc. Jaap Jansma and me discussed some more with Lobo on IRC and we decided that we would start with extension reviews. We feel that the extension mechanism is really cool and helps us a lot as developers and as users. We want to stay true to our community spirit, do things together and make sure we can all contribute. And reviewing extensions seemed like a good idea to us.
CiviCooP role and yours
As CiviCooP we will review 2 extension each month. This review is ofcourse nothing more and nothing less than our opinion. And obviously it has very little merit if it just stays our opinion. It might tell you more about Jaap and me than about the actual extension :-) However, if YOU review an extension as well, we suddenly might have a couple of reviews, which actually starts adding value! So please add your reviews of extensions whilst we are in the process of thinking this through and taking our first steps.
Review mechanics
Our initial thoughts are that we take a look at these topics:
- Documentation: is there a clear description of the extension, does it tell us what it does and does it work as documented
- Functionality: are we happy with what the extension does, does it seem logical, do we think it will have merit for (some of) our customers
- Code QA: does the code follow the CiviCRM standard practices, does it seem logical to use and is it relatively easy to understand if we would want to enhance it
- Ease of use: is it easy to install, configure and use
We aim to use 1-5 stars for each topic. Ideally we would want to see those reviews within the extension information on the CiviCRM website, but for our first reviews we will add blog posts.
We'd love to hear your thoughts and see your reviews too!
Erik
Comments
Great Idea Erik, count us in to review 1-2 extensions a month!
Personally I think that the CiviCRM Extensions model to follow is something similar to the Drupal's Modules.
Each Drupal's module has its own "web page " with reviews, documentations, commits, followers, versions to download. So it's very easy to undestand (or should be) what the module is about, and all the info in centralized in one place.
cheers!
Erik - I think this is a great idea & makes sense. But on a personal level - my preference would be to opt out because I'm in two minds already as to whether publishing extensions creates too much pressure on me & would prefer to unpublish them then to try to figure out what I might need to do to them to feel comfortable with someone publishing a review on them. The extensions I have out there are in various stages & published (or not) to the ext dir for different reasons - often on request.
I know discussion is good & some of the things I have been working on (GUI api import, embedding drupal views in CiviMail, making some contribution pages only support renewals (ie. membership options removed from price sets if not a renewal), support extension for managing entity settings, relationship based permissions) could probably do with some exposure - but the idea of having someone do a review on code I don't feel is ready for a review - or that is due for a re-write makes me uncomfortable. Probably most of the stuff I have published is fairly old as I don't usually publish stuff now.
Eileen, I can understand what you are saying, and would feel the same on a personal level. On the other side, if we have a list of downloadable extensions on our site we are suggesting to end users they can download and install them, and they work. Perhaps it would be an idea to categorize, so we would feel free to publish extension in the category 'love to share but might need some tlc'. On a personal level, I would probably be happy to post some extensions within that category that I would now not publish because I can not guarantee they work out of the box. What do you think?
so the premise of open source has always been: share stuff and someone somewhere will find a use for it.
Yes, there are lots of cases where we feel that the code is not ready to be distributed (and hence reviewed etc). But that same code could be a great starting point for someone else to take a look at and contribute improvements and patches and address some of the first implementation drawbacks
think of it as more of an agile model, where you are releasing code that does the job, works for you but can still benefit from a ton of improvements and you have lots of ideas on where to go with it but dont have the time to commit to it right now
I think that is a good channel to get more folks to share and then contribute stuff back to extensions that they've tweaked and modified. Our ecosystem is small enough that I do think sharing of all things is worthwhile.
The reviews can help and get folks interested in contributing to focus on improving some of the things mentioned in the review. Sharing is not a guarantee that they do what everyone needs, it just means it does work in your specific instance and solves your problem
my 2 cents
lobo
It's probably the star ratings that make me uncomfortable. There is a difference between someone writing "this code appears to be a work in progress" & giving it 2 stars. The first is constructive - the second is a disincentive. I don't necessarily have a problem with the stars being put against extensions that have been polished for distribution but feel uncomfortable with them on less complete offerings
So the idea of having two "categories" of extensions would help you to continue contributing?
I don't think the star notation is a good idea, and it pushes a message that is opposite as what you want to achieve:
IMO, reviews should be a way to help on the promotion (and we all need a hand on that department ;), provide constructive feedback and, ideally help finding contributors to improve.
On the other hand, giving stars sounds like a teacher judging the work of a student, not quite what we aimed at (unless you give good points to everyone, of course ;)
One thing missing in the review is "I'll tell you the -or some- use case for that extension and how it could help you". If you don't have this story for a specific extension, it's probably better to review another one instead. Actually, I'd suggest renaming "review" as "showcase".
Lastly, it's great to review several, but it would be better to space their publication, to avoid the wall of reviews on the the blog. Say publish the first one, wait a few other type of blog post and publish the other review.
As a Joomla user that has had at least one issue where the extension didn't behave as I thought (this was a while ago so I won't name names...) I'm also curious how "rigorous" testing will be against all supported platforms.
After all, isn't that part of the point of extensions?
Do I hear an offer to start rigorous testing against all platforms?
Good points, Eileen and Xavier! Obviously we do not want to keep people from sharing! At the same time, personally I do not mind a review with stars (knowing I will fail miserably myself). I think it is beneficial for the community to have some statement about the extensions. I am sure we all have our own ways of selecting what Drupal module, Joomla extension or Wordpress plugin to use. And somewhere down the line we have our own way of starring.
My proposal would be to better use the info.xml (the information we provide with our extension). If I want to share an extension, but only share it and not be burdened with keeping it up to date or making sure it still works with every new release AND do not see no benefit in it being reviewed because I know it might not work under 'strange' circumstances, I set it to 'no active maintainer' and keep it at the 'beta' stage in the info.xml.
If I do enter my name as a maintainer and classify it as a 'stable' extension, I am stating that I actively check this extension against new releases, have made sure the documentation is up to standards and it is all singing and dancing...and I am happy for the extension to be reviewed and starred.
Does that make sense?
Thanks for all points for the discussion. With reviewing or show casing extensions we have to keep a clear vision on what we want to achieve. Some of the aims might be in conflict with each other ;-)
The second and third point could be contradictionary with each other. meaning that as we set the level to high less extensions are published and vince versa. When the eco-system is working the extensions which have a decent amount of quality (again quality is not defined) they are probably downloaded more often... So in the ideal world reviews aren't neccesary... But we aren't living in the ideal world and probably wouldn't be. The extension review is therefore a way to achieve those four goals.
I like the idea of having all of this info attached to the extension page itself like Drupal does. How are you going to soft through the blog to find a review for an extension or even know one exists at all. Find to post to the blog but have to find a way to link from the extension's page itself. I think the coder needs to realize that most end users are looking to extensions to solve problems. Having a good understanding of how well the extension works and the maturity level of the extension along with use case and "Good" documentation is key to the extension. The user assumes that an extension works as described (barring there is a good extension) so separating extensions into two areas ("good to go" and "needs some work") might help the makers feel more comfortable. I also think the review helps the maker create a better extension. It's great to share with the community (I know I certainly appreciate so many "makers" here on so many levels) but not clearing defining the maturity/limits of an extension only confuses the user and could potentially lead to data disasters that users are unaware of because of the level of assumption they have with published extensions. Drupal defines their's by having a "Dev" release and then alpha, beta, stable releases. These levels are very clear what progress has been made with a module for the user and the risks associated with it. Reviewing "mature" extensions is important and helps to be sure that those being labeled as "mature" truly have achieved this distiction. As the community grows the dependence on extensions also grow. I know it's not pleasent to receive negative feedback but constructive criticism (it does X great but falls short in doing Y) is very important in our growth regardless of the area in which it stems from. Makers should not see this as a reason to release things but rather as a way to improve upon their work, with that said the importance of separating extensions that are really just work in progresses from those that perform at their most basic level helps to protect the feelings of the makers. Makers, we appreciate everything you do for us non-coders, I couldn't do my job without some of your gifts, thank you very much!
... seems like a simple way to differentiate the developer's intent in sharing an extension on the extensions directory. We can mark extension releases as Stable if and when we feel they are generally useful for a wider audience and should be reviewed for automated distribution via the in-app Manage Extensions feature. Beta and Alpha releases represent work in progress, and / or work that meets a specific use case with no guarantee or commitment for further development.
Does this seem like an approach that keeps barriers for sharing low AND helps clarify things for users? Since sharing and not reinventing the wheel is definitely on 'unfair advantage' of our platform (in the best way), I sure hope we can figure out ways to maximize sharing w/o placing undue burden's on the innovators in our community.
I have made things stable in the past because only stable seems to be downloadable through the UI & people have requested it. However, in some cases I would probably prefer they were beta - if they were still downloadable. D.org does allow the 'recommended release' to be marked something other than stable
I wouldn't discourage a descriptive/ promotional review at any stage - ie. this is what this does & this is what is incomplete & this is what works. It's the idea that something is being 'marked' against criteria it is not intended to meet that I find off-putting
Clear :-) So the solution where we clearly state that if you put your extension up as 'stable' and 'actively maintained' you get marks, and if not you might get a review but no marks would work for you. I would sincerely hope so because anything that would stop you from sharing would be a wrong solution!
that means they can't be automatically distributed doesn't it?