For a while we have been thinking about some new infrastructure for CiviCRM's user and administrator documentation. We'd like to move the guide somewhere that has better support for more of the features that we want, which include
- simple user interface that makes it easy for people to contribute
- produces clean source code
- produces a 'good looking' html version of the book
- ability to export to different format (including ebook and pdf and output suitable for traditional publishing)
- version control so that people can work on different 'branches' of the documentation at the same time (that we can merge independently when ready for publication)
- translation infrastructure, so that our language communities can easily create different language versions of the manual, and can easily keep these up to date when the manual is updated
- can handle growing documentation needs (including end user documentation for extensions)
- uses widely used standards that will outlive any particular platform choices we make
Following a few different conversations at different sprints over the years we grew fond of the idea of markdown stored in a git repository as a solid base that will help us meet many of the above requirements and started to look for a set of open source tools that would allow us to do this. Gitbook "Modern book format and toolchain using Git and Markdown" seemed to tick a lot of boxes and so (while waiting for my plane at Denver airport) I thought I would take advantage of the free wifi and give it a test drive. I wrote a simple script https://github.com/michaelmcandrew/booki2gitbook that takes output from booki (our current tool) and turns it into something suitable for gitbook. You can see the result here: http://gitbook.civicrm.org/ (this link will likely stop working once we find it a proper home). It is an alpha - lots to neaten up and lots of details to work out, but a good starting point for discussion.
You can see the source code that creates this book here: https://github.com/civicrm/civicrm-user-guide. Each part of the book is a directory and each of the chapters in that part are files in that directory. All images are stored at /images.
Some areas we need to think through some more, with some initial thoughts:
How do we handle translations?
Symfony also store their documentation in github and have parallel repositories for each language e.g. https://github.com/symfony/symfony-docs and https://github.com/symfony-fr/symfony-docs-fr. I suspect that they have a fairly well developed procedure for handling the translation and updates of their documentation that we could learn from. Anyone have experience of similar documentation projects and how they do their translations?
What about extension documentation?
I think the obvious approach is for each extension to get its own repository which can then be registered somehow on civicrm.org so we can create a library of extension documentation.
What about the user friendly editor interface?
Or more generally, how can we make it as easy as possible for people to contribute to documentation? I think we have two broad types of editors that we want to encourage.
- drive by edits from people who want to quickly make a correction or add some missing info - these people want a simple login to a web based interface.
- large and ongoing edits by power editors. These people will probably want to take advantage of desktop editors and so on, and might be branching and merging in git.
For better or worse, the vast majority of people in our current documentation community fit into this second category and I think they will be well served by this move since they will have a lot of options to choose from (both desktop and web-based). But I am not sure yet what the best option is for a simple UI for the drive by editors. Right now we could ask them to use the gitbook interface
- sign up for a gitbook account
- import the the repository
- make the at gitbook.com using their nice web-based gitbook interface
- submit a pull request back to the main repo
Is this too onerous? It wil definitley discourage some drive by edits I suspect, and it would be great to see if we can put together a simpler process.
Why gitbook? what about readthedocs?
Just to clear it up, although gitbook run a hosted service, "The GitBook toolchain is open source and completely free, the source code of the tool is available on GitHub." (from http://help.gitbook.com/index.html) so it seems like an appropriate tool for us. And since we are using git and markdown as our underlying infrastructure we can complement it with other open source tools if we want to, and migrate away in the future if a better platform surfaces. The other project / community that I looked at a bit is http://readthedocs.org/ and the associated http://www.mkdocs.org project. They offer a fairly similar set of tools and it is probably worthwhile investigating what a readthedocs set up would look like and how they compare. Anyone have experience with readthedocs? The export script could be fairly easily updated to output to readthedocs, I think.
How does this sound to people? What have I forgotten? It would be great to hear your ideas on how we could move forward on this, and also great to get a few people together who can help get it ready for the sprints we have coming up in September in DC, and October in London. Some hints for getting stuck in: either sign up for a gitbook account and import the repo https://github.com/civicrm/civicrm-user-guide or download the desktop editor, clone the repo locally and open it in the editor. And let me know what you think!
I particularly like how Symfony shows translations in the HTML version of their docs, ex:
Eventually it would be nice to have it per-version (on the HTML generated book), and therefore the translations don't appear until they have been validated for the new version (although that might be a ton of work).
Michael - thanks for taking the lead, a new documentation system is vastly needed and this is no small task!
I have been looking at readthedocs for a while (remember our discussion at the Tahoe Sprint) and would highly recommmend it. It has been widely adopted by other open source projects (Django universe, bootstrap, phpMyAdmin, plone, ...) and I have only heard great feedback on the tool. Just like gitbook the full toolchain is open source, relies on github as back-end storage, and they provide free hosting for Open Source projects. The user interface is nicely laid out, it interfaces with github hooks, manages multiple versions of the documentation, export to PDF, HTML &ePub and ... handles localization (http://read-the-docs.readthedocs.org/en/latest/localization.html).
As a side note, here is what Drupal has proposed for a D8 user manual: https://groups.drupal.org/node/470648. Nothing significant though.
Great work, Michael! I think moving the CiviCRM documentation to at git based framework would be a huge step forward, be it GitBooks or ReadTheDocs. Especially, if there is tooling available to manage the localised versions/branches - to my understading this has been a major pain so far.
Just one thought on the "drive-by-contributors": maybe a simple commenting function would be sufficient. These people usually wouldn't write whole paragraphs, but rather correct errors or add clarifications. And it would be rather easy for an editor to process the comments every week or so. Now, I haven't seen such a feature on GitBooks, but I know that GitHub has it, so there might be a reasonable way to get it.
Another thought re: "drive-by-contributors" -- using Git/Github/Markdown seems like a plus from my perspective. Part of the barrier to making a drive-by-contribution (in any medium -- wiki, book, code) is learning the process, creating an account, setting up the tools. For one constituency (developers), using the same set of tools/accounts/workflows should make it easier to do drive-by-contributions.
This seems great. Does this mean we'll be able to use pull-requests, offline editing, and other neat features from git?
The focus here is on the User Guide, but I'm wondering about the developer documentation. When working on core code (APIs/interfaces), I often wish the devdocs were part of the main civicrm-core.git -- we could then make more atomic changes (ie using one pull-request to add an API *and* document it) and perhaps colocate the files for easier cross-referencing (eg putting an *.md file in the same folder as the corresponding code). But I'm not too familiar with the Gitbook/ReadTheDocs design. How sensible or difficult would this be?
Agree that read the docs is worth investigating some more. IMO we should stick with markdown over RST since the syntax is a lot easier to understand. Readthedocs has a related project called www.mkdocs.org that defines a standard for markdown related documentation. I can further develop that export script so that it creates a mkdocs compatible repository.
Read the docs assumes that the docs are part of the main repository so we can potentially have a user and developer guide built into civicrm-core at docs/user and docs/developer. Not sure how unweildy this would become wrt all the images that we like to include in the user docs and translations of the user documentation into different langauges but one option would be keep developer docs in civicrm-core and the user documentation in a seperate repository.
To be honest, I don't think that either readthedocs or gitbook have particularly brilliant translation handling to the extent that Mathieu is talking about but along with gut they provide a solid base to start.
I agree re: comments, and a commenting system is definitley part of what we'd like, though I still think that we want to make it as easy as possible for people to graduate from commenters to editors.
I have just spent 7 days using GitBook in a fairly concentrated fashion to fix all the formating errors introduced when the book was migrated from booki (3,000+ lines changed individually). I created an account, imported the repository, linked it to GitHub, clicked on the pencil icon and started editing straight away Any changes I made were automatically pushed to my github repo when I saved the file. I found it easy to use when just editing text, but re-odering sections was a bit more challenging. Overall though I think with a little bit of "how-to" documentation I think this would be easy for a non-techie to use. This is important as at the moment it is mostly non-techies who update the user and admin guide.
Then I decided to try Read The Docs. I created an account and imported the repository. I linked it to my github account and looked for the edit icon - there isn't one!. After hunting around a bit I believe I discovered that I had to install MkDocs and I could only do that if I already had Python installed. If that is really what is needed then I can't see that Read the Docs is a viable contender for hosting the user and admininstrator guide. Did I miss something?