(Developers) Revising the CiviCRM installer: Library, CLI, WordPress

The civicrm-setup library aims to replace the CiviCRM installer. Following the December/January iteration, it's available for use as a CLI installer and as a web-based WordPress installer.

What's wrong with the old installer?

• Entropy: The main installer's code has grown messy. For example, it mixes active code (which executes immediately) with declarative code (classes/functions/utilities) -- and smashes them all into two files (install/index.php and install/civicrm.php) with no particular ordering. Conditionals are sprinkled randomly in a way which makes it hard to follow the big picture.
• Extensions: In the years since the installer was originally written, extensions became more important in the CiviCRM ecosystem; increasingly, key features and revisions are distributed as extensions. However, the current installation experience is split: first, install Civi; then, install the extensions. We'd like to bundle some extensions as first-class citizens -- so that official extensions are activated by default.
• Leverage: It's hard to make procedural changes to the installation process -- because there are several different implementations (e.g. the main civicrm/install/index.php web-based installer; e.g. drush and wp-cli installers; e.g. bin/setup.sh; e.g. civibuild's civicrm_install; e.g. Joomla web-based installer). The installer should have a common library so that one change to the installer can go-live on all channels.
• Design: What are the first 3 or 4 decisions an administrator should make for a new installation? On one hand, consider DB credentials: the old installer UI puts a lot of emphasis on DB credentials. But they're a bit obscure, and in many cases you can autodetect them from the CMS. On the other hand, consider components and extensions: the ones you choose to activate (or deactivate) will have an immediate impact, leading to more bigger-or-smaller menus and more-or-less configuration work. The design should put more emphasis on components than on DB credentials.
• Automation/CLI: The old installer is only available through the web UI, but many contemporary users need the ability to automate installation. This is particularly true for cloud and SaaS users. Can you perform a scripted installation? Yes, but it's a little rough: there's no way to validate system-requirements, and you have to build the civicrm.settings.php on your own, and you have to figure out the right mix of *.mysql files.
• SQL file workflow: The installer builds on a series of *.mysql files. Originally, the workflow had the upside of producing a singular *.mysql file that you could use to quickly initialize a new installation. However, over time, several requirements were mixed-in, such as translating DB content and seeding the DB with example data -- leading to an (overall) obscure workflow for building+distributing the SQL files. The architecture could be simpler and more robust, but it would require moving a few tasks from external scripts to the installer.
• Optional civicrm.settings.php: As discussed in cloud-native, maintaining civicrm.settings.php is occasionally a point of friction in sophisticated hosting environments. Making this file optional could improve portability/maintainability.

Stop whinging! How do we make it better?

Of course, it's easy to write a wishlist with a bunch of nitpicks. But if you start making big changes to an existing piece of code, that creates risks -- risks of causing regressions, risks of breaking existing workflows, etc. And if you try to fix everything on the list, then your schedule may drag on forever (without shipping anything!).

To manage these risks, one might follow the Leap by Extension philosophy. In particular:

• The major work of reorganizing the installer goes into a new package -- the civicrm-setup library.
• The civicrm-setup library is optional -- if it's specifically enabled, use it; otherwise, fallback to the existing code. This may require some narrowly-tailored patches.
• The initial release focuses on a subset of the user-base/use-cases who have an unmet need. This reduces the systemic risk to everyone else, and it provides immediate upside for the target group.

The December/January iteration

During December/January, we set out to create a new zip/build with a revised installation screen suitable for publishing on the wordpress.org plugin directory. Some of the installer issues -- such as the general entropy of the code and the design of the UI -- were clearly on our path. So we bit the bullet and started the installer leap. This produced a few concrete deliverables:

1. civicrm-setup: The library is now published. It reorganizes most the installer code using a documented API, data-model, and small plugins.
2. civicrm-X.Y.Z-wporg.zip: The civicrm.org infrastructure provides a new build, civicrm-X.Y.Z-wporg.zip, which incorporates civicrm-setup and a new installation screen. This screen presents a different mix of options, chosen to meet policy/design guidelines for the wordpress.org plugin directory. (The new build is currently available as a NIGHTLY autobuild for v4.7.31+ on https://download.civicrm.org/latest; for an official release, it should appear in March.)
3. cv: The command-line utility now uses civicrm-setup to implement subcommands cv core:install, cv core:uninstall, and cv core:check-req. The subcommands have automated test-coverage for CLI installation on single-site deployments of Backdrop, Drupal 7, and WordPress.
4. Developer documention: There's a summary of main concepts, a getting started tutorial, and example snippets for writing new installers or new plugins.

Of course, we're not trying to solve every possible issue with installation -- for example, the Drupal 7 installer hasn't changed. But we've made progress. How does this iteration stack-up against our original list of issues?

• Entropy: The civicrm-setup library is better organized and better documented.
• Extensions: To activate an extension during installation, write a small plugin with a statement like $e->getModel()->extensions[] = 'org.example.myext'.
• Leverage: civicrm-setup is a library/API which is amenable to use in many channels/installers. However, we should eventually update more installers to use civicrm-setup.
• Design: The prebuilt installer UI puts less emphasis on DB obscura and more emphasis on site configuration. Downstream installers can use/revise this design (as in civicrm-X.Y.Z-wporg.zip), or they can build new UI's with the installer API (as in cv, which implements a command-line interface).
• Automation/CLI: The cv command supports automated installation, uninstallation, and requirements-checking. The PHP API is available for other forms of automation.
• SQL file workflow: There's no functional change, but the relevant code is better organized -- making it more ripe for replacement.
• Optional civicrm.settings.php: cv core:install requires a CMS-first bootstrap script, and it now has a first-draft implementation. As discussed in cloud-native #7, the CMS-first bootstrap script is an important component which will help remove civicrm.settings.php. However, that's a bigger project -- for now, the installer must still generate civicrm.settings.php.

Future iterations

We don't currently have any budget or manpower allocated for future iterations -- the Dec/Jan iteration has already covered more ground than we needed to meet our original goal. However, there are several topics that are ripe for progress -- contributions of code or funding would enable them to move forward. For example:

1. Expand test-coverage for multi-site deployments
2. Adapt Drupal and Backdrop web UI
3. Auto-activate on WordPress
4. Add Joomla CLI support
5. Simplify SQL and translation pipeline
6. Reconcile drush/wp-cli installers