If you attended the San Francisco CiviCon in May this year, or watched Coleman's very entertaining "Less Code, More Features" presentation on the evolvement of how data structures have been modeled in CiviCRM over the years, you have seen a seemingly unconspicious CiviCRM extension in action, the Entity Construction Kit, or ECK in short.
And while, due to its name and lack of distinct functionality for a specific use case, you might think that this is just yet another toolbox extension that developers tend to add as dependencies for the actual stuff, this extension has been a remarkable journey for us at SYSTOPIA, where this extension was first born as a utopian thought, then emerged into something that caused major improvements in CiviCRM Core, and is now a viable swiss-army knife for both, developers and site builders, that - in our opinion - could well use some more advertisement. So in this blog post, I want to recapitulate the development history and present some real-world use cases that made our lives so much easier, and actually made us implement CiviCRM for some of our internal processes ourselves.
We need "Things"
The ECK had lived rent-free in my head since I started to work with CiviCRM as a developer. Coming from Drupal, exposing the concept of entities as an abstraction for modeling data structures of basically anything to the user was not new to me and I actually wondered why CiviCRM data structures were so limited compared to Drupal.
Also, creating new entity types used to be a pain, as Coleman impressively showed in San Francisco; you had to do over 20 steps of development (which were actually writing code!) and repeat a lot of work for every similar requirement. And as developer resources are limited, I remember resorting to repurposing existing entity types for different use cases, e.g. having contact entites for projects, activities for anything related to contacts (because of the implicit relation to contact entities), or campaigns for whatever had a relation to entities that can be assigned camapigns.
However, none of those requirements came with a budget large enough to justify a more generic approach when the alternative was no initial development at all, and developer resources were needed for other things. But then came up another somewhat generic requirement with a promising long-term objective and thus a budget. We started creating CiviResource, a framework for managing human and non-human resources and for assigning them to e.g. events. The Workers' Samaritan Federation in Germany (Arbeiter-Samariter-Bund, ASB) need to keep track of their volunteers and a lot of equipment for both, events, but also operations with their ambulance vehicles.
CiviResource non-human resources were what the ECK would perfectly be able to provide data structures for, because we didn't want to hard-code an entity type "Equipment" or "Vehicle", as more types of things were foreseeable to come up, and a generic "Thing" entity didn't seem viable either. And so I finally started the development of ECK. The concept seemed easy enough, and the name was snitched from a module that did the same thing in Drupal, so I started digging and debugging what Core provided us with.
Obstacles
The ECK introduces a new entity type called "Entity Type". In terms of data structure, this isn't special, because, as any other entity type in CiviCRM, this is a static concept, a database table basically, and a UI for creating records. One thing that caught my eye though was the missing differentiation between "Entity Type" and "Entity" in CiviCRM terminology (and so in Core code). For example for contact entities, the term "entity" referred to:
- the data structure (entity type) for contacts
- a contact record itself
- the API entity named "Contact"
- the result of an API call to the "Contact" API
This made reading and understanding relevant code tough, and filled the list of obstacles: DAO classes as controllers for entity records were not made for dynamic entities at that time; fields and table names were stored in static members, CRUD (Create, Read, Update, Delete) methods couldn't receive parameters for passing the entity type. So the plan of using a single controller for all ECK entity types didn't work initially.
Core Team Attention
That's where I reached out to the Core team, and Coleman answered …
So I'm excited about your project and I was going to set aside an hour or so today to dig into it with you
This turned into several calls and eventually dozens of hours on his end being put into pull requests to Core for clearing the path for the ECK, and made Core itself a lot better as well, especially as APIv4, SearchKit, and FormBuilder were in the making and required some adjustments to entity structures anyway. Coleman opened almost 80 PRs to the ECK repository until now, so thanks a lot for this joint effort! I also learned a lot about Core development, as this was the first time for me to really dig deep into it, also resulting in some PRs to CiviCRM Core.
I took the ECK to the 2022 Berlin CiviCamp, but obviously failed to generate enough attention so that just Coleman and me sprinted remotely on some UI improvements.
Also, we created a new custom field type for referencing entities, just as the Contact Reference field type did, but allowing all CiviCRM entities, including ECK entities. Thanks to my colleague Dominic for helping especially with follow-up improvements like support for merging contacts and referential delete behavior!
The Result
The first stable version 1.0.0 has finally been released, after 15 beta releases, which included a lot of improvements following modern development principles, like tests, static code analysis, code style checks, etc. and has been in use in some of our customers' installations for 2-3 years now.
With the ECK, you can create new entity types with just a couple of clicks, add custom fields to them, build relations using entity reference fields, and model your data structures in whatever way you like. The use of APIv4 provides you with automatic and customizable UIs powered by SearchKit and FormBuilder, and leveraging all those fundamental functionality, you can get views in the contact summary, lists, searches, forms, a REST API, exports through managed entity support, and developer integration via hooks and Symfony events for your custom entity types. The entity reference field type makes it actually a really powerful tool for integrating with existing entities and functionality.
If you want to see it in action, have a look at Coleman's presentation, where he builds CiviZoo as a probably not-so-practical example, but outlines the concept and how big of an advantage this is over all those many steps of manual development that the ECK marks obsolete.
Examples
Let's have a look at some real-world example use cases which we implemented using the ECK.
ASB Wünschewagen
As mentioned, the Workers' Samaritan Federation Germany, more precisely their regional chapter for the German state of Schleswig-Holstein, were first implementers. "Wünschewagen" is a German wordplay for "dare to wish" with a second meaning of something like the "Wish Wagon", and is a project for fulfilling wishes of seriously ill people, mostly even last wishes, like seeing the ocean, visiting a theme park, swimming with dolphins, or attending a soccer game. The "Wünschewagen" is an ambulance vehicle fitted with equipment like respirators and staffed with trained personnel for guaranteeing a safe excursion.
The ECK is used for keeping track of data of the vehicles, with custom fields for crucial information like whether all equipment is up-to-date and disinfected properly. Also, there are references to contacts and the integration to CiviResource for assigning staff and planning operations in CiviEvent.
HiOrg-Server integration
HiOrg-Server is a German-made software for managing staff for aid organizations and has been used by the ASB Schleswig-Holstein as well. Ideally, CiviCRM would be used instead, but the software offers some advantages out of the box that would be at least a lot of configuration in CiviCRM, e.g. multi-client capability, and was already in use. As it comes with an extensive REST API, we built a synchronization for importing data into CiviCRM.
Since there are a lot of contact-related records like education, regular validity checks, licenses, etc., the ECK is used to make all of those records ECK entities with a reference to the particular contact. This also allows for easier additions to those records, e.g. a new date field for a due date, or fields for external IDs of all those records in HiOrg-Server, which makes regular synchronizations easier. The implementation uses ECK's APIv4 endpoints for saving entities, which the ECK comes with. The extension is thus quite lightweight compared to what would have been necessary before ECK. And the best thing is, the synchronisation is still generic enough to be used for other implementers, with minor adjustments being made in a project-specific extra extension that hooks into the process where needed.
SYSTOPIA CRM
Although we've been providing CiviCRM implementations and service for over 10 years now, we didn't use CiviCRM ourselves until a year ago, and that had to do with limitations of data structures and thus modelling processes. We used (and still use) other tools, but since with APIv4, SearchKit, FormBuilder, and the ECK, storing data, building views, docking onto other apllications via the REST API, etc. got so much easier.
One thing we use the ECK for is an internal directory for all our 100+ extensions that we maintain. So there are entities of the entity type "Extension". Custom fields contain links to the repository, the documentation, lead maintainers, current and next versions, level of maintenance we provide for each extension, and so on.
Outlook
With a first stable major version, the ECK is ready to use for all your creative use cases, but such a generic thing is never finished.
There is one open PR for adding integration with CiviRules, which combines another powerful generic concept, making it even possible to add functionality to ECK entities without writing code.
Another open PR is for allowing ECK entities to get assigned tags.
Allowing ECK entities to be soft-deleted (aka moving to and restoring from the Trash) is another thing that might be easy to implement.
Also, some technical issues, like adapting for changes to the CiviCRM entity schema, are on the roadmap and can use your help or support.
There have been considerations to integrate the ECK in CiviCRM Core eventually, according to the LEXIM (Leap by Extension, Iterate by Month) principle that CiviCRM is following. At least we talked about it with Tim Otten and Eileen McNaughton with the Core team at the 2024 Hamburg CiviCamp, when the ECK was still in early alpha.
Epilogue
Thanks to all people involved in creating the Entity Construction Kit, helping with development, answering questions, testing, writing documentation, etc. Thanks for this journey!
And to everyone using CiviCRM or considering to do so: Try the ECK for your use cases, be creative and let us know about your stories, or help make the ECK even better by reporting issues, improving the documentation, or reach out to us for more support.
Jens.
Comments
Great writeup! What a journey it's been. Glad to have been a part of it.