First off, I don't know if anyone is working on a desktop CiviCRM application and I've pinged Lobo and Dave Greenberg on the subject and they haven't any knowledge of it either so this is all just conjecture unless someone is currently working on a desktop based CiviCRM connected application.
Background
The other day as I was thinking about some of the projects we are doing with CiviCRM, it struck me that the web server request/respond bottleneck can be a real problem for some of the day to day tasks a CiviCRM administrator tends to do. Nobody complains about the massive administrative feature-set that CiviCRM encompasses but I do hear complaints about the speed it takes to do various searches, import/exports or updating contact records on a per-contact basis. For me the speed seems pretty typical for a website based application but I suppose if you're used to using Act, Filemaker, Access, Excel or whatever desktop solution was used before it can seem a bit time consuming as you wait for the page to load with your results.
One Workaround
On occasion our clients have used Filemaker Pro for their contact/membership needs so to help ease them into CiviCRM, we've tied a Filemaker Pro front end into our MySQL CiviCRM database (using a Filemaker ODBC plugin) and mapped it so that some common activities can be managed through the Filemaker Pro interface. This isn't an ideal solution and doesn't provide access to all of CiviCRM's capabilities but it has helped a few of our clients make the leap to CiviCRM and as a side effect it provides an efficiency improvement for the many contact record related activities providing the computer it's running on has a decent network connection.
A Better Solution
While thinking about the pros/cons of bottlenecks and workarounds it struck me that the real solution would be to have CiviCRM on the desktop. An application that provides all of the CiviCRM goodness but it lives on your workstation as an application and connects to the CiviCRM data source.
Beyond the idea and looking to see if anyone has started a project like this, I haven’t pursued it any further but I’m interested in hearing other people’s ideas and opinions. Who knows, maybe we can start a working group and make this a reality. Let me know what you think.
Cheers,
Andrew
Comments
Andrew,
with sympathy for the need for the desktop user to have faster access, data synchronization always becomes the issue. I add an activity for a group of contacts, you send an emailing. Whose activities should make it into the 'master' ?
I do believe a desktop client is useful, albeit a tough task to complete. IMhO, it would use the API as a webservice ... still impeding the performance somehat, but still better than running the HTML over a web server connection.
I don't think there will be any issues with data synchronization and even data collisiions can be avoided by using whatever rules CiviCRM currently uses to decide who takes precident when a contact updates their account and the same time an administrator does. In the case of an email newsletter, once it's set to go, it is queued and runs when cron is pinged so again no worries. If you add your contacts to the group I'm sending to and you don't get them added until 5 minutes after I send the email, they won't get it but that wouldn't be any different than if I were logged in in Vancouver and you were logged in elsewhere... It's more of an organizational issue rather than a technical glitch.
An API service would be the safest route I think. We did have very quick results working directly with the database of course and that's what got me thinking about it in the first place.
synchronisation is one of the really really hard problems to do right. CiviCRM doesn't have any mechanism and safely avoid the problem with a lock (warning that someone else works) for the contact and for the vast majority, simply takes the one that modified it last as the winner.
If import is your example, it means that you will have pop-ups asking your help to decide which part is the correct one very often. It means that you can't really synchronise automatically because users would hate being interrupted in the middle of a meeting with a reqest to decide what to do with the fax number of acme that is different between the versions. Or put a complex queue/sync in place
Seriously, it's a massive effort to do it right, almost easier to migrate to couchdb ;)
Probably ways of making it easier first, but then the risk is that it will destroy data or create duplicates.
You're not going to get any argument from me that synchronization isn't a problem but I don't see how updating or running an import via the existing web interface will be any different from importing via a desktop application if the desktop follows the same "rules" as the web based version. The behaviour should at least be consistent.
If network slowness is the problem and you don't need to share the database over the network, you can actually install civicrm all on one computer - developers do this all the time to have copies to work on. You'd need decent RAM/CPU/disk or else it will still be slow, but if the problem is mostly network or you're running on lower-end shared hosting it would eliminate that.
But if you're saying you want the database to still be elsewhere/shared then network will still slow it down.
Hi Dave,
It's not the connection that's the problem.... The bottleneck is the typical request/respond latency in the world of websites. A whole lot of interpreted code has to run before the task is accomplished and the page is refreshed with the results. What we found is that when we hooked the desktop up to the database directly (via a Filemaker Pro front end), it was almost like we were running an application on the desktop with a local database. It wasn't as snappy as if it were local but it was a lot faster for searching and updates.
Oh and yes, we definitely want to run CiviCRM on the internet rather than locally on an inhouse box somewhere. The idea is to centralize the organization's data using a CiviCRM based website but have many tools to expose, access and manipulate the data.
Ok, got it. I misunderstood.
My personal preference has always been desktop programs. I think I've just accepted the tradeoffs when trying to get buy-in.
I was thinking about your statement "A whole lot of interpreted code has to run before the task is accomplished and the page is refreshed with the results." This is not always the case with careful use of Ajax/JavaScript. For example, the introduction of the Ajax inline editing of contacts in version 4.2 made a big difference to my users, as it avoided the whole reload of the page issue.
I think if we can think about the tasks/processes that are currently slow as compared to a desktop solution, then it would be useful to think about how Ajax can be used to streamline thoses tasks and minimize the the whole issue of forcing the user to wait on a page reload.
Oh, that's a good point; Ajax makes a pretty big difference. I've been using it in my extensions and it really does speed things up quite a bit. It could be that as things develop and the interface gets more AJaxy, the percieved sluggishness becomes a non-issue.
The desktop version solves two different issues:
1) being able to have access to the contacts when offline
One option might be a "offline ready" page that uses localstorage to keep a copy and offer an option to search (simple UI like civimobile). Or tools to sync with the exiting local address books (I'm working on a ldap interface for thunderbird/outlook)
(I'm assuming a read-only version being ok enough)
2) better "batch focus" UI. Something like excel but online (the ajax suggestions above).
Eileen is working on a grid based tool for some data that would IMO be acceptable
Not desktop, but close enough.
The UI tool exists, it's "simply" a matter of implementing and funding the work.
3) remove the need of extra requests
Some of the slugness impression might be due to ajax. If you have a page that loads fast but then needs to fire a few requests in the background, it will look slow.
For instance the activity tab, when you click on the tab, it queries to get that tab, that fire one or two queries to get the table that lists the activities.
Another one is the checkboxes to select contacts that is an ajax request for each click (on new versions). It introduce a small delay for every time your select a contact in the list and some of my users want to get rid of civi because they found the lack of imediate action unbearable.
Bottom line, it takes a lot of dev time to make a fast user interface ;)
I just wanted to comment and thank you for pointing me towards Eileen's work. I'll have a look at that. I'm also thinking about the offline idea as well. We've actually done something like that with one of the Filemaker front ends. We would run it offline and then do a batch update when it was ready to go. It was vary good for the limited work that was being done and of course appeared to be very quick.
Thanks,
Andrew
I haven't got anything on the batch entry to make available yet - however, my focus is specifically on making it possible to use existing profiles but to enter them in a more spreadsheet style layout. I'm focussing on the data-entry first & then process all at once because our requirement is to do that manual validation.
I am wondering if this could be accomplished by using the "N to 1" multisite capability. You would follow the multisite instructions to create a civicrm instance on the local machine, and point the DSN to the mysql database on the remote production server. Apache and PHP would run locally along with the browser. This assumes that the server host would allow this remote access, or could be talked into doing so.
Well it's a bit difficult to say because this is all just conjecture at the moment but as far as I'm concerend a best case scenario for speed and efficiency is going to connect directly from the desktop application to the CiviCRM database and completely bypass apache/php.
I'm not sure how multisite would affect this because I've never had a case that required a multisite CiviCRM installation. If this idea gets any momentum it will be a bridge we'll need to cross. I suppose the app could read CiviCRM's civicrm.settings.php file to figure out how many CiviCRM sites there are and then spin up a new instance and map to the appropriate fileds for each one. I don't know??
Andrew
The latency problem is solved by using a local data store in the browser and sending batches of updates across the wire. That is how your ODBC connection via FileMaker Pro will be doing it. If you display 20 rows in the browser, then the typical design pattern is to have 1 more page going backward and 1 more page going forward in the local dataset store. This allows scrolling and saving without latency that is apparent to the user. As the current row the user is working on moves, the code in the browser reads another batch of records beyond the end of the local store in whichever direction the user is moving (which is much faster than doing 1 row per trip) and saves a batch from the other end of the local store if there were updates, or otherwise just throws them away. This is original design pattern for AJAX - see http://en.wikipedia.org/wiki/Ajax_(programming).
If we developed a batch API for reads and updates, and extended Eileen's work on the batch membership edit to incorporate the design pattern above, we'd be there.