Just to set better context, here is our load testing scenario:

• Implement the College Prep Application for 2007 on CiviCRM v1.6
• Deploy the above application on the network as configured when it goes live.
• Use a load testing program to simulate n students hitting the application simultaneously with random sleep/think times. If possible, simulate some form errors also. Data to be used from last year's application which has been randomized and munged with. The data is hierarchical and in a simple xml format.
• Increase the number of students in a methodical manner and measure various parameters including: load and memory utilization on each web server and database machine, network bandwidth, overall throughput, processing time per page and application.
• Optimize the architecture / CiviCRM code and improve results. Ideally we'd like to be able to handle 2K - 5K users simultaneously quite efficiently.

I did a bit more research to figure out what open source solution we could potentially use. Convinced myself that Selenium would not really help us do the above even though it is fairly easy to use etc.I suspect we might be able to tailor Selnium's output to feed into the load tester software we use

h2. JMeter

There was a fair amount of information on the net about JMeter that I felt obligated to give it a whirl before making a decision. As with most jakarta projects there is quite a bit of documentation out there that it makes things a bit easy. Downloaded and installed JMeter, got it up and running and tried it out on my small test case

• login into Drupal
• click on CiviCRM
• do a search
• view a contact
• click on Administer CiviCRM
• logout

After a few false starts and dead ends I did manage to get the load tester up and running against demo.civicrm.org with the above testcase. The UI is fairly clean and intuitive (once u get the hang of it and understand the underlying concepts). I could soon see some decent results, aggregate numbers and a few pretty graphs. Was a bit surprised to see the low throughput I was getting on both drupal and CiviCRM pages. It could potentially be my network and machine (am running the test from a box in NZ to a box in US), but seems like we do have some amount of work ahead of us!

I checked out the "test file" which was in xml and realized there might be a slight issue trying to parameterize this with our external data file. In contrast, Grinder and TestMaker both use Jython which makes things like reading from an external source and deciding what to do based on the data a wee bit easier. Did a couple of searches to figure out a solution and came up short (JMeter does have a CSV data import, but seems like it will be a hassle for us since the application adjusts dynamically to the data entered).

Between Grinder and TestMaker, I'm more inclined to choose Grinder because its a true open source project and meets our needs. More info on Grinder and how things progress in the days to come ...