We have received a great early response so far to our recent release of Mule MQ, MuleSoft’s new JMS messaging server.  The response is not unexpected, since the Mule ESB user base has been requesting that we provide a JMS server which has been tested and is supported along with Mule ESB.  Hundreds of organizations have downloaded Mule MQ for evaluation and we have been hearing great feedback from the Mule community.

Some people have been very interested in the performance of Mule MQ, and they have asked us to provide more details about the tests we have run to validate performance. It’s clear that performance is one of the most important criteria for selecting a JMS messaging product, so our product team spent a great deal of time benchmarking Mule MQ against the most popular JMS products currently available.

One of the projects we selected for comparison was Apache ActiveMQ (most product vendors prohibit publishing benchmark data). ActiveMQ is the leading open source JMS technology, and is popular with the Mule community. For that reason, we wanted to understand how Mule MQ would compare ActiveMQ under some common scenarios.

A few things that we considered as we designed our tests:

• Industry-accepted test harness - Benchmark tests can sometimes be subject to  manipulation.  We wanted to use a test harness that is publicly available and industry-accepted. We chose the Sonic Test Harness from Progress Software
• Request/reply and Pub/sub – Some of our customers have queue-based (request/reply) requirements, while others are looking for topic-based (publish/subscribe) messaging. We wanted to include both request/reply and pub/sub test cases in our suite of tests.
• Variety of use case scenarios – We wanted to test both performance and scalability, and we also wanted to see how various products performed with different sized messages. So, we created multiple tests for both request/reply and pub/sub scenarios that covered a decent set of common usages.
• Minimal product tuning – To make sure that the playing field is as level as possible, we used each product, including Mule MQ, with its “out-of-the-box” configuration, with no advanced tuning

Details on the test cases, test harness, hardware profile, and results can be found here:
http://www.mulesoft.org/display/MQ/Performance+Tests

In the end, Mule MQ performed well against all of the messaging providers that we tested, including ActiveMQ. In some of the use cases, Mule MQ and ActiveMQ performed about equally to one another. In others, Mule MQ performed better than ActiveMQ, and in other test cases ActiveMQ outperformed Mule MQ. We originally benchmarked the 5.2.0 release of ActiveMQ.  In our most recent testing, using ActiveMQ 5.3.0, the performance of ActiveMQ improved, so we included both benchmarks in our test results.

In messaging, performance testing can be a tricky since it depends highly on your specific use case, and JMS products can be tuned to optimize with a number of different attributes for different scenarios. Any JMS vendor can publish a performance test that beats all others for a certain use case. We acknowledge this fact, and we have striven to make sure that we created an objective test.  To understand performance for your own use case, I would urge you to download Mule MQ and try it for yourself!

Tags: ActiveMQ, performance

This entry was posted on Tuesday, January 26th, 2010 at 1:58 pm and is filed under Mule ESB, Mule MQ. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.