Ross Mason on Monday, July 6, 2009

To ESB or not to ESB

40

Update: I created a series of follow up posts on this topic (but read this post first):

Many of us have had to ponder this question. Technology selection is notoriously difficult in the enterprise space since the criteria and complexity of the problem is often not fully understood until later in the development process.
There is an interesting post from ThoughtWorker Erik Dörnenburg with the unfortunate title “Making the ESB pain Visible”. Erik provides a real-world example of when not to use an ESB citing that -

“Based on conversations with the project sponsors I began to suspect that at least the introduction of the ESB was a case of RDD, ie. Resume-Driven Development, development in which key choices are made with only one question in mind: how good does it look on my CV? Talking to the developers I learned that the ESB had introduced “nothing but pain.” But how could something as simple as the architecture in the above diagram cause such pain to the developers? Was this really another case of architect’s dream, developer’s nightmare?”

Later, Erik quite rightly points out that an ESB architecture should not have been used in this scenario. This is a fairly common problem for ESBs and enterprise technology like BPM/BPEL where the technology is not chosen for the right reasons and then the technology gets blamed when the project struggles. Given that much of the enterprise software disillusionment today stems from the incorrect usage of the technology I thought I’d offer some rough guidelines for selecting an ESB architecture.

ESB selection checklist

Mule and other ESBs offer real value in scenarios where there are at least a few integration points or at least 3 applications to integrate. They are also well suited to scenarios where loose coupling, scalability and robustness are required.
Here is a quick ESB selection checklist –

  1. Are you integrating 3 or more applications/services? If you only need to communicate between 2 applications, using point-to-point integration is going to be easier.
  2. Will you really need to plug in more applications in the future? Try and avoid YNNI in your architecture. It’s better to keep things simple re-architect later if needed.
  3. Do you need to use more than one type of communication protocol? If you are just using HTTP/Web Services or just JMS, you’re not going to get any of the benefits if cross protocol messaging and transformation that Mule provides.
  4. Do you need message routing capabilities such as forking and aggregating message flows, or content-based routing? Many applications do not need these capabilities.
  5. Do you need to publish services for consumption by other applications? This is a good fit for Mule as it provides a robust and scalable service container, but in Erik’s use case all they needed was an HTTP client from their front-end Struts application.
  6. Do you have more than 10 applications to integrate? Avoid big-bang projects and consider breaking the project down in to smaller parts. Pilot your architecture on just 3 or 4 systems first and iron out any wrinkles before impacting other systems.
  7. Do you really need the scalability of an ESB? It’s very easy to over-architect scalability requirements of an application. Mule scales down as well as up making it a popular choice for ‘building in’ scalability. However, there is a price to be paid for this since you are adding a new technology to the architecture.
  8. Do you understand exactly what you want to achieve with your architecture? Vendors often draw an ESB as a box in the middle with lots of applications hanging off it. In reality, it does not work like that. There is a lot details that need to be understood first around the integration points, protocols, data formats, IT infrastructure, security etc. Starting small helps to keep the scope of the problem manageable and keep the fuckupery to a minimum. Until you understand your architecture and scope it properly you can’t really make a decision as to whether an ESB is right for you.
  9. Generally, always validate a product solution for your needs. Don’t choose an ESB or any other technology because –
    • It will look good on my resume
    • I don’t need the features today but there is a remote chance that I _might_ in future
    • I had a great golfing weekend with the head of sales

This checklist is not exhaustive, but will help clarify when not to use an ESB. Once you have decided that an ESB is a good fit for your project you’ll want to add additional selection criteria such as connectivity options, robustness, error management, service repository, performance, data support, etc. The important thing to remember is that there is no silver bullet for good architecture and you need to know your architecture before making a technology decision.

With this checklist in mind it’s easy to see that Erik’s example never needed an ESB architecture in the first place.

However, if the architecture looked something more like this, then an ESB would have probably been a good fit.

Choosing Mule

Obviously, as the creator of Mule I have some bias for wanting everyone to use Mule. However, it is critical to the continued success of the Mule project and to MuleSource that users understand when not to use an ESB architecture. Open source makes a lot of sense for enterprise software because projects need time to try out the technology and refine their proposed architecture. Having access to the product and source code helps a huge amount in this discovery process and allows the customer to make an informed decision.

In fact the MuleSource adoption model hinges on the ability of the user to self-select Mule and approach us only when they need enterprise capabilities or  support. This has been working very well for everyone involved. Customers get to do a proof of concept (PoC) with our product knowing that if they end up using it for a mission critical application that they can get enterprise 24×7 support. For MuleSource it means that we enable our customers buy from us rather than us selling to them, so they always get what they want – this is a far cry from the old proprietary upfront license model that used to hold the enterprise market hostage.

Follow: @rossmason@mulesoft

Update: I have started a series of follow up post s on the topic of ESB or not to ESB that start here.

No related posts.


40 Responses to “To ESB or not to ESB”

Patrick Santana July 6th, 2009, 3:09 am

Good post.

Josh July 6th, 2009, 8:05 am

For a project I did previously, I was an am very happy in choosing Mule to do the “heavy lifting” for me. In some ways, it was practically a stereotypical usage I think. Instead of loan brokering, it did health insurance quote aggregation. Being able to *easily* fire off multiple requests and then transform and aggregate them together was immensely useful.

Additionally, when other providers came on, none of the core code had to be changed as it wasn’t even aware of individual providers. Just had to implement the code and add to the outgoing services.

Erik Doernenburg July 7th, 2009, 8:40 pm

Great checklist, and I couldn’t agree more; when used for the right reasons an ESB like Mule can drastically simplify a project. Unfortunately, though, we still see a lot of architectures that wouldn’t make it past even the first point on your checklist. And it was those that I had in mind when I wrote my article.

I have an interest in using visualisation techniques to help people better understand code and architecture. What I wrote about in that article was a technique that I have used successfully to highlight the pain that was caused by the misguided use of an ESB. I certainly did not want to imply that all uses of an ESB are misguided or that the ESB itself was the problem.

Thiru July 8th, 2009, 4:17 am

A pretty comprehensive and useful list to start with. From the post also learnt a Welsh word :-)

Schalk Neethling July 8th, 2009, 12:57 pm

Hey there Ross,

Thanks for this thoughtfull article. I am doing some architecture for a current system that needs to scale and there is a lot of talk about BPEL, ESB etc. It is awesome to have something like this checklist as back-up when you have to prove that adding, for example, and ESB to the mix is not going to add anything to the performance and/or scalability of the application.

Ed Brown July 9th, 2009, 5:37 am

This posting NAILS it!

Robert Keahey Blog | An ESB provider’s perspective on the need for ESBs – interesting July 9th, 2009, 4:41 pm

[...] ran across this blog entry by Ross Mason, the CTO at MuleSource.  Having lived in the systems integrator world for a lot of [...]

Gaurav Malhotra July 11th, 2009, 1:23 am

Spring integration opens new arena in enterprise integration.Why worry about ESB??

MKK July 12th, 2009, 12:48 pm

Ross
Wonderful article.

When I sent this article to my engineering here is what that have to say…

* We have 100+ web services so without ESB we need to manage all the communication.

* ESB would help us to Integration Services efficiently. ESB would take care of Service Transaction, Security Context, Routing, Transformation. What EJB-Container does for EJB-Object, ESB container would do same for Services.

* ESB is mandatory for Integrating In-house applications as well as global application. So its suitable for Local Companies as we as Globally spread companies.

While I am trying to get better understanding of SOA myself..
I have some questions:

* In our project we are talking 100+ services. Without ESB how do they communicate (isn’t it too much ) ( First of all I am trying to understand what makes us create 100 of services)

* Without ESB how will we get Service Transaction, Security Context, Routing, Transformation.

Can you please share some thoughts.

Thanks
Zing

Zac July 14th, 2009, 8:19 am

What’s “YNNI”? Poimt 2: “Try and avoid YNNI in your architecture.”

andy July 14th, 2009, 8:20 pm

YNNI = You’re not (gonna) need it

or something close to that, anyway.

Ed July 29th, 2009, 6:22 pm

Superb stuff! Great post…

Kamal August 9th, 2009, 11:58 pm

Thanks for the article and pointing out a set of criteria for choosing a ESB. As you mentioned, generally people tend to use these just for the sake of trying them in a real project and things get worse later; people should choose the technologies/tools if and only if the project needs them.

Should you be using an ESB? | The Magister’s Terrace August 25th, 2009, 10:14 am

[...] Mulesource’s Ross Mason has written a very nice blog post regarding wether you should use and ESB or not. Although he won’t go as far as I did in this post questioning the real need for the tools, his questions can set you on the right track. [...]

Usman Shaheen February 28th, 2010, 3:07 am

Great points to decide whether we need an ESB. quite helpful. Thanks

John May 7th, 2010, 1:19 am

Good post Ross.

I am eager to know what we need if we only get “yes” of some of your questions…. In my cases, some of checklists are yes, like multiple applications/services, plugable in the future, or publish integrated service to other applications. But some are not, like multiple protocol, forking and aggregation, or complex routing. What’s your suggestion on this??

Barry June 8th, 2010, 4:02 pm

Excellent article – concise and informative without a lot of extraneous proselytizing. I think it may be a bit idealistic to think that resume-driven decisions will not occur or are necessarily always bad. For one thing, introducing avant-garde technologies usually attracts good developers and keeps things fresh for your existing developers. I think it might be more reasonable to say that “good for resume” is a desirable, not a required, goal.

Gany November 18th, 2010, 10:57 am

Would Mule be a good idea when I want to listen to a directory where files may be added or edited asynchronously?

GaryB January 8th, 2011, 1:55 am

I want to create an event driven integration between two apps, can mule handle that exclusively as a message broker of some kind, or will I have to modify application ‘A’ where the message event originated?

Ross Mason January 8th, 2011, 11:35 pm

Gary, you shouldn’t ever need to alter the applications that you are integrating with. You just need a way of getting data out of the application i.e Web Service interface, HTTP, File, JMS…

Robbe January 13th, 2011, 10:26 pm

YNNI: You’ll never need it
or from the Welsh: energy or vigour

Waqas Sadiq February 3rd, 2011, 5:37 am

Could we also use mule at top of the web-site/servlets layer as you have describes in architecture about Rest, JMS etc. If yes, what would be the Pros and cons for using this approach.

SOA, EAI, ESB – Mule nebo Spring Integration « AspectWorks Blog March 29th, 2011, 5:15 am

[...] článku To ESB or not to ESB naleznete kontrolní seznam pro vyjasnění, zda je pro vás integrační platforma vhodná, nebo [...]

To ESB or not to ESB? « Bernard's Technocratic Blog April 23rd, 2011, 11:05 pm

[...]  But to support ESBs, here is a blog and checklist for those thinking about using a ESB: http://blogs.mulesoft.org/to-esb-or-not-to-esb/ On the PopPressed Radar Print Magazine's New Visual Artists Saint Petersburg Unveils [...]

To ESB or not to ESB? « Bernard's Technocratic Blog April 23rd, 2011, 11:05 pm

[...]  But to support ESBs, here is a blog and checklist for those thinking about using a ESB: http://blogs.mulesoft.org/to-esb-or-not-to-esb/ On the PopPressed Radar Print Magazine's New Visual Artists Saint Petersburg Unveils [...]

Srikanth May 11th, 2011, 4:33 am

Good post

Edwin Pino May 25th, 2011, 12:54 pm

Nice

From the Mule’s Mouth » ESB or not to ESB Revisited – Part 1 June 8th, 2011, 4:20 am

[...] wrote a blog post a while back about when to use an ESB and when not to.  It got a fair bit of pick up and I’ve [...]

From the Mule’s Mouth » ESB or not to ESB Revisited – Part 1 June 8th, 2011, 4:20 am

[...] wrote a blog post a while back about when to use an ESB and when not to.  It got a fair bit of pick up and I’ve [...]

Fernando Ontiveros July 27th, 2011, 2:57 pm

very nice post, in fact I learned why I will use Mule in my project.

and why will not use mule for other features

From the Mule’s Mouth » Choosing the right ESB platform August 2nd, 2011, 3:23 am

[...] often and provided some context about the benefits and the considerations of each. You can read the original post and follow-up parts 1, 2, and 3. Here is a quick [...]

Mule ESB « Another Word For It October 2nd, 2011, 4:35 pm

[...] Below is a quick ESB selection checklist. To read a much more comprehensive take on when to select an ESB, read this article written by MuleSoft founder and CTO Ross Mason: To ESB or not to ESB. [...]

Mehool January 10th, 2012, 2:24 am

Great Post

Mayur February 21st, 2012, 5:58 am

Dear All,

I am not from a technology background but contemplating to start a product firm. The product/service will have multiple applications (java based) but the user will have a single window to access all the functionality from all the applications. Some these applications are Content some of them are financial and few of them are MIS based.

DO you think ESB is the right way to go? Your insight would be highly appreciated. Thanks

Luigi March 9th, 2012, 3:29 am

Hi, I’ll try to be shortest as possible.
I’m to decide how to operate in a VisualBasic, .NET, and other legacy tecnology environment (with thousand of single batch/stand alone app) that now are comunicating together only via shared dbatabase (with thousand of tables/views)…do not leave any comment on this please! -:). I have to migrate it in a per-services architecture, politely changing few pieces each time, and to convert very old sw in java-services and trying to reuse at least .NET sw. Can Mule ESB help me in this hard challenge?

Jack April 27th, 2012, 3:23 am

@Luigi, the short answer is that yes, Mule ESB can help you. Other options are Microsoft BizTalk or to roll your own solution using MSMQ (or JMS if switching from .Net to Java is a requirement). While it seems a bit dated now, the book “Enterprise Integration Patterns” would probably help you to understand the options and challenges better. A word of warning from experience… you will likely have some unanticipated data type conversion issues communicating between .Net and Java. Just make sure you really need Java in the middle for your ESB before you select Mule ESB as part of your solution.

Was ist ein ESB und wofür kann man ihn nutzen? | codecentric Blog November 30th, 2012, 7:45 am

[...] von MuleSource) hat im Blog von MuleSoft einen schönen Blog-Post zu dem Thema geschrieben: To ESB or not to ESB. Ich werde hier nur auf die wichtigsten Punkte der dort abgedruckten Checkliste eingehen, sonst [...]

Leave a Comment