Complex event processing engines are a natural fit for event driven platforms like Mule. Native   support has been available in Mule since version 3.2 by way of the Drools Module.  The Esper Module now offers an alternate way to leverage CEP in your integration applications.   Esper is a robust, performant, open source, complex event processing engine.  Let’s take a look at how to use with Mule and then see how it compares to ’ CEP support.

Subscribing to a Stream of Tweets

The following Gist shows a Mule application that reads status updates from Twitter’s public timeline and logs instances of updates that contain the “#mule” hashtag.

To begin to use the Esper module we first need to create a configuration file that defines our event types.  Future versions of the module will likely remove this restriction, but for now you need to manually define an Esper configuration, like the following:

This registers an event stream called “Tweets”  whose events are the appropriate domain object in the twitter4j API.  Esper also supports Maps and XML documents as event types.

The “main” flow in the application above polls Twitter every 2 minutes for a sample of status updates (note that a “real” implementation of this application would probably subscribe to the firehose.)  A collection-splitter it used to split the List of status updates into atomic messages which are injected into the “Tweets” event stream using the <esper:send> message processor.

Now that we’re populating the event stream we can use Esper’s query language, EPL.  EPL is SQL-esque that simplifies querying event streams.  The <esper:listen> message source in the application above takes an EPL query to subscribe to events off a stream.  In this case we are setting a query that will generate a composite event when status updates with a “#mule” hashtag match the query.  These events are logged with the logger message processor.

Comparing to Drools CEP Support

Let’s see how Esper compares to Drool’s CEP support.  This Gist shows the Drools rules definition for the stock broker CEP example that ships with the latest version of Mule. This example demonstrates using CEP to alert when a given stock changes by a certain threshold in a given time window.  The Drools example using DRL is fairly verbose, as we can see from the Gist above.  The following illustrates the  same relevant functionality implemented with Esper:

The EPL statement could be even more concise if we externally defined the average difference function.  If your application isn’t already using Drools then you may want to consider using Esper as an alternative.  The fact EPL is SQL-like also makes it an attractive option, as you don’t need to learn DRL to begin working with event streams.

Rate Limiting Traffic

I find CEP, and Esper in particular, extremely useful when developing and maintaining integration applications.  I previously blogged about using Esper in this context for doing QoS checks on endpoints.  You can also use it to rate limit traffic on a flow as illustrated in the following example:

This example uses the <esper:filter> processor to stop traffic from passing if the given EPL statement evaluates to false.  In this example we’re stopping traffic if more then 1000 messages are received in a 5 minute window.

Functional use cases for CEP also abound.  The stock example that ships with Mule is an obvious one.  Rapidly changing data like stock prices, auction prices, RFID tag locations, etc all lend themselves to an event driven processing model.  A less obvious use case is observing the state of domain objects as they pass through a system.  You could, for instance, generate an event whenever an Order is submitting for a web store application.  A listener could then subscribe to the stream and generate a composite event when the order has been in an unfilled state for over a certain threshold of time.

Conclusion

Complex event processing is often a natural fit for integration applications built with Mule.  While Mule ships with CEP support via Drools’, the Esper module provides an alternative with a more concise configuration.

No related posts.


9 Responses to “Twitter Complex Event Processing (CEP) with Esper and Drools”

Tomas Blohm February 9th, 2012, 11:22 pm

Really nice!!! Just what I needed!
Well done!

/Tomas Blohm

Fadi E. February 10th, 2012, 6:50 am

This API seems to be very powerful.

The problem though i think it’ll only be a matter of time before it starts being misused by some.

I like how you’re passing a SQL query in the statement… Where can one find the schema of the twitter tables?

john.demic February 10th, 2012, 2:52 pm

Hey Fadi,

The query that looks like SQL is actually EPL, the event processing language which you supply to Esper to query from event streams. Sine EPL supports POJO event types the query in the listing is executing against the domain model. In this case its the StatusJSONImpl class of the twitter4j API, which we use to select the Tweets that contain a #mule hashtag. Please let me know if you want me to elaborate further.

Dmitry March 26th, 2012, 12:11 am

I tried to make this example work but failed… How it is correctly add esper support to MuleStudio? I’ve done all described in esper connector installation guide but MuleStudio doesn’t see esper connector and project fails to start on loading esper classes. So could anybody show step by step guide how to add and configure esper CEP in MuleStudio?

kronik March 26th, 2012, 12:14 am

I tried to make this example work but failed… How it is correctly add esper support to MuleStudio? I’ve done all described in esper connector installation guide but MuleStudio doesn’t see esper connector and project fails to start on loading esper classes. So could anybody show step by step guide how to add and configure esper CEP in MuleStudio?

Mule, Esper CEP and non-events | Mule ESB June 5th, 2012, 11:33 am

[...] was reading John D’Emic’s brilliant post Twitter Complex Event Processing (CEP) with Esper and Drools and got truly inspired. After playing around with the combination of Mule and Esper I was amazed [...]

clover5 July 25th, 2012, 6:58 am

Which version of the twitter4j is this example using? twitter4j.StatusJSONImpl is internal in the later version. Do we need to create our own version of it?

John D'Emic July 31st, 2012, 6:35 am

Its using 2.2.3 of twigger4j which is a transient dependency of the 2.5 version of the mule-module-twitter. This appears to be the analagous class in the most recent version:

http://twitter4j.org/en/javadoc/twitter4j/Status.html

You should be able to use this instead of twitter4j.StatusJSONImpl in my example.

David Moore February 4th, 2014, 6:11 am

You gave nice explantation to use this api. But i have problem with it, i had tried it but didnt get any result. will you tell me how to add it correctly.

Leave a Comment