One thing that often comes up when discussing the future of CEP is how we will model metadata and domain specific processing rules in the future.  Whenever this topic arises, my mind turns to Semantic Web technology in general, and more specifically to the ontology modeling facilities that have developed in that context.  This sense was especially strong at the recent OMG conference on the future of CEP standards, where many of the requirements aired by the participants seemed tailor-made for this sort of solution.

My general intuition is that we could leverage previous work on using domain-specific ontologies to speed the implementation of automated systems that reason using first order logic.  An obvious experiment that I haven’t had time to conduct would be to specify a small rule-based application starting from an OWL (Web Ontology Language) ontology.

It was thus with good cheer that I read TIBCO’s recent blog post on this topic, which affirms that if I am mad, my madness is at least shared by others in the field.
(Full disclosure: I’ve done significant work in the SemWeb area, authoring a triple store some years ago and kibitzing on a number of large projects, all of which may contribute to my optimism for this approach).

An article in yesterday’s Wall Street Journal caught my eye:

“Racing to Gain Edge on Multicore Chips” (by Don Clark, 17 March 2008,  page B4) explains how Intel and Microsoft are turning to universities for research into new programming techniques to make it easier to take advantage of the parallel processing capabilities of modern multi-core architectures.

What struck me was that the Aleri CEP engine is just such a programming tool: it uses a dataflow programming model to allow someone defining a complex set of business logic to break it up into a series of discrete operations, where the output of one becomes the input of another. The Aleri Streaming Platform is fully multi-threaded, so each “stream” runs in it’s own thread, allowing streams to run in parallel, taking full advantage of all the cores and CPUs that are available. An application developer, using Aleri’s high level authoring tools, defines the business logic that will be applied to incoming events to produce the desired output, without needing to have any awareness of threads, cores and parallel processing. Thus Aleri takes care of the scalability and how to harness all available cores; the user simply focuses on the business logic.

We received word this week that Aleri was the winner of the prestigious Jolt Productivity Award in the highly competitive “Database Engines and Data Tools” category.  This is the 18^th year in which Dr. Dobbs Magazine and CMP handed out these awards.

The Jolt Awards, and more specifically the “Database Engines and Data Tools” category, go far beyond simply Complex Event Processing, so it really is a testament to how much notoriety and recognition the complex event processing industry is receiving.  Aleri’s CEP offering, the Aleri Streaming Platform, was the only CEP product to be given a Jolt Award this year, in any category.

Each year the Jolt Product Excellence and Productivity Awards recognize those products, books, and websites that have “jolted” the industry in the past year. The fact that Aleri won the award is a testament to our commitment to providing an innovative product that provides our customers and partners with a strong competitive edge.

A special thanks goes out to our clients and to our stellar development team whose expertise and tireless work has molded our CEP offering into what it is today – award winning.  Please click here to learn more about Aleri’s exciting Jolt Productivity Award win.

At last year’s Second International Workshop on Event-driven Architecture, Processing and Systems (EDA-PS’07), co-located with VLDB07, I presented a short paper on the internals of our log-structured storage system, which allows the Aleri Streaming Platform to absorb and persist streaming data at rates that are impossible using a conventional database.  Those of you who enjoy short, fairly technical papers might want to check it out.

In this week’s blog, Chris Skinner, one of the leading bloggers in the European Financial Services Sector makes a very strong case stating that in order for banks to remain competitive in the rapidly evolving financial markets, they need to ensure that they take full advantage of the latest advances in trading technology, especially in the area of low latency.

“In capital markets, where algorithmic trading analytics can make the difference between success and failure, technology is critical to competitiveness. If you cannot compete with another bank’s new derivatives capabilities, which are driven by technical excellence, you might as well give up the ghost. If you lose a millisecond of speed in the investment markets, you are dead. That is why low latency is at the core of capital markets today and technology strategy is at the core of latency. This is why an exchange said to me, “If it takes more than 500 milliseconds to process then it is of no value because it is out-of-date”. Technology strategy is the core focal point for the exchange. This is why an investment firm said to me, “We moved servers to Moscow because it takes 60 milliseconds to route a trade from London to Tokyo via Moscow compared to 240 milliseconds if we process the trade via New York”. Technology strategy is the core focal point for the bank. This is why investment firms are co-locating their servers within exchanges. It is also why we are seeing market blips occurring on a regular basis, because markets move in real-time.In capital markets, technology strategy is at the core of business strategy.” http://www.finextra.com/community/blogs.aspx?mem_id=29084

Chris is absolutely right, however there are additional dimensions that need to be taken in to account when banks look to establish and maintain competitive advantage through the use of technology.

Yes it’s true that once you’ve decided to trade, you want to execute the trade with minimal delay to avoid it being “out-of-date”. However, now that the markets are becoming more and more fragmented and trading decisions are becoming more complex, then not only do you want to get the execution right, you also want to get the when, how, and why right with the same speed and focus.

• For the “when” you could use the latest trade signal generation algorithms that combine indicators such as sentiment, momentum, volatility, liquidity, trade activity and other measures to provide market entry timing points for your trades.
• For the “where” you could use visible and dark pool analysis tools to identify the trading venues that are most likely to enable you to meet your “probability of execution” objectives.
• For the “why” you could use real-time analysis tools that enable you to more fully assess the ebbs and flows of the market to identify trading opportunities and risk.

The markets are becoming more and more complex as they continue to fragment and evolve and technology is the key to keeping ahead of the competition.  But just a word of caution – bad decisions made quickly are still bad decisions.

I heard an interesting comment yesterday from Coral8, claiming to be “the only fully deterministic engine that guarantees consistent results”.  Here’s a bit more of what was said:

There is a number of CEP engines that will claim to have / to be a deterministic engine.  Well, certainly, we are not only going to make that claim, but we back it up, right.  We provide a fully deterministic that we will guarantee consistent results.  We recommend that you look very closely at these guarantees and test determinism. We guarantee deterministic results, no matter how complex your queries, no matter how many streams they operate across, and more.  We don’t believe that any / all the other CEP engines can back that claim to support fully deterministic results across complex queries.

But their CEP engine is not the only one to take consistency seriously.  We designed the Aleri Streaming Platform to be deterministic and consistent from day one, and our first application in production, the Aleri Liquidity Management System (LMS), relies on it.

What exactly do we mean by “determinacy” or “consistency”?  By “determinacy” we mean that a set of input data produces the same output data.  In other words, the order in which the data arrives doesn’t matter.  By “consistency” we mean that the output is predictable from the input.  Mapping these concepts to the Aleri Streaming Platform is straightforward.  In the Aleri Platform, one configures “source streams” and “derived streams”, and data, in the form of records, enters the source streams and flows to derived streams.  Derived streams are, for the most part, bog-standard relational (SQL-like) operations: filtering (selection), computation, joins, aggregations (group-by).  So by “determinacy”, we mean that the same records supplied to the source streams yield the same results at the derived streams.  By “consistency”, since the derived streams are SQL/relational operations, we mean that the results are the ones we’d expect.

The Aleri Streaming Platform satisfies these criteria if one sticks to the core stream operations.  But it goes further, in providing FlexStreams that allow the user to code up pretty much any computation on record events with conditionals, loops, etc.  The upshot: if you remain with the relational subset (as LMS does), the Platform guarantees consistency.  If you need to step outside, you can.

We go one step further in our determinacy/consistency story: the Aleri Streaming Platform, unlike Coral8, supports updates, upserts, and deletes as well as inserts.  Those operations can be used to control the size of data, as a previous entry has noted.  It’s harder to guarantee these properties with these operations, of course, but they can be extremely important.  I wouldn’t want a CEP engine without them.

The following is in response to a blog that I found interesting by Michael Groner, VP of Engineering, at Appistry regarding EAF providing service virtualization.  Michael, I hope you’re ok with me referencing your blog here, given I think this may be of interest to both CEP and EAF advocates & consumers.  So first, the conclusion of Mike’s blog entry and url to it (indented), followed by mine:

http://www.appistry.com/blogs/michael/appistry-eaf-and-service-virtualization

EAF takes advantage of service virtualization to weave together large numbers of computers into a single computational fabric.  This virtualization greatly facilitates the job of the application developer and fabric administrator.  By depending on the fabric, application developers get to focus on the business logic of their application.  By depending on the fabric, administrators know that they have a manageable infrastructure which is able to painlessly scale to their needs. 

I like the concept of Enterprise Application Fabric (EAF) providing service virtualization.  I also like that this virtualization facilitates the job of the application developers in that they focus on the business logic rather than production operational aspects of the application.  This latter point echoes a significant value proposition of Complex Event Processing (CEP) systems, where the CEP platform is an enabling technology for building real-time event analytics applications.  With CEP, the application developers can also just focus on the business rules/logic, given the CEP platform provides all the needed operational properties such as pub/sub, scalability, on-demand queries, retention windows and persistence via storage management, distributed services, recovery, etc.  While there is some overlap in that EAF and CEP both deal with distributed services and recovery, I can see EAF and CEP coming together to provide complex event processing service virtualization, it’s a nice Service-Oriented Architecture (SOA).

While CEP solutions, or perhaps I should just speak for Aleri’s Streaming Platform, provide hot fail-over and distributed services, many applications of CEP have application services beyond what is provided within the CEP Platform.  In those cases, EAF can provide the service management or virtualization inclusive of the CEP platform instances and pub/sub adapters, AS WELL AS application service instances.  For example, at Aleri, we’ve built an application on top of our Streaming Platform for Liquidity Management, which is a cash & collateral position management solution for global bank treasuries to ensure the proper reserves are maintained throughout the business day to handle the daily business transactions & commitments.  Beyond the complex event processing around position management that is done within the Aleri Streaming Platform, our LMS solution provides other services (i.e., application-level processes) for Payment Flow Control and Collateral Management to name just two.  So EAF could enhance a CEP offering by providing virtualization & management of the entire solution.

A wave of blog entries and forum posts about “hype” recently crested over the CEP community.  Some feel that too much has been said about the promise of CEP at a time when there are too few customer success stories.  It’s true that the major vendors have been able to cite only a handful of customers, but — if Aleri’s business is any indication — this has more to do with the privacy needs of customers than with their number.

In the ensuing conversation, it was a suggested that technological evolution would widen the scope of the technology to include more use cases, and thus more customers; that more large enterprises would begin to feel the sort of pain that CEP cure; that standards would make it easier for customers to make informed purchase decisions with less fear of expensive mistakes, and so on.  These are all good points, and each vendor is doubtlessly working feverishly to improve their offering.

It’s hard, then, to see a nascent but growing market that already supports a number of vendors who are hard at work to produce great new features as a ship becalmed in the Sargasso Sea.

It’s great to see the recent announcements in the market from companies such as Endace and TS-Associates et al offering Latency Monitoring and Analysis feeds based on specialist hardware appliances designed to timestamp and analyse trade messages down to the sub milli-second.

Under the MiFID Best Execution rules, price is not the only factor in determining Best Ex. Factors such as speed of execution and likelihood of execution are also important in determining where and how to execute orders in the rapidly fragmenting European equities market.

To date it has been very difficult to deal with the plethora of wire protocols used by the various exchanges and trading venues in the European market. This has made it very difficult to monitor and analyse the various proprietary trade message flows that trading houses have had to use when connecting with these trading venues. And believe me, trying to reverse engineer some of these protocols to try and reconstruct the messages at the software level is both difficult and very latent.

These new hardware based solutions that embed the protocol analysis functions within their hardware logic now make it much easier for downstream Algo Trading and Smart Order Routing to react to the rapidly changing dynamics of the market and better achieve their Best Ex objectives.

With these new “latency” feeds, the ability to create a virtual order book that has been adjusted to reflect the various latencies across the constituent trading venues is now a real possibility. Doing this at wire speed is really awesome.

True “Smart” Order Routing systems are now at the forefront of the new MiFID Algo Trading wars.  These new latency feeds bring the market one step closer to realising this vision.

First, a simple and uncontroversial statement: creating any CEP application involves specifying inputs, outputs and some sort of processing logic for a set of one or more streams — which configuration is known as “the model.” 

Many vendors provide a visual layout mechanism by which to draw the application’s model as a dataflow diagram, then drill down and define the specifics of the model’s subcomponents.  This is undoubtedly a useful feature, but most power users prefer, at the very least as an option, a linguistic modeling paradigm to supplement the “boxes and arrows” approach.  It is here that the trouble starts. 

The two dominant kinds of modeling language in the CEP world are, loosely, “query-based” and “rule-based.”  Query-based languages build on relational algebra and prior art in database query languages, particularly SQL, to provide a familiar means of specifying models. 

Rule-based systems are an outgrowth of logic-programming languages, like PROLOG, and their descendants in the business rules community. 

Some pundits and vendor representatives have made the case that one of the two linguistic approaches renders certain applications impossible.  While this is technically true for a few corner cases, it’s quite rare to find a customer for whom those cases are a concern.  A more honest assessment is that certain classes of problem are quite onerous to model using one or the other approach — on the on side, involving case-by-case hand re-coding of complex pieces of relational algebra to model queries with rules, and on the other Byzantine linguistic contortions to model business logic with SQL. 

The way forward for CEP vendors is to integrate the best features of these two paradigms within a single product.  Here, too, there is prior art: in the late-80s and early-90s, database researchers worked on integrating PROLOG-like rules languages with traditional query languages to improve expressivity; functional programming languages like Haskell and ML developed new pattern-matching techniques; LINQ from Microsoft Research offers an innovative method to specify queries over collections.  Some of these projects presented solutions for problems that few had, but the time has come to dust off these papers and get to work.