Marco’s CEP Blog mentioned that the upcoming 3.0 release of the Aleri Streaming Platform will include enhancements to SPLASH.  SPLASH (an acronym for “Streaming Platform LAnguage SHell”—obviously coined by programming geeks) is our embedded scripting language.  It allows you to write your own event-processing logic without being constrained by stateless relational operations.  I’ve talked about SPLASH, and its wrapper, the FlexStream, in a white paper before.  And now SPLASH will be more powerful and easier to use.

First, we’ve added variables and user-defined functions.  The syntax comes close to C syntax, while preserving some of the features of SQL expression syntax (e.g., equality test is “=” and assignment is “:=” in SPLASH.)  Here, for instance, is the rendering of a recursive factorial function:

int32 factorial(int32 x) {
if (x <= 0) {
return 1;
} else {
return factorial(x-1) * x;
}
}

Functions can be mutually recursive too.  Variables and functions can be defined globally for all streams, or locally for the computational streams: Compute, Filter, Join, Aggregate, Flex, and our new Pattern Stream.

Second, we’ve enhance the language with some familiar data structures (vectors, dictionaries), some less familiar ones (ordered record sets), and some novel ones (event caches, which hold a limited or unbounded collection of insert/update/deletes, organized into buckets by key).  Any decent scripting language provides vectors (or arrays) and dictionaries: AWK, Perl, Python, to name a few.  They’ve stood the test of time.  The other data structures have proved helpful in client use-cases.

It’s important to note that SPLASH programming doesn’t break the inherent programming model of the Aleri Streaming Platform.  The Platform is still a dataflow programming environment: streams respond to and produce events, and events are queued by streams before being processed.  SPLASH simply allows more flexibility in building streams.

Last week Innotap announced their plans to power their new RealDeal product with Complex Event Processing from Aleri. RealDeal  will provide real-time decision support tools for the energy trading business. Innotap represents the latest software vendor to see the power of CEP in enhancing their product, and the value of using off-the-shelf CEP technology from Aleri.

Back at the end of 2005, when we first starting marketing Aleri CEP, we ran into a lot of firms that listened to our story but then said “…but we generally build things in-house.” We would patiently point out that while they may build their own apps in-house, they almost certainly don’t built their own databases, messaging, etc - and our view all along was that it was only a matter of time and education before firms routinely placed CEP in that same category of infrastructure. We’ve come a long way since then, in terms of awareness and understanding. And while Innotap isn’t the first firm to build a commercial application on CEP from Aleri or others, they are still leading edge. Over time, we believe more and more software vendors will turn to commerical complex event processing technology to enhance the capabilities of their products.

Today is the first day of the SIFMA technology show, the big event of the year in the US for trading technology in the capital markets. It’s particularly significant for Aleri, since we announced version 3.0 of Aleri CEP yesterday, and will be previewing it for the public starting today at SIFMA.

We’re really excited by this new release. While we’ve always aimed to deliver a CEP platform that was unsurpassed in terms of breadth of functionality, performance and scalability, and enterprise-class features, those of you who have gotten to know current - or earlier - versions of the product know that it’s not exactly the easiest thing to learn how to use.

With 3.0 we focused our attention on reducing the learning curve for new users, making experienced users more efficient, and making it easier and faster to get up and running quickly. Part of this simply involved improving the look and feel of the Aleri Studio to make it more intuitive, but the even bigger gains were around our approach to connectivity, a new enhanced syntax for pattern matching, and some general simplifications to our language.

Connectivity is probably the biggest difference. 3.0 comes with integrated connectivity. The base product includes a range of built-in connectors for different types of data sources and destinations. Connections can easily and quickly be defined from within the Studio. A data discovery feature allows you to explore a data source and import the data schema directly into your Aleri data model. Field-installable adapters for specialized systems are discovered by the Studio and function the same as the built in connectors.

The new Pattern Matching operator uses a new syntax for expressing complex patterns of events on a single stream or across multiple streams, including things like the time window over which to watch for a pattern, the sequence and relationship of the events that make up the pattern, and even the absence of events as part of the pattern. When a pattern is detected, an output event is generated with the desired information. More info  on the new syntax will be available in the days to come on aleri.com.

And while I’m excited about all the features that make Aleri CEP easier to use, I’d be remiss if I didn’t mention that there are also a range of new features that increase the capabilities, including some significant extensions to Aleri’s SPLASH scripting language. SPLASH is an imperative language that overcomes the limitations of SQL. Standard SQL operators provide simplicity for many common operations such as filtering, joins or aggregation, but there are times when you just need a greater level of control. That’s where SPLASH comes in, with Aleri’s programmable Flex Stream that invokes methods written in SPLASH on the arrival of each new event. In 3.0 SPLASH has become more powerful, with new advanced data structures, user defined functions, some new pre-defined functions and greater control.

More to come in the coming weeks. 3.0 is currently available for previews. A public beta version is expected to be downloadable from aleri.com in early July.

The current memes making the rounds of the CEP blogosphere are hype and platform maturity, the best articles about which so far are the ones by Hans Glide and Opher Etzion. As I’ve just completed a review of several commercial and open source CEP engines, I can say — based on those findings — that the mainstream platforms have collectively reached a level of maturity where real work is getting done, bugs in the field are few, and business value is being added. This is more or less the same situation one finds in other mature industry sectors, for example relational databases or web servers.

There’s more work to be done, of course, especially in terms of user interfaces, authoring/modeling tools, and the adaptation of statistical methods from the data-mining and artificial intelligence communities, just as new techniques are still being developed to improve relational databases. In fact, as all seasoned software engineers know, the only technologies for which there is no more research to be done are those that have been abandoned to the dustbin of history.

The good news — right now, today — is that customers have a wide variety of high-quality CEP platforms from which to choose. This situation can only lead to competition among the major vendors to introduce new capabilities and improve usability, which is exactly the situation that will serve customers best. So, though we’re well served to discuss the future of CEP, damning the state of the art because research opportunities remain is, well, probably a little immature.

Sun’s Project Fuji is a big Java-based ESB (Enterprise Service Bus) platform designed to assist developers in creating applications that deal with messages from multiple heterogeneous sources. ESB systems share a great deal of functionality with CEP systems, which can be clients/servers of ESB-based architectures. As a consequence, they share many of the same challenges with regards to authoring, and Project Fuji, like many CEP platforms, offers both a language (IFL, the Integration Flow Language) and a visual authoring system.

One of the interesting things about Project Fuji’s visual authoring environment is that it’s an entirely browser-based AJAX front-end that uses CSS layers and canvases to provide the niceties of a desktop application. To get a sense of how this works, have a look at this screencast of their language-based IDE (which runs in NetBeans), and then check out this screencast of the prototype of their web front-end. Slick stuff, that.

Project Fuji is open source, and Andi Egloff tells me in his blog comments that they’re considering releasing their UI toolkit as a separate OS project sometime in the near future.

This week marks a milestone in the evolution of CEP with the formal launch of the Event Processing Technical Society. While this group has been meeting and working together informally for the last couple of years, the process of formalizing the group began last September. Congratulations to Dr. Opher Etzion of IBM who was the primary catalyst behind the formation of the group and served as both the informal group, and now the formal group’s chair. I know that a lot of effort went into establishing this as a formal industry body.

There has been a lot of discussion lately in the blogosphere about the maturity of CEP as an industry, as a technology, as an architecture, and as a field of research. In fact some of the debate has been about which of these things it is (a concept? an industry?). From my perspective it’s clearly a technology and a field of research. Whether it’s an architecture or an element of Event Driven Architecture is a debate I’ll leave to the architects. And clearly it’s rapidly evolving into a distinct segment within the IT industry as a tool for building applications that analyze events, in the context of other events, in real-time.

I think the formation of EPTS as a formal industry group points to the evolving maturity of this field. As a founding member of EPTS, Aleri believes that the various firms and individuals that have an interest in CEP can, through some level of cooperation, work to remove some of the barriers to adoption, build common understanding, and collectively contribute to the evolution of the field. This doesn’t mean that we foresee the commoditization of CEP - individual CEP products will continue to have their own unique features and specializations.

A common language will go a long way toward common understanding - and thus the early efforts on the Event Processing Glossary. I’m looking forward to seeing less time spent on debating what CEP is and more focus on where it is delivering benefit. And we at Aleri look forward to working with the other members of EPTS to not only build a common understanding, but also to shepherd this new technology through what will no doubt be a rapid evolution.

In the past several weeks I have had a number of conversations with various people involved in some form or fashion with algorithmic trading. The idea that computers are making the trader obsolete is a bit of a myth. In fact, it seems like the more sophisticated the trader, and the more heavily reliant they are on the use of automation in their trading strategies, the more likley they are to recognize the human value. The movement to electronic trading accross multiple asset classes and multiple geographies present an increasingly large opportunity for automation to play a key role in trading strategies. The human trader in unlikely to absorb, process, and act on the vast amount of data, including the relationships among that data, in order to effectively take advantage of the opportunity. And yet, the power in automation is both a blessing and a curse, and the best traders seem to understand this more than anyone. As a result, they look for an ability to interact with the system in order to monitor and adjust the strategies on the fly.

The power of automation can provide for significant and rapid trading profits. But it can quickly rack up losses as well. Complex Event Processing is very powerful technology as a platform for enabling these strategies, but can be even more powerful if constructed in a fashion that allow for the algorithms to be dynamicaly tuned based on the traders sense of the market. While the importance of automation is clearly increasing, it is unlikely to become an all or nothing proposition, and therefore, the ability to recognize and accommodate the relationship between traders and automation is an essential element of employing this technology.

I will be Aleri’s representative at DEBS 2008. It looks to be a great conference, and I’m especially excited about ESTEEM 08 — the First International Workshop on Emergent Semantics and cooperaTion in opEn systEMs — which will be co-located with DEBS.

Let me know in the comments if any of you would like to get together to discuss CEP, Semantic Web technology, or the intersection of the two — preferably over a fine Roman feast.