Interesting article over at Information Week about the spread of CEP beyond the capital markets. That’s consistent with what we’re seeing at Aleri. In fact just this week we announced a new customer - banks.com - a financial services portal on the web. Banks.com is using Aleri CEP for real-time insight into traffic patterns on their site, allowing them to fine tune their marketing and advertising programs in response to emerging trends.

While it’s certainly true that Aleri established it’s CEP technology in the capital markets, we always expected to branch into other markets. There are many industries that can benefit from continuous insight and immediate reponse to changing conditions and emerging opportunities and threats. Banks.com is an ecommerce firm that sees business benefit from faster response to emerging trends. Other industries where we are seeing opportunities include: telecoms, network security, energy management, and logistics.

Hans Glide wrote a piece entitled EP is real-time data mining? in April, 2008, and the Register has been wondering the same thing since at least June, 2006.  My opinion, expressed in a comment on Mr Glide’s blog, is that one of CEP’s most fertile sources for new algorithms and techniques is the existing body of work on data mining.

With that in mind, allow me to direct your attention to the Orange project from the A.I laboratory at the University of Ljubljana, Slovenia.  Orange provides a visual (”boxes and arrows”) programming environment similar to the visual authoring paradigm available in the products of Aleri, Streambase, and others, but with boxes that implement various A.I.-inspired data mining and visualization functions (Classification trees, K-Means clustering, Nomograms, and so on). In addition to a data path/routing-based workflow, scripting functionality is offered via embedded Python. Check out this short white paper on the current set of widgets.

I would be very surprised if we didn’t see some of this sort of technology appearing in the leading CEP platforms over the next year or two.

Now that STAC has provided a venue for verifiable third party benchmarks, we really have a chance to see what is behind CEP vendor claims. While I can hardly consider myself to be objective, I do believe that the strength of the Aleri architecture will be, or at least can be verified by such valid comparison tests. I further expect that some use cases will show the various CEP solutions closer to parity, while others - the tougher ones which will benefit from a more robust architecture, which natively handles updates and deletes like Aleri does - should begin to show a clear advantage for Aleri. It is entirely reasonable to test a range of use cases such as market data absorption, algo trading, market making, order book consolidation, risk aggregation and analysis, and others including those beyond financial services, as that is truly representative of the real world. I could be wrong about how Aleri would fare in such comparisons, but we are happy to subject ourselves to that test.  The question I have now, is where are the other vendors? STAC has been around for a while now. As of today, we have run two tests, and not a single other CEP test has been run. I am happy for Aleri to run any test in STAC provided by another vendor, assuming they would also be willing to run the test which we originally ran a few weeks back, which was also re-run by Intel to show results for their 6-core “Dunnington” processors as compared to the original benchmark that was run on the Tigerton processor. So the benchmark is there. Perhaps I am making too much of the need for vendors to back their claims. I am interested in your thoughts. Please let me know what you think about the opportunity for vendors to stand behind their claims by using STAC, and your thoughts regarding vendors willingness or lack thereof to step up to such transparency. Either respond to this blog, or contact me directly at don.deloach@aleri.com. Thanks! 

This week STAC (the Securities Technology Analysis Center) published a report measuring the performance of the Aleri CEP engine running an order book aggregation model on a Sun Server (running Solaris 10)  with the Intel Caneland chip set.  While this may only represent a single data point (or set of data points actually) in characterising the performance of an event processor, it’s the first time that anyone has published objective, verifiable performance numbers. In a field with a lot of talk about performance - both throughput and latency - STAC is providing a useful service by cutting through the marketing hype put out by various vendors (yes, including ourselves) to provide a level of transparency. Longer term, we expect that there will be industry-standard benchmarks that can serve as a tool to compare products. Aleri is a member of the STAC Benchmark Council and an active participant in the working group that is in the process of working toward the definition and agreement of such standards.In the mean time, the report published this week provides a level of transparency that has previously been missing.

Now this report doesn’t provide comparative data across products. And it only measures performance in a certain context - in this case order book aggregation. In fact, we didn’t choose to do the testing with a model that would show eye-popping throughput numbers, but rather chose to measure a use-case we are seeing in the field, where the numbers would be useful to people. In this case it shows throughput more than sufficient to consolidate the equity market with the highest traffic - the U.S. market - while maintaining a low latency profile. Other models (i.e. different data types and business logic) will display very different performance characteristics - some faster, some slower.

As my last blog post talked about, different applications have very different performance requirements. Some need very high throughput, some need very low latency, some need both, some don’t need either. I would characterize this use case as being very latency sensitive, demanding lowest possible latency, while having a mid to high throughput requirement (up to 150,000 messages per second, compared to some uses cases that are only dealing with 10’s or 100’s per second, and others that climb up to or even beyond a million per second - though the latter are relatively rare).

Bottom line, we were pleased to have the opportunity to work with STAC, Sun and Intel to get this report published and hope that people find it useful.

Just read an interesting article over at Intelligent Enterprise, titled: “Report Warns Of Data Warehouse ‘Bottleneck’ In Real-Time Analytics“. It certainly reflects what we have been seeing, where people are looking to Event Processing technology (whether you call it Complex Event Processing or Event Stream Processing) to overcome the limitations of more traditional data collection, analysis and reporting technologies - such as operational data stores and data warehouses - when it comes to real-time analysis and monitoring.

A couple of observations:

“Real-time” means different things to different people.  While the CS purists are quick to point out that real-time really just refers to predictably response time, in general usage people use it to mean “right now” or “as soon as something happens”. But as the article points out, for some users, a few seconds or even a few minutes can be “right now”. For someone who goes from an overnight report to data that’s a few minutes old, it certainly feels light “right now”. But for other users, and other applications, “right now” can mean milliseconds. Placing a trade in the markets before the opportunity is lost, requires sub-second response. Shutting down that nuclear reactor that just sprung a leak…I hope they don’t consider a few minutes to be “right now”. 

This is where event processing can help. Data can be streamed into an event processor - as soon and as fast as it appears, and the event processor can apply complex rules and operations to combine it with other data, filter, summarize, compute new values, look for patterns, etc - producing a stream of new “events” as a result. The lag time between the input and the output - for a high performance complex event processing (CEP) engine - will be sub-second (or even sub-millisecond in some cases).

One thing I think the article didn’t mention is the case that we see most often: using an event processor to complement a data warehouse. The article talks about how event stream processing is an alternative to a data warehouse. But frankly I have yet to see a customer use Aleri’s CEP engine as an alternative to a data warehouse. What I do see, however, is CEP being deployed to complement a data warehouse.  The data warehouse still serves as the ultimate data repository, and the event processor takes on one (or both) of two functions:

1) as a real-time monitoring application, the event processor sits along side the data warehouse, monitoring and analyzing the same data that is flowing into the data warehouse. But while the data warehouse holds it to make it available for querying, the event processor analyzes it as soon as it arrives in order to generate alerts, initiate an immediate response, or to make low-latency information available to a user to support real-time decision making.

2) as a front-end pre-processor for data flowing into the data warehouse. This is the case that I think gets overlooked a lot. You can almost think of this as “STL” in the sense that it’s an alternative to ETL that is event-driven. Here, data streams into the event processor in real-time. The event processor performs real-time data correlation, transformation, etc and collects the data to be loaded into the data warehouse in micro-batches. This micro-batch loading overcomes the speed limitations with which the data warehouse can absorb data, and at the same time avoids large batches that introduce unnecessary latency.  Because of the event processor’s ability to combine data from different sources and transform the data in real-time, the data gets loaded into the data warehouse in a format that is optimal for reporting. This can significantly improve the speed of applications that use the data in the warehouse, since the data is already in the form they need, rather than being stored in the warehouse in a raw form, and then being combined and transformed at query time.

Bottom line, at Aleri we see event processing (or CEP) as a powerful ally of the data warehouse rather than as an alternative to the data warehouse.

As a former Sybase employee myself, I was pleased for Aleri to receive an innovation award from Sybase for our work integrating Sybase RAP and Aleri, and specifically for our joint efforts at Mitsubishi. I have always considered Sybase to be a true leader in database technology, and their recent efforts with Sybase RAP - The Trading Edition, are no exception. Furthermore, this technology is a great complement to Aleri’s CEP offering, the Aleri Streaming Platform, as well as for use cases like Aleri Market Liquidity Analysis which we will be demonstrating in our booth at Techwave this week. I see this award as a leading indicator of more good things to come between Aleri and Sybase, and certainly appreciate Sybase for the recognition.

Today is the first day of the SIFMA technology show, the big event of the year in the US for trading technology in the capital markets. It’s particularly significant for Aleri, since we announced version 3.0 of Aleri CEP yesterday, and will be previewing it for the public starting today at SIFMA.

We’re really excited by this new release. While we’ve always aimed to deliver a CEP platform that was unsurpassed in terms of breadth of functionality, performance and scalability, and enterprise-class features, those of you who have gotten to know current - or earlier - versions of the product know that it’s not exactly the easiest thing to learn how to use.

With 3.0 we focused our attention on reducing the learning curve for new users, making experienced users more efficient, and making it easier and faster to get up and running quickly. Part of this simply involved improving the look and feel of the Aleri Studio to make it more intuitive, but the even bigger gains were around our approach to connectivity, a new enhanced syntax for pattern matching, and some general simplifications to our language.

Connectivity is probably the biggest difference. 3.0 comes with integrated connectivity. The base product includes a range of built-in connectors for different types of data sources and destinations. Connections can easily and quickly be defined from within the Studio. A data discovery feature allows you to explore a data source and import the data schema directly into your Aleri data model. Field-installable adapters for specialized systems are discovered by the Studio and function the same as the built in connectors.

The new Pattern Matching operator uses a new syntax for expressing complex patterns of events on a single stream or across multiple streams, including things like the time window over which to watch for a pattern, the sequence and relationship of the events that make up the pattern, and even the absence of events as part of the pattern. When a pattern is detected, an output event is generated with the desired information. More info  on the new syntax will be available in the days to come on aleri.com.

And while I’m excited about all the features that make Aleri CEP easier to use, I’d be remiss if I didn’t mention that there are also a range of new features that increase the capabilities, including some significant extensions to Aleri’s SPLASH scripting language. SPLASH is an imperative language that overcomes the limitations of SQL. Standard SQL operators provide simplicity for many common operations such as filtering, joins or aggregation, but there are times when you just need a greater level of control. That’s where SPLASH comes in, with Aleri’s programmable Flex Stream that invokes methods written in SPLASH on the arrival of each new event. In 3.0 SPLASH has become more powerful, with new advanced data structures, user defined functions, some new pre-defined functions and greater control.

More to come in the coming weeks. 3.0 is currently available for previews. A public beta version is expected to be downloadable from aleri.com in early July.

Just want to draw  your attention to the STAC announcement today, that Aleri has joined the STAC Benchmark Council and has placed Aleri’s CEP technology - the Aleri Streaming Platform - in the STAC lab. While Aleri joining the benchmark council isn’t new news per se (yes, we were there from the start), having CEP technology resident in the STAC lab is a milestone. Now that Aleri is “Racked and STAC’d” as STAC calls it, it’s available for evaluation by firms that want a fast track way to doing an evaluation without having to set up their own test bed. It will also be available to support the work of the STAC benchmark council.  For anyone not familiar with the work of STAC and the services they offer, if you’re in the financial industry I suggest you check it out.

Pulsed subscriptions, which deliver an optimal coalesced block of updates to a subscriber at a fixed, periodic interval, has recently been added to the Aleri Streaming Platform. This subscription method rounds out a number of other subscription types that the Aleri Platform already supports: normal low latency lossless subscription, SQL projection/selection during subscription and both lossy and
dropable subscription models. A short description of each subscription type can be found in a short paper I put together on subscription methods.

I would enjoy a discussion on the subscription modes outlined in the paper, and other extensions to subscription types that would bring value when connecting to a Complex Event Processing engine.

Before we go into the the discussions of features for building the processing loops, let’s make up a simple example. It would make the explanations easier. For this let’s try to design a primitive trading system. For most of the readers this design is absolutely obvious but I still want to go through it to have a concrete example to refer to. Bear with me and eventually we’ll get to the interesting stuff. I’ll be using ASCII art diagrams, since they’re easier to draw, and since I haven’t figured out yet how to upload the real pictures to this blog engine.

Let’s start the design. Our system would receive the bid (buy) and ask (sell) offers from the participants. So we make one stream receiving bids and one receiving asks:

(incoming bids)
(incoming asks)

Then we put some logic that selects the best bid and the best ask. This can be implemented internally as multiple streams, but for simplicity let’s consider them as single black-boxes:

(incoming bids)->(best bid)
(incoming asks)->(best ask)

Finally, we add the logic that takes the best bid and best ask and checks if they match. If they do, it’s considered a trade and sent as the output of the model.

(incoming bids)->(best bid)->|
                             |->(match trade)->
(incoming asks)->(best ask)->|

Neat and easy. But by now you can see the drawback: it produces only one trade and that’s it. In this sense it’s like the digital logic that takes a set of signals on the input and produces a set of signals on the output. As long as the inputs fluctuate, the outputs would fluctuate too (with a delay) but once the inputs settle down, the outputs stay fixed. Both effects are not what we need from a real trading system. We don’t want the done trades to fluctuate. We also want to remove the offers that have already resulted in a trade from the incoming offers. So we need to add a loop:

+...........................................................+
.                                                           .
+>(completed trades)---+--+                                 .
                       |  |                                 .
                       V  |                                 .
  (incoming bids)->(incomplete bids)->(best bid)->|         .
                          |                       |->(trade)->
                          V                       |
  (incoming asks)->(incomplete asks)->(best ask)->|

When a trade is produced, it loops back (shown as a dotted line) to the stream of completed trades. Those are used to filter out the offers that have been already completed to a trade (the arrow from “completed trades” is going under “incomplete bids” to reach the “incomplete asks” too). Now only the incomplete offers participate in the best offer calculation, and the next trade can be matched up.

This diagram is still missing something. From reading the yesterday’s discussion, can you guess, what? The answer is coming in the next installment…