Most CEP platforms were designed under the assumption that they would run primarily as services.  The idea was that someone would author an application — perhaps with a desktop dashboard — that would reside on a server and run more or less in the manner of a traditional database application.  This is a fine approach to many problems, but I’d like to call attention to some trends I’ve noticed on the web over the last couple of years that point to another mode of use for this technology.

Yahoo released Yahoo Pipes some time ago, and it hasn’t escaped the community’s notice that it’s quite similar in concept to the boxes-and-arrows authoring mode available in Aleri, Streambase, &c.  In my opinion, it sets a decent standard for a service-oriented RSS-based CEP platform that allows arbitrary users to create any number of applications on the fly, rather than depending on someone else to build the applicaion for them.  The data sources available to this model continue to grow, including Google’s recent announcement that they’ve added RSS feed support to their customizable news alert service.  This is one vision of continuous queries implemented in the cloud, and we should certainly watch this space.

A more recent trend, and one that I think will be important moving forward, is the rise of end-user visualization tools designed to help make sense of all the data the web has made available: the New York Times has created a “visualization lab” to allow readers to mine public data, Swivel and the Many Eyes project from IBM provide similar services, iCharts says they want to be the “YouTube of interactive charts,” a goal that seems to be shared by YouCalc, Trendrr, TracknGraph, and a host of others.

Some of these newcomers, like Widgenie and Google’s new Visualization gadget builder, produce little Javascript applets that can be anchored to one’s desktop in Mac OS X and late-model Windows.

Looking at this explosion of visualization tools converging with Internet-based streaming  data sources, I see the future of at least one kind of desktop CEP, especially if the data-mining and visualization tools grow more powerful, along the lines what’s promised by Orange.

Not surprisingly, we’ve been seeing an uptick in interest around technology for risk management of late. In the past, selling technology to improve a firm’s ability to monitor and manage risk was a challenge. Risk management is (was?) viewed as a cost, rather than revenue enhancement. And projects for revenue enhancement find an easier path to funding. Looking ahead, that’s not likely to be the case. In this article by Larry Tabb in Wall Street & Technology, he predicts that “…risk management will be the key industry focus for the next three or four years. While firms have invested significantly in their risk infrastructures over the past 10 years, we will see some significant investment and modifications not only in the way that firms develop their risk infrastructures, but also in the way they manage risk.” While this article was published back in June, it’s more relevant than ever (and I bet Larry’s feeling confident about his predictions right now).

Now when it comes to risk management, traditionally the focus has not been on real-time and CEP has certainly not played a major role. But we have been working with firms on real-time risk monitoring solutions for several years now and we expect interest to grow. In addition to Tabb, another WS&T article from this summer, this one by Julio Gomez, also pointed to the fact that leading firms will be “pushing risk management systems to real-time”.

This is where CEP can help, by providing a real-time platform to consolidate, normalize and aggregate data across heterogeneous systems, distributing summary data to the systems that need it, and powering dashboards and alerting systems for those that need to monitor current exposures. A recent report by the Aite Group also touches on this aspect of CEP, looking at “how capital markets firms apply distributed caches, complex event processing engines, reference data solutions, grid computing, and messaging infrastructures to distributed data architectures.” (the full report can be downloaded for free here – registration required).

One key aspect to all this – something that I’ve touched on before – is thinking about the potential for CEP in applications that:

1. Don’t require millisecond-level latency, and
2. Aren’t focused on pattern detection. So much of the focus on CEP seems to be on those two things, that the versatility of some CEP technologies, such as Aleri’s CEP platform, can get overlooked. In this context we’re still talking about low latency, it’s just that “low” is relative. Compared to overnight, hourly data represents lower information latency. The fact that CEP can deliver it continuously is just a bonus.

For more on our predictions on trends in 2009, including the focus on risk, check out this article on aleri.com.

Below is a link to a new v-log highlighting Aleri’s Liquidity Management Solution, our relationship with WallStreet Systems, and our recent announcement regarding the new module for LMS called LORAM, (Liquidity Operational Risk Analysis Module) which set for commercial availability in late Q1 or early Q2 of 2009.  This new module will add liquidity stress testing to Aleri’s Liquidity Management Solution’s ability to manage, monitor, and optimize liquidity risk across the bank.

Interesting article over at Information Week about the spread of CEP beyond the capital markets. That’s consistent with what we’re seeing at Aleri. In fact just this week we announced a new customer - banks.com - a financial services portal on the web. Banks.com is using Aleri CEP for real-time insight into traffic patterns on their site, allowing them to fine tune their marketing and advertising programs in response to emerging trends.

While it’s certainly true that Aleri established it’s CEP technology in the capital markets, we always expected to branch into other markets. There are many industries that can benefit from continuous insight and immediate reponse to changing conditions and emerging opportunities and threats. Banks.com is an ecommerce firm that sees business benefit from faster response to emerging trends. Other industries where we are seeing opportunities include: telecoms, network security, energy management, and logistics.

The video below is a short conversation with Neil McGovern of Sybase talking about the unique partnership Aleri and Sybase have forged to create more sophisticated, smarter trading algorithms that can react instantaneously to market conditions as well as accessing massive amounts of historical data to leverage in the decision making process.  Click here to learn more about their partnership

Now that STAC has provided a venue for verifiable third party benchmarks, we really have a chance to see what is behind CEP vendor claims. While I can hardly consider myself to be objective, I do believe that the strength of the Aleri architecture will be, or at least can be verified by such valid comparison tests. I further expect that some use cases will show the various CEP solutions closer to parity, while others - the tougher ones which will benefit from a more robust architecture, which natively handles updates and deletes like Aleri does - should begin to show a clear advantage for Aleri. It is entirely reasonable to test a range of use cases such as market data absorption, algo trading, market making, order book consolidation, risk aggregation and analysis, and others including those beyond financial services, as that is truly representative of the real world. I could be wrong about how Aleri would fare in such comparisons, but we are happy to subject ourselves to that test.  The question I have now, is where are the other vendors? STAC has been around for a while now. As of today, we have run two tests, and not a single other CEP test has been run. I am happy for Aleri to run any test in STAC provided by another vendor, assuming they would also be willing to run the test which we originally ran a few weeks back, which was also re-run by Intel to show results for their 6-core “Dunnington” processors as compared to the original benchmark that was run on the Tigerton processor. So the benchmark is there. Perhaps I am making too much of the need for vendors to back their claims. I am interested in your thoughts. Please let me know what you think about the opportunity for vendors to stand behind their claims by using STAC, and your thoughts regarding vendors willingness or lack thereof to step up to such transparency. Either respond to this blog, or contact me directly at don.deloach@aleri.com. Thanks! 

Just ran across this article in Advanced Trading:

Exclusive Interview With Nick Leeson: An Inside Look at Rogue Trading by Advanced Trading

and noticed the argument he makes for real-time risk aggregation. OK, so maybe you don’t normally want to be quoting a convicted criminal to bolster your argument, but in this case it has merit. We have long showcased “real-time risk aggregation” as a solution that can be implemented using Aleri CEP. The idea is that firms have data in any number of different systems that don’t talk to each other. While each system produces position and risk information that can give you one picture in isolation, the big picture - when you combine the data across all the systems - can look quite different. Unfortunately many firms do not have the ability to easily combine the data across these disparate systems, and if they do it, happens on a daily (or even less frequent) basis. There’s clear evidence that this isn’t good enough, as Leeson asserts:

“At the end of 1994, “I probably had 500 million pounds from the bank to fund the illegal positions that I had, and the capital base of the bank was only 250 million pounds,” Leeson adds. “And the legal limit that you can lend to a subsidiary is only 20 percent of the bank’s capital, so it was 10 times in breach of that limit.”In SocGen’s case, if Kerviel was able to build up an exchange-traded position that exceeded the capital base of the bank by 13 times, that would suggest some very simple risk management functions were not being performed, Leeson continues. “They may have been checking some complex risk management algorithms but ignoring the most simple things,” he suggests.

Leeson also points to disparate systems as an opening for his rogue trades, something that he says more-unified risk management could help avoid. “At Barings, different systems were looking after plain vanilla equities and something else was looking after futures and options and something else for the warrant products,” he relates. “Only when those reports were actually brought together could you actually see what exposure the bank had and could you compare things. It enabled me to circumvent the accounting and risk management processes.”

Aleri CEP can be used to consolidate and aggregate data from disparate systems in real-time. Based on an event-driven architecture, and with full data normalization capabilities, Aleri can be deployed as an overlay to existings systems. It doesn’t require a massive overhaul of current infrastructure to get everything onto a common system with a common data model. The result is the ability to see the big picture, in real-time, and set up monitors that will generate alerts when limits are reached or exposures pass a critical point.  To learn more about an Aleri based risk managment system, please join our webcast produced by Wall Street & Technology on September 17 at 9:00am PT / 12:00pm ET.

There was a very good post over on the Apama blog, commenting on the recent announcement by Streambase and Oracle that will work together towards the development of a “standard” for applying SQL to event streams.  I think Louis at Apama sums it up well.  As he correctly points out, we added an imperative scripting language (that we call SPLASH) in our 2.0 release  and have continued to enhance it, with significant enhancements in our recent 3.0 release.  We continue to offer SQL-based authoring as a convenience, for those that like the familiarity of SQL. But we find that for many use cases, it’s simply too difficult, if not impossible, to express the logic in SQL - thus the need for SPLASH.

I do think it would be useful to have a standard form of SQL for event streams - for those cases where SQL is a good fit.  To that end, I would welcome an open and inclusive effort to standardize where there is commonality.

This week STAC (the Securities Technology Analysis Center) published a report measuring the performance of the Aleri CEP engine running an order book aggregation model on a Sun Server (running Solaris 10)  with the Intel Caneland chip set.  While this may only represent a single data point (or set of data points actually) in characterising the performance of an event processor, it’s the first time that anyone has published objective, verifiable performance numbers. In a field with a lot of talk about performance - both throughput and latency - STAC is providing a useful service by cutting through the marketing hype put out by various vendors (yes, including ourselves) to provide a level of transparency. Longer term, we expect that there will be industry-standard benchmarks that can serve as a tool to compare products. Aleri is a member of the STAC Benchmark Council and an active participant in the working group that is in the process of working toward the definition and agreement of such standards.In the mean time, the report published this week provides a level of transparency that has previously been missing.

Now this report doesn’t provide comparative data across products. And it only measures performance in a certain context - in this case order book aggregation. In fact, we didn’t choose to do the testing with a model that would show eye-popping throughput numbers, but rather chose to measure a use-case we are seeing in the field, where the numbers would be useful to people. In this case it shows throughput more than sufficient to consolidate the equity market with the highest traffic - the U.S. market - while maintaining a low latency profile. Other models (i.e. different data types and business logic) will display very different performance characteristics - some faster, some slower.

As my last blog post talked about, different applications have very different performance requirements. Some need very high throughput, some need very low latency, some need both, some don’t need either. I would characterize this use case as being very latency sensitive, demanding lowest possible latency, while having a mid to high throughput requirement (up to 150,000 messages per second, compared to some uses cases that are only dealing with 10’s or 100’s per second, and others that climb up to or even beyond a million per second - though the latter are relatively rare).

Bottom line, we were pleased to have the opportunity to work with STAC, Sun and Intel to get this report published and hope that people find it useful.

I recently suggested that David Luckham add latency as a measure of “power” in his strawman for features that CEP users are interested in. This generated some discussion, and the question as to whether I was asserting that all CEP applications require low latency, so I thought I’d share some observations.

When people look at the performance, in terms of speed and capacity, of a CEP platform, they are generally concerned with one or both of two dimensions: throughput and latency. Throughput is typically measured in messages per second, i.e.  how many incoming messages (events) per second can be processed without getting overloaded. Latency is the elapsed time between an event and a response. You could also call it the response time or reaction time.

Different applications have very different needs. Most care about throughput at some level, because they need to ensure that their application can keep up with the expected message rate. This can vary widely. One application might expect a few events a day, another might expect a few hundred thousand a second (the most extreme I’ve run across was a firm that was designing a system for a capacity of 15 million messages per second! yikes). So you care about throughput in so far as you need to have enough for your intended application. Beyond that, excess capacity is irrelevant other than to give you head room for growth.

The same is true for latency in that most applications will care about latency at some level, but expectations may vary widely. We see trading applications that are latency-sensitive to the level where every millisecond counts. We also see many BAM-type applications where the results are being displayed on a dashboard for consumption by a human user. In this case, the requirements might be for latency in the range of 150ms to a few seconds. Demanding latency of less than 150 ms in this context is pointless, since that’s below the range of human perception. In still other cases, a reaction time within the hour might be perfectly acceptable to the application - and yet it still might be an application that can benefit from CEP. At some level, however, you probably don’t need CEP. If you truly don’t care about latency - if it’s ok that your reponse comes hours, days or weeks after the fact, then you can probably accomplish what you want through conventional data analysis tools that run periodicall against historical event data. Sure, you could still use CEP, but if you aren’t trying to analyze  events in real-time, as they occur, there are other tools available. And the only reason to go to the trouble to analyze them as soon as they occur, is because you care about latency, even if your idea of acceptable latency differs from someone else by seconds or minutes.

Bottom line, I think the fact that some CEP technology can be used for those very demanding high performance applications that require very high throughput or very low latency sometimes distorts the discussion. The big numbers (or extremely little numbers in the case of ultra-low latency) get peoples attention, but they shouldn’t cause people to think that they only need CEP if they are building a high speed application. If there is a need to analyze and act on event data in real-time, then CEP can help, regardless of how fast the events arrive or how quickly you need to respond.

Finally, lest I create the wrong impression, performance is merely ONE feature of CEP that users care about. It is by no means the only one. In fact a lot of the time I think too much attenion gets focused on performance at the expense of other critical factors - the most fundamental one being: can it address the use case in question - at any speed? But that’s for another day.