Most CEP platforms were designed under the assumption that they would run primarily as services.  The idea was that someone would author an application — perhaps with a desktop dashboard — that would reside on a server and run more or less in the manner of a traditional database application.  This is a fine approach to many problems, but I’d like to call attention to some trends I’ve noticed on the web over the last couple of years that point to another mode of use for this technology.

Yahoo released Yahoo Pipes some time ago, and it hasn’t escaped the community’s notice that it’s quite similar in concept to the boxes-and-arrows authoring mode available in Aleri, Streambase, &c.  In my opinion, it sets a decent standard for a service-oriented RSS-based CEP platform that allows arbitrary users to create any number of applications on the fly, rather than depending on someone else to build the applicaion for them.  The data sources available to this model continue to grow, including Google’s recent announcement that they’ve added RSS feed support to their customizable news alert service.  This is one vision of continuous queries implemented in the cloud, and we should certainly watch this space.

A more recent trend, and one that I think will be important moving forward, is the rise of end-user visualization tools designed to help make sense of all the data the web has made available: the New York Times has created a “visualization lab” to allow readers to mine public data, Swivel and the Many Eyes project from IBM provide similar services, iCharts says they want to be the “YouTube of interactive charts,” a goal that seems to be shared by YouCalc, Trendrr, TracknGraph, and a host of others.

Some of these newcomers, like Widgenie and Google’s new Visualization gadget builder, produce little Javascript applets that can be anchored to one’s desktop in Mac OS X and late-model Windows.

Looking at this explosion of visualization tools converging with Internet-based streaming  data sources, I see the future of at least one kind of desktop CEP, especially if the data-mining and visualization tools grow more powerful, along the lines what’s promised by Orange.

Not surprisingly, we’ve been seeing an uptick in interest around technology for risk management of late. In the past, selling technology to improve a firm’s ability to monitor and manage risk was a challenge. Risk management is (was?) viewed as a cost, rather than revenue enhancement. And projects for revenue enhancement find an easier path to funding. Looking ahead, that’s not likely to be the case. In this article by Larry Tabb in Wall Street & Technology, he predicts that “…risk management will be the key industry focus for the next three or four years. While firms have invested significantly in their risk infrastructures over the past 10 years, we will see some significant investment and modifications not only in the way that firms develop their risk infrastructures, but also in the way they manage risk.” While this article was published back in June, it’s more relevant than ever (and I bet Larry’s feeling confident about his predictions right now).

Now when it comes to risk management, traditionally the focus has not been on real-time and CEP has certainly not played a major role. But we have been working with firms on real-time risk monitoring solutions for several years now and we expect interest to grow. In addition to Tabb, another WS&T article from this summer, this one by Julio Gomez, also pointed to the fact that leading firms will be “pushing risk management systems to real-time”.

This is where CEP can help, by providing a real-time platform to consolidate, normalize and aggregate data across heterogeneous systems, distributing summary data to the systems that need it, and powering dashboards and alerting systems for those that need to monitor current exposures. A recent report by the Aite Group also touches on this aspect of CEP, looking at “how capital markets firms apply distributed caches, complex event processing engines, reference data solutions, grid computing, and messaging infrastructures to distributed data architectures.” (the full report can be downloaded for free here – registration required).

One key aspect to all this – something that I’ve touched on before – is thinking about the potential for CEP in applications that:

1. Don’t require millisecond-level latency, and
2. Aren’t focused on pattern detection. So much of the focus on CEP seems to be on those two things, that the versatility of some CEP technologies, such as Aleri’s CEP platform, can get overlooked. In this context we’re still talking about low latency, it’s just that “low” is relative. Compared to overnight, hourly data represents lower information latency. The fact that CEP can deliver it continuously is just a bonus.

For more on our predictions on trends in 2009, including the focus on risk, check out this article on aleri.com.

Below is a link to a new v-log highlighting Aleri’s Liquidity Management Solution, our relationship with WallStreet Systems, and our recent announcement regarding the new module for LMS called LORAM, (Liquidity Operational Risk Analysis Module) which set for commercial availability in late Q1 or early Q2 of 2009.  This new module will add liquidity stress testing to Aleri’s Liquidity Management Solution’s ability to manage, monitor, and optimize liquidity risk across the bank.

Interesting article over at Information Week about the spread of CEP beyond the capital markets. That’s consistent with what we’re seeing at Aleri. In fact just this week we announced a new customer - banks.com - a financial services portal on the web. Banks.com is using Aleri CEP for real-time insight into traffic patterns on their site, allowing them to fine tune their marketing and advertising programs in response to emerging trends.

While it’s certainly true that Aleri established it’s CEP technology in the capital markets, we always expected to branch into other markets. There are many industries that can benefit from continuous insight and immediate reponse to changing conditions and emerging opportunities and threats. Banks.com is an ecommerce firm that sees business benefit from faster response to emerging trends. Other industries where we are seeing opportunities include: telecoms, network security, energy management, and logistics.

The video below is a short conversation with Neil McGovern of Sybase talking about the unique partnership Aleri and Sybase have forged to create more sophisticated, smarter trading algorithms that can react instantaneously to market conditions as well as accessing massive amounts of historical data to leverage in the decision making process.  Click here to learn more about their partnership

Hans Glide makes a good point about CEP platforms acting as multicore-optimized application construction kits.  The high-performance “boxes and arrows” systems, like Aleri and Streambase, provide a ready-made environment in which to build applications that leverage multiple cores by mixing a dataflow architecture with thread pools.

As with most of these technologies, there’s a great deal of prior art from other research areas.  One paper I recommend to anyone thinking about parallel execution, dataflow architectures and programming language design is Robert Iannucci’s Toward a Dataflow / Von Neumann Hybrid Architecture, which concerns itself with hardware architectures, but makes several observations that are relevant to modern software architectures.

Hans Glide wrote a piece entitled EP is real-time data mining? in April, 2008, and the Register has been wondering the same thing since at least June, 2006.  My opinion, expressed in a comment on Mr Glide’s blog, is that one of CEP’s most fertile sources for new algorithms and techniques is the existing body of work on data mining.

With that in mind, allow me to direct your attention to the Orange project from the A.I laboratory at the University of Ljubljana, Slovenia.  Orange provides a visual (”boxes and arrows”) programming environment similar to the visual authoring paradigm available in the products of Aleri, Streambase, and others, but with boxes that implement various A.I.-inspired data mining and visualization functions (Classification trees, K-Means clustering, Nomograms, and so on). In addition to a data path/routing-based workflow, scripting functionality is offered via embedded Python. Check out this short white paper on the current set of widgets.

I would be very surprised if we didn’t see some of this sort of technology appearing in the leading CEP platforms over the next year or two.

Now that STAC has provided a venue for verifiable third party benchmarks, we really have a chance to see what is behind CEP vendor claims. While I can hardly consider myself to be objective, I do believe that the strength of the Aleri architecture will be, or at least can be verified by such valid comparison tests. I further expect that some use cases will show the various CEP solutions closer to parity, while others - the tougher ones which will benefit from a more robust architecture, which natively handles updates and deletes like Aleri does - should begin to show a clear advantage for Aleri. It is entirely reasonable to test a range of use cases such as market data absorption, algo trading, market making, order book consolidation, risk aggregation and analysis, and others including those beyond financial services, as that is truly representative of the real world. I could be wrong about how Aleri would fare in such comparisons, but we are happy to subject ourselves to that test.  The question I have now, is where are the other vendors? STAC has been around for a while now. As of today, we have run two tests, and not a single other CEP test has been run. I am happy for Aleri to run any test in STAC provided by another vendor, assuming they would also be willing to run the test which we originally ran a few weeks back, which was also re-run by Intel to show results for their 6-core “Dunnington” processors as compared to the original benchmark that was run on the Tigerton processor. So the benchmark is there. Perhaps I am making too much of the need for vendors to back their claims. I am interested in your thoughts. Please let me know what you think about the opportunity for vendors to stand behind their claims by using STAC, and your thoughts regarding vendors willingness or lack thereof to step up to such transparency. Either respond to this blog, or contact me directly at don.deloach@aleri.com. Thanks! 

Just ran across this article in Advanced Trading:

Exclusive Interview With Nick Leeson: An Inside Look at Rogue Trading by Advanced Trading

and noticed the argument he makes for real-time risk aggregation. OK, so maybe you don’t normally want to be quoting a convicted criminal to bolster your argument, but in this case it has merit. We have long showcased “real-time risk aggregation” as a solution that can be implemented using Aleri CEP. The idea is that firms have data in any number of different systems that don’t talk to each other. While each system produces position and risk information that can give you one picture in isolation, the big picture - when you combine the data across all the systems - can look quite different. Unfortunately many firms do not have the ability to easily combine the data across these disparate systems, and if they do it, happens on a daily (or even less frequent) basis. There’s clear evidence that this isn’t good enough, as Leeson asserts:

“At the end of 1994, “I probably had 500 million pounds from the bank to fund the illegal positions that I had, and the capital base of the bank was only 250 million pounds,” Leeson adds. “And the legal limit that you can lend to a subsidiary is only 20 percent of the bank’s capital, so it was 10 times in breach of that limit.”In SocGen’s case, if Kerviel was able to build up an exchange-traded position that exceeded the capital base of the bank by 13 times, that would suggest some very simple risk management functions were not being performed, Leeson continues. “They may have been checking some complex risk management algorithms but ignoring the most simple things,” he suggests.

Leeson also points to disparate systems as an opening for his rogue trades, something that he says more-unified risk management could help avoid. “At Barings, different systems were looking after plain vanilla equities and something else was looking after futures and options and something else for the warrant products,” he relates. “Only when those reports were actually brought together could you actually see what exposure the bank had and could you compare things. It enabled me to circumvent the accounting and risk management processes.”

Aleri CEP can be used to consolidate and aggregate data from disparate systems in real-time. Based on an event-driven architecture, and with full data normalization capabilities, Aleri can be deployed as an overlay to existings systems. It doesn’t require a massive overhaul of current infrastructure to get everything onto a common system with a common data model. The result is the ability to see the big picture, in real-time, and set up monitors that will generate alerts when limits are reached or exposures pass a critical point.  To learn more about an Aleri based risk managment system, please join our webcast produced by Wall Street & Technology on September 17 at 9:00am PT / 12:00pm ET.

There was a very good post over on the Apama blog, commenting on the recent announcement by Streambase and Oracle that will work together towards the development of a “standard” for applying SQL to event streams.  I think Louis at Apama sums it up well.  As he correctly points out, we added an imperative scripting language (that we call SPLASH) in our 2.0 release  and have continued to enhance it, with significant enhancements in our recent 3.0 release.  We continue to offer SQL-based authoring as a convenience, for those that like the familiarity of SQL. But we find that for many use cases, it’s simply too difficult, if not impossible, to express the logic in SQL - thus the need for SPLASH.

I do think it would be useful to have a standard form of SQL for event streams - for those cases where SQL is a good fit.  To that end, I would welcome an open and inclusive effort to standardize where there is commonality.