Aleri CEP Blog
Wall Street Systems acquires Aleri Global Banking, a division of Aleri, Inc.
We’re excited to announce that on April 18, 2008 Wall Street Systems entered into a definitive agreement with Aleri Inc. to acquire Aleri Global Banking (AGB), the wholesale banking division of Aleri Inc. This acquisition derives from Aleri’s decision to exclusively focus on complex event processing (CEP).
Aleri CEP available for eval in STAC Lab
Just want to draw your attention to the STAC announcement today, that Aleri has joined the STAC Benchmark Council and has placed Aleri’s CEP technology - the Aleri Streaming Platform - in the STAC lab. While Aleri joining the benchmark council isn’t new news per se (yes, we were there from the start), having CEP technology resident in the STAC lab is a milestone. Now that Aleri is “Racked and STAC’d” as STAC calls it, it’s available for evaluation by firms that want a fast track way to doing an evaluation without having to set up their own test bed. It will also be available to support the work of the STAC benchmark council. For anyone not familiar with the work of STAC and the services they offer, if you’re in the financial industry I suggest you check it out.
Pulsed Subscriptions
Pulsed subscriptions, which deliver an optimal coalesced block of updates to a subscriber at a fixed, periodic interval, has recently been added to the Aleri Streaming Platform. This subscription method rounds out a number of other subscription types that the Aleri Platform already supports: normal low latency lossless subscription, SQL projection/selection during subscription and both lossy and
dropable subscription models. A short description of each subscription type can be found in a short paper I put together on subscription methods.
I would enjoy a discussion on the subscription modes outlined in the paper, and other extensions to subscription types that would bring value when connecting to a Complex Event Processing engine.
The Future of Complex Event Processing Authoring
Aleri sees a few common customer situations in the field. Sometimes we’re called in because a homebrew system has collapsed under expanding load, in which case the customer usually has a pretty good idea of the problem they want to solve and how they’d like to solve it. These are simple sales predicated on performance, and there’s not a great deal that needs to be said about them. Likewise, there are those who have decided to move away from ad hoc solutions because they want a better development environment for streaming data application, usually with an eye to reducing implementation turnaround time, preventing duplicated effort and increasing reliability. Again, these cases are pretty straightforward.
There is, though, another class of customers — one that we see with increasingly frequency — who’ve decided to explore Complex Event Processing technology, but don’t know exactly what they’re going to do with it. I want to talk about them today because they’re quite poorly served by available authoring tools. In many cases, they need — in addition to a means by which to specify their application — a way to explore and discover the possibilities hiding in their data.
Significant research remains to be done on user interface strategies that allow application authors to bring together elements from multiple data sources and recombine them in various ways as part of a progressive authoring process. The best sources for this sort of work that I’ve been able to find are semi-structured data interfaces, especially those based on faceted navigation, and data quality browsers.
To get a sense of the possibilities offered by faceted navigation, check out this elastic list demo from Moritz Stefaner’s thesis, then imagine exploring a data set this way and how that might lead smoothly into application modeling using a similar interface. I suspect that tools offering this sort of user experience would be quite popular.
Analogies between CEP models and digital logic (Part 2)
Before we go into the the discussions of features for building the processing loops, let’s make up a simple example. It would make the explanations easier. For this let’s try to design a primitive trading system. For most of the readers this design is absolutely obvious but I still want to go through it to have a concrete example to refer to. Bear with me and eventually we’ll get to the interesting stuff. I’ll be using ASCII art diagrams, since they’re easier to draw, and since I haven’t figured out yet how to upload the real pictures to this blog engine.
Let’s start the design. Our system would receive the bid (buy) and ask (sell) offers from the participants. So we make one stream receiving bids and one receiving asks:
(incoming bids) (incoming asks)
Then we put some logic that selects the best bid and the best ask. This can be implemented internally as multiple streams, but for simplicity let’s consider them as single black-boxes:
(incoming bids)->(best bid) (incoming asks)->(best ask)
Finally, we add the logic that takes the best bid and best ask and checks if they match. If they do, it’s considered a trade and sent as the output of the model.
(incoming bids)->(best bid)->|
|->(match trade)->
(incoming asks)->(best ask)->|
Neat and easy. But by now you can see the drawback: it produces only one trade and that’s it. In this sense it’s like the digital logic that takes a set of signals on the input and produces a set of signals on the output. As long as the inputs fluctuate, the outputs would fluctuate too (with a delay) but once the inputs settle down, the outputs stay fixed. Both effects are not what we need from a real trading system. We don’t want the done trades to fluctuate. We also want to remove the offers that have already resulted in a trade from the incoming offers. So we need to add a loop:
+...........................................................+
. .
+>(completed trades)---+--+ .
| | .
V | .
(incoming bids)->(incomplete bids)->(best bid)->| .
| |->(trade)->
V |
(incoming asks)->(incomplete asks)->(best ask)->|
When a trade is produced, it loops back (shown as a dotted line) to the stream of completed trades. Those are used to filter out the offers that have been already completed to a trade (the arrow from “completed trades” is going under “incomplete bids” to reach the “incomplete asks” too). Now only the incomplete offers participate in the best offer calculation, and the next trade can be matched up.
This diagram is still missing something. From reading the yesterday’s discussion, can you guess, what? The answer is coming in the next installment…
Analogies between CEP models and digital logic (Part 1)
Sergey Babkin, Sr. Engineer
It’s been nagging me for a while, how the model we have, with its consistency rules, is very much like the logical circuits in hardware. This seems to be true for pretty much any Complex Event Processing (CEP) engine but especially so for the Aleri approach that treats the streams as a sort of materialized views, with changes in the underlying tables being propagated throughout the dependent views. The data processing in the models is a pure waterwall. Each stream takes data from upstream, does some processing, and outputs the result downstream. Exactly like the logical circuits! The data comes in here, goes through a few intermediate steps, and the results come out there. No loops. And for a good reason: as every hardware engineer or data flow engineer knows, when used carelessly, the loops can easily bring the device into a generation mode. Sometimes it’s desired but usually “generation” means that the device produces random junk in ever-increasing quantities.
The hardware people have learned to deal with it by using the memory. A device can be seen as an automaton, with the state of the automaton being the “memory”:
input -> [ device (logic, memory) ] -> output
At each step the device takes the input signal, processes it according to the current memory state, and produces the output and the new memory state. How does it know when to make the next step? The synchronization signal does that. So it can be expressed more precisely as:
(input, synchronization) -> [ device (logic, memory) ] -> output
The synchronization signal may have a fixed period (what we’re used to in reading about the CPUs) or be determined from the input. An important point is that once a synchronization cycle starts, the input can’t change any more. We collect the input and then say “that’s it, it’s the processing time”, and we stop accepting the input; any input coming in after that stays beyond the synchronization barrier. Then we process the collected input, send out the output and update the memory state. The cycle has completed, we’re ready for the next cycle: take in the input collected beyond the synchronization barrier, and repeat.
(Note: this also has direct analogies in the RDBMS transactions, but more on this later).
What is missing in the CEP model from the complete hardware analogy, is the “memory”, the state of the whole model. Sure, each stream has its own state, and this state may have non-trivial dependencies on the order of data received. Aleri FlexStream is a good example of this. But this state is localized to a stream. It’s not visible to the model as a whole and does not directly affect other streams. And there is no easy way to make it so, since each stream operates independently, and data may be queued between them in an unpredictable way.
But there are components that can be strung together to break out of the waterfall paradigm. To build the loops, global model state and proper synchronization. More on this in the next installment…
FlexStreams - an Aleri differentiator
Jon Riecke, Lead Platform Architect for Aleri, recently wrote an article about a unique feature of the Aleri Streaming Platform called FlexStreams. While SQL is great for most common data operations, we have seen many use cases that run up against limitations of a declarative language. FlexStream extends Aleri’s complex event processing offering by enabling a programmable stream operator allowing for procedural programming and retaining state from one event to the next. Without giving too much away, please download and enjoy the article called “Programming with FlexStreams in the Aleri Streaming Platform” at http://www.aleri.com/support/technical-articles/
Semantic CEP
One thing that often comes up when discussing the future of CEP is how we will model metadata and domain specific processing rules in the future. Whenever this topic arises, my mind turns to Semantic Web technology in general, and more specifically to the ontology modeling facilities that have developed in that context. This sense was especially strong at the recent OMG conference on the future of CEP standards, where many of the requirements aired by the participants seemed tailor-made for this sort of solution.
My general intuition is that we could leverage previous work on using domain-specific ontologies to speed the implementation of automated systems that reason using first order logic. An obvious experiment that I haven’t had time to conduct would be to specify a small rule-based application starting from an OWL (Web Ontology Language) ontology.
It was thus with good cheer that I read TIBCO’s recent blog post on this topic, which affirms that if I am mad, my madness is at least shared by others in the field.
(Full disclosure: I’ve done significant work in the SemWeb area, authoring a triple store some years ago and kibitzing on a number of large projects, all of which may contribute to my optimism for this approach).
Dataflow programming as a technique for easy multi-threading
An article in yesterday’s Wall Street Journal caught my eye:
“Racing to Gain Edge on Multicore Chips” (by Don Clark, 17 March 2008, page B4) explains how Intel and Microsoft are turning to universities for research into new programming techniques to make it easier to take advantage of the parallel processing capabilities of modern multi-core architectures.
What struck me was that the Aleri CEP engine is just such a programming tool: it uses a dataflow programming model to allow someone defining a complex set of business logic to break it up into a series of discrete operations, where the output of one becomes the input of another. The Aleri Streaming Platform is fully multi-threaded, so each “stream” runs in it’s own thread, allowing streams to run in parallel, taking full advantage of all the cores and CPUs that are available. An application developer, using Aleri’s high level authoring tools, defines the business logic that will be applied to incoming events to produce the desired output, without needing to have any awareness of threads, cores and parallel processing. Thus Aleri takes care of the scalability and how to harness all available cores; the user simply focuses on the business logic.
Aleri CEP Wins 2008 Jolt Productivity Award
We received word this week that Aleri was the winner of the prestigious Jolt Productivity Award in the highly competitive “Database Engines and Data Tools” category. This is the 18th year in which Dr. Dobbs Magazine and CMP handed out these awards.
The Jolt Awards, and more specifically the “Database Engines and Data Tools” category, go far beyond simply Complex Event Processing, so it really is a testament to how much notoriety and recognition the complex event processing industry is receiving. Aleri’s CEP offering, the Aleri Streaming Platform, was the only CEP product to be given a Jolt Award this year, in any category.
Each year the Jolt Product Excellence and Productivity Awards recognize those products, books, and websites that have “jolted” the industry in the past year. The fact that Aleri won the award is a testament to our commitment to providing an innovative product that provides our customers and partners with a strong competitive edge.
A special thanks goes out to our clients and to our stellar development team whose expertise and tireless work has molded our CEP offering into what it is today – award winning. Please click here to learn more about Aleri’s exciting Jolt Productivity Award win.