We’re excited to announce that on April 18, 2008 Wall Street Systems entered into a definitive agreement with Aleri Inc. to acquire Aleri Global Banking (AGB), the wholesale banking division of Aleri Inc. This acquisition derives from Aleri’s decision to exclusively focus on complex event processing (CEP).
Just want to draw your attention to the STAC announcement today, that Aleri has joined the STAC Benchmark Council and has placed Aleri’s CEP technology - the Aleri Streaming Platform - in the STAC lab. While Aleri joining the benchmark council isn’t new news per se (yes, we were there from the start), having CEP technology resident in the STAC lab is a milestone. Now that Aleri is “Racked and STAC’d” as STAC calls it, it’s available for evaluation by firms that want a fast track way to doing an evaluation without having to set up their own test bed. It will also be available to support the work of the STAC benchmark council. For anyone not familiar with the work of STAC and the services they offer, if you’re in the financial industry I suggest you check it out.
Pulsed subscriptions, which deliver an optimal coalesced block of updates to a subscriber at a fixed, periodic interval, has recently been added to the Aleri Streaming Platform. This subscription method rounds out a number of other subscription types that the Aleri Platform already supports: normal low latency lossless subscription, SQL projection/selection during subscription and both lossy and
dropable subscription models. A short description of each subscription type can be found in a short paper I put together on subscription methods.
I would enjoy a discussion on the subscription modes outlined in the paper, and other extensions to subscription types that would bring value when connecting to a Complex Event Processing engine.
Aleri sees a few common customer situations in the field. Sometimes we’re called in because a homebrew system has collapsed under expanding load, in which case the customer usually has a pretty good idea of the problem they want to solve and how they’d like to solve it. These are simple sales predicated on performance, and there’s not a great deal that needs to be said about them. Likewise, there are those who have decided to move away from ad hoc solutions because they want a better development environment for streaming data application, usually with an eye to reducing implementation turnaround time, preventing duplicated effort and increasing reliability. Again, these cases are pretty straightforward.
There is, though, another class of customers — one that we see with increasingly frequency — who’ve decided to explore Complex Event Processing technology, but don’t know exactly what they’re going to do with it. I want to talk about them today because they’re quite poorly served by available authoring tools. In many cases, they need — in addition to a means by which to specify their application — a way to explore and discover the possibilities hiding in their data.
Significant research remains to be done on user interface strategies that allow application authors to bring together elements from multiple data sources and recombine them in various ways as part of a progressive authoring process. The best sources for this sort of work that I’ve been able to find are semi-structured data interfaces, especially those based on faceted navigation, and data quality browsers.
To get a sense of the possibilities offered by faceted navigation, check out this elastic list demo from Moritz Stefaner’s thesis, then imagine exploring a data set this way and how that might lead smoothly into application modeling using a similar interface. I suspect that tools offering this sort of user experience would be quite popular.
Before we go into the the discussions of features for building the processing loops, let’s make up a simple example. It would make the explanations easier. For this let’s try to design a primitive trading system. For most of the readers this design is absolutely obvious but I still want to go through it to have a concrete example to refer to. Bear with me and eventually we’ll get to the interesting stuff. I’ll be using ASCII art diagrams, since they’re easier to draw, and since I haven’t figured out yet how to upload the real pictures to this blog engine.
Let’s start the design. Our system would receive the bid (buy) and ask (sell) offers from the participants. So we make one stream receiving bids and one receiving asks:
(incoming bids) (incoming asks)
Then we put some logic that selects the best bid and the best ask. This can be implemented internally as multiple streams, but for simplicity let’s consider them as single black-boxes:
(incoming bids)->(best bid) (incoming asks)->(best ask)
Finally, we add the logic that takes the best bid and best ask and checks if they match. If they do, it’s considered a trade and sent as the output of the model.
(incoming bids)->(best bid)->|
|->(match trade)->
(incoming asks)->(best ask)->|
Neat and easy. But by now you can see the drawback: it produces only one trade and that’s it. In this sense it’s like the digital logic that takes a set of signals on the input and produces a set of signals on the output. As long as the inputs fluctuate, the outputs would fluctuate too (with a delay) but once the inputs settle down, the outputs stay fixed. Both effects are not what we need from a real trading system. We don’t want the done trades to fluctuate. We also want to remove the offers that have already resulted in a trade from the incoming offers. So we need to add a loop:
+...........................................................+
. .
+>(completed trades)---+--+ .
| | .
V | .
(incoming bids)->(incomplete bids)->(best bid)->| .
| |->(trade)->
V |
(incoming asks)->(incomplete asks)->(best ask)->|
When a trade is produced, it loops back (shown as a dotted line) to the stream of completed trades. Those are used to filter out the offers that have been already completed to a trade (the arrow from “completed trades” is going under “incomplete bids” to reach the “incomplete asks” too). Now only the incomplete offers participate in the best offer calculation, and the next trade can be matched up.
This diagram is still missing something. From reading the yesterday’s discussion, can you guess, what? The answer is coming in the next installment…
Sergey Babkin, Sr. Engineer
It’s been nagging me for a while, how the model we have, with its consistency rules, is very much like the logical circuits in hardware. This seems to be true for pretty much any Complex Event Processing (CEP) engine but especially so for the Aleri approach that treats the streams as a sort of materialized views, with changes in the underlying tables being propagated throughout the dependent views. The data processing in the models is a pure waterwall. Each stream takes data from upstream, does some processing, and outputs the result downstream. Exactly like the logical circuits! The data comes in here, goes through a few intermediate steps, and the results come out there. No loops. And for a good reason: as every hardware engineer or data flow engineer knows, when used carelessly, the loops can easily bring the device into a generation mode. Sometimes it’s desired but usually “generation” means that the device produces random junk in ever-increasing quantities.
The hardware people have learned to deal with it by using the memory. A device can be seen as an automaton, with the state of the automaton being the “memory”:
input -> [ device (logic, memory) ] -> output
At each step the device takes the input signal, processes it according to the current memory state, and produces the output and the new memory state. How does it know when to make the next step? The synchronization signal does that. So it can be expressed more precisely as:
(input, synchronization) -> [ device (logic, memory) ] -> output
The synchronization signal may have a fixed period (what we’re used to in reading about the CPUs) or be determined from the input. An important point is that once a synchronization cycle starts, the input can’t change any more. We collect the input and then say “that’s it, it’s the processing time”, and we stop accepting the input; any input coming in after that stays beyond the synchronization barrier. Then we process the collected input, send out the output and update the memory state. The cycle has completed, we’re ready for the next cycle: take in the input collected beyond the synchronization barrier, and repeat.
(Note: this also has direct analogies in the RDBMS transactions, but more on this later).
What is missing in the CEP model from the complete hardware analogy, is the “memory”, the state of the whole model. Sure, each stream has its own state, and this state may have non-trivial dependencies on the order of data received. Aleri FlexStream is a good example of this. But this state is localized to a stream. It’s not visible to the model as a whole and does not directly affect other streams. And there is no easy way to make it so, since each stream operates independently, and data may be queued between them in an unpredictable way.
But there are components that can be strung together to break out of the waterfall paradigm. To build the loops, global model state and proper synchronization. More on this in the next installment…
Jon Riecke, Lead Platform Architect for Aleri, recently wrote an article about a unique feature of the Aleri Streaming Platform called FlexStreams. While SQL is great for most common data operations, we have seen many use cases that run up against limitations of a declarative language. FlexStream extends Aleri’s complex event processing offering by enabling a programmable stream operator allowing for procedural programming and retaining state from one event to the next. Without giving too much away, please download and enjoy the article called “Programming with FlexStreams in the Aleri Streaming Platform” at http://www.aleri.com/support/technical-articles/