Recently, Aleri entered into another agreement with Wall Street Systems. This time, it is for a partnership in selling the Aleri Liquidity Management System (LMS). We originally worked with a customer, Barclays (link to case study), to develop this system. It has subsequently gone into a number of other banks, and we feel it represents the most thorough offering in the market for providing an intraday, holistic view of liquidity across the bank. In fact, the AITE Group just came out with a report entitled “Liquidity Management Solutions: Targeting Financial Institution Treasury” in which they position Aleri as a solid option for “financial institutions looking to provide their group treasury with a comprehensive, enterprise-wide (liquidity management) system.”

In the interaction we had with Wall Street Systems as they were acquiring the Global Banking Division of Aleri, we gained a healthy respect for their presence in the treasury space, as well as the reach, depth, and quality of their organization. I certainly felt they also developed a healthy respect for the quality of both Aleri LMS and our underlying Complex Event Processing (CEP) technology, the Aleri Streaming Platform. The partnership, which enables Wall Street to co-sell Aleri LMS into and beyond its customer base, only seemed natural. It pairs an award winning product with a market leader. And, because LMS provides a wonderful complement to the solutions Wall Street Systems bring to their customers, it will greatly extend the global reach of LMS. We are excited about the future of this partnership.

Well, it’s an exciting day for me - and for many of my Aleri colleagues. We’ve been hard at work on version 3.0 of the Aleri Streaming Platform - our high performance CEP engine, since October of last year. Needless to say, this is a VERY significant release and we think people will find a lot to like about it. The biggest changes are in connectivity and usability. For connectivity, we’ve eliminated the need to use external adapters for many common data interfaces and the base product now includes a set of built-in connectors. We will expand the set of built in connectors over time (tell us what you need!).  We still use external adapters for specialty connections, like market data via Reuters RMDS, ACTIV Financial, Bloomberg, Fixnetix and others - but upcoming releases of these adapters will work with the new 3.0 integration framework, making them configurable from the Aleri Studio.

As for the Studio, it has a new look and feel that makes it much easier to use. We’ve also streamlined and enhanced the underlying languages, both AleriML, our xml-based modeling language, as well as SPLASH, our scripting language.

You can read more about it on the aleri website, or even bettter, download it and take it for a spin. Hope you like it!

Here’s an interesting pattern, one that uses features from our upcoming 3.0 release of the Aleri Streaming Platform: internal pulsing to control the amount of recalculation.

The problem is simple. Suppose there are a lot of updates flowing into a stream (updates are supported natively in the Aleri Streaming Platform). A good example is a stock market feed that keeps the last tick for each symbol. Some of the downstream calculations might, however, be computationally expensive, and you might not even need to recalculate on every change. You might want, for instance, to recalculate only every second or every ten seconds. How can you collect and pulse the updates so that the expensive recalculations are done periodically instead of continuously?

[Note: the Aleri Streaming Platform has a related feature—called “pulsed subscription”—that allows an external client to coalesce changes every n seconds. But that deals with data sent to the outside world, not processed internally.]

In 3.0, the new dictionary data structure and the timer facility allow you to code internal pulsing. Let’s suppose that the stream to control is called “InStream”. We’ll write a small FlexStream that does the pulsing. First, let’s define two local variables in the FlexStream:

  int32 version := 0; 

  dictionary(typeof(InStream), int32) versionMap;

These two variables keep a current version, and a version number for each record. The SPLASH code handling events from the input stream is

  { 

    versionMap[InStream] := version; 

  }

The special Timer block within the FlexStream sends the inserts and updates:

  { 

    for (k in versionMap) { 

      if (version = versionMap[k]) output setOpcode(k, upsert); 

    } 

    version := version + 1; 

  }

(You can configure the interval between runs of the Timer block in numbers of seconds.) Notice how only those events with the current version get sent downstream, and how the version number gets incremented for the next set of updates.

That’s it. There are other patterns built from data structures, and I’ll give some more examples in future blog posts.

In the past few weeks I have been in a number of meetings where the outlook for the banking sector has been discussed. Nobody is optimistic. It feels like most banks are bracing themselves for the other shoe to drop. The credit crunch of last August may have been the leading indicator of other market conditions which may further negatively impact banking worldwide. Emerging on almost everyone’s radar is the concern about Credit Default Swaps. This is interesting in that it highlights the more basic fear of not really understanding exposure. The lack of transparency, especially in many over the counter contracts, combined with the growing volumes of data and the increased velocity of trading and corresponding data make it near impossible to keep track of what you have, what you owe, to whom, and when. And while most banks have spent a great deal of time and money trying to mitigate risk, few seem comfortable that they are on safe ground. So how does this relate to CEP? If you are operating in the dark, the first step is to bring light back into the room. CEP is emerging as an effective tool to bridge many of these gaps. We have several customers who are managing positions across multiple silos to better understand what they have, where they have it, what it looks like, where they got it, and who needs it.  The Aleri Liquidity Management Systems is a great example of this. There are many others. The point is that as the market braces itself for another storm, CEP is and will continue to emerge as an effective tool for gaining a better understanding of firm-wide or bank-wide information quickly and fairly easily. And while this will certainly not solve all of the bank’s problems or prevent any disaster, it can certainly help to solve some problems, prevent some disasters, and position the bank or the firm better than they would be otherwise. And to that end, while we see cutbacks taking place in many organizations, we also note that many of those same organizations continue to add to their CEP teams and increase their interest in CEP projects.  

Marc Adler recently blogged that CEP engines might benefit from using distributed object caches (such as Velocity, Tangasol, Gemfire and GigaSpaces) for persistence/recovery or on-demand queries; see http://magmasystems.blogspot.com/2008/07/first-experiments-with-velocity.html.  Marc suggested that persistence and recovery using object caches would be faster than persisting to external databases as some CEP solutions have chosen to do.  This may be true, but I don’t see how distributed object caches would be faster for persistence and recovery than an embedded storage manager that’s been optimized for CEP.  I agree with Marc that it would not be that difficult for most CEP vendors to support publishing to object caches for querying and persistence/recovery, but I believe that there are better solutions for CEP persistence & recovery relative to performance.  There may, however, be other valid reasons to integrate CEP with object caches, such as wanting to exceed physical memory, and I am interested in that.

At Aleri, we built our own log store that is very much optimized for event stream analytics and persistence/recovery thereof. Using this approach we normally see a factor of two performance degradation when running with persistence on, rather than the order-of-magnitude performance hit we saw with other storage management approaches we tried. Jack Rusher wrote a nice paper on utilizing log stores for CEP which was accepted and presented at VLDB last year; see http://www.aleri.com/images/pdfs/A-Log-Structured-Store-for-Streaming-Data.pdf.  Aleri provides three types of storage managers (log, memory, and stateless). All streams in the Aleri Streaming Platform get assigned to an instance of a storage manager, and one or more instances of a storage manager can be defined in a model, so one can persist the streams of interest in one or more log stores. Please see Scott Kolodzieski’s short article describing these storage managers and their differences at http://www.aleri.com/images/pdfs/CEP-and-Storage-where-does-all-the-data-go.pdf.

Marc also pointed out that concurrent on-demand queries may impact continuous query processing, but I believe that usually having the CEP support both types of queries (i.e., continuous and on-demand) would require less hardware than a CEP that’s back-ended by an object cache or database, plus it avoids the data replication.  There are cases, however, where this functional separation of continuous query processing from on-demand SQL query processing may make sense, especially if the data retention windows are different across these query types or the on-demand SQL query load is extremely high or volatile.

Marc asked for the database, CEP, and object cache vendors to come up with a common querying method.  For the Aleri CEP and most commercial RDBMSs this is currently SQL, but we would also be happy with LINQ.   In case anyone is curious, we’ve supported on-demand SQL queries and our 3 storage managers since our first commercial release (available in 4Q2005).

Colin suggests that the existing CEP solutions need to focus more on production quality features.  I agree that many of the CEP solutions did not focus enough up front on production features such as hot fail-over, distributed services, persistence that performs, dynamic service modifications (i.e., change an existing continuous query without service interruption), and storage management flexibility.  While we at Aleri may not have initially focused enough on ease-of-use (something we’ve corrected in our upcoming 3.0 release), I think we did nail these kinds of production features early on in our roadmap and I invite all prospective customers to take a closer look.  Of course, we’re always looking for feedback and where to focus next.