Tag Archives: CEP

Complex Event Processing with Esper

mpUs9Ca7BTaEsKnojmH7A_gEsper is a component for complex event processing (CEP) and event series analysis.

Complex Event Processing (CEP): Event processing is a method of tracking and analyzing streams of information about events and deriving conclusions. Complex event processing, or CEP, combines data from multiple sources to infer events or patterns. The goal of complex event processing is to identify meaningful events such as opportunities or threats and to respond quickly [Wikipedia].

Overview: Esper can process historical data, real-time in eventshigh-velocity data, and high-variety data.  Esper has been described as highly scalable, memory-efficient, in-memory computing, SQL-standards-based, minimal latency, real-time streaming-capable, and designed for Big Data.  SQL streaming analytics is a commonly used term for the technology.

Domain Specific Language: Esper offers a Domain Specific Language (DSL) for processing events. The Event Processing Language (EPL) is a declarative language for dealing with high frequency time-based event data. The designers of EPL created the language to emulate and extend SQL.

Use Cases: Use cases include business process management and automation, process monitoring, business activity monitoring (BAM), reporting exceptions, operational intelligence, algorithmic trading, fraud detection, risk management, network and application monitoring, intrusion detection, SLA monitoring, sensor network applications, RFID reading, scheduling and control of fabrication lines, and air traffic control.

Bryophyllum daigremontianum or mother of thousands plantData Windows, Indexes, and Atomic Operations: Data windows support managing fine-grained event expiry, event retention periods, and conditions for events discarding.  Esper supports explicit indexes as hash and btree, update-insert-delete, also known as merge or upsert, and select-and-delete in atomic operations.

Tables, Patterns, Operations, Contexts, and Enumerations: Tables provide aggregation state.  Patterns support specifying complex time-based and correlation-based relationships. Available operations include grouping, aggregation, rollup, cubing, sorting, filtering, transforming, merging, splitting or duplicating of event series or streams. Context declarations allow controlling detection lifetime and concurrency. Enumeration methods execute lambda-expressions to analyze collections of values or events.

Scripting Support: Scripting integration is available for JavaScript, MVEL and other JSR 223 scripts.  This integration allows you to specify code as part of EPL queries.

Approximation Algorithms: Approximation Algorithms support summarizing data in streams.  For instance, the Count-min sketch (or CM sketch) is a probabilistic, sub-linear space, streaming algorithm that can approximate data stream frequency and top-k, without retaining distinct values in memory.

Event Representation and Inheritance: Events can be represented as Java objects, Map interface implementations, Object-arrays, or XML documents, and do not require transformation among these representations.  Esper supports event-type inheritance and polymorphism for all event types including for Map and object-array representations.  Event properties can be simple, indexed, mapped or nested.

For more information, see the Codehaus page on Esper.