Presented By O'Reilly and Cloudera
Make Data Work
Feb 17–20, 2015 • San Jose, CA

Stock Market Order Flow Reconstruction in HBase on AWS

Tigran Khrimian (FINRA)
2:20pm–3:00pm Thursday, 02/19/2015
Hadoop in Action
Location: 210 A/E
Average rating: ****.
(4.25, 4 ratings)

FINRA, an independent regulator charged with protecting investors, processes 30 billion market events per day and analyzes the data in search of patterns that indicate possible manipulation of US financial markets. These orders may be split up and passed through many hands—brokerage firms, securities exchanges, dark pools, etc—before being executed or cancelled. Finding market manipulators in this sea of data requires linking all events in the entire order flow as input to automated and human-driven data analysis.

FINRA uses Hadoop/HBase to create a graph of stock market order flow using data collected from firms and securities exchanges. The power of Hadoop and the elastic nature of the Cloud allowed FINRA to build a system that processes multi-petabyte data volumes in a predictable manner while being able to react quickly to data volume spikes due to market volatility. The resulting graph data is stored in an HBase cluster that provides rapid access to downstream applications and regulatory analysts.

This talk provides an overview of the approach, the use of HBase to link and manage a large (2 trillion records) HBase cluster on AWS. The presentation includes lessons learned in our approach to optimize rowkey design, data backup strategies, performance tuning and running a persistent HBase cluster on Amazon Web Services.

Photo of Tigran Khrimian

Tigran Khrimian

FINRA

Tigran Khrimian is a Senior Director at FINRA responsible for the development of the organization’s Big Data ingestion and management platform, which processes billions of market order events on daily basis. He oversees technology that reconstructs stock market order flow to facilitate the monitoring of the trading practices of securities market participants.

Prior to joining FINRA, Mr. Khrimian was as a key member of the technology team at Chessiecap – an arm of an investment banking company specializing in M&A. Before working at Chessiecap, Mr. Khrimian worked at the Adrenaline Group as a Software Architect, performing rapid software development of commercial products to help start-up companies and well-established businesses with their online presence. Mr. Khrimian started his professional career working as a developer at IBM, building software products in support of IBM’s business critical operations.

Mr. Khrimian holds a BS in Computer Science from University of Maryland and MS in Computer Science from Johns Hopkins University.