Scaling visualization for big data and analytics in the cloud

Description

The Financial Industry Regulatory Authority (FINRA) is dedicated to investor protection and market integrity through effective and efficient regulation of broker-dealers. In its mission to safeguard the investing public against fraud and bad practices, FINRA analyzes data in the petabyte scale, looking for market manipulation, compliance breeches, and other potentially illegal activities—acquiring as many as 101 billion market events per day, which are ingested, aggregated, and analyzed for the purpose or performing surveillance of the US markets. FINRA uses cloud computing and leverages big data technology to process ever-increasing volume of financial data.

One of the challenges of dealing with such an immense ecosystem of data is being to be able to navigate through all the available information or narrow it down and focus on specific datasets to gain insight and detect and visualize violations. Using rich visualization metaphors and tools, FINRA leverages the usage of machine learning and data analytics to that end.

Jaipaul Agonus and Daniel Monteiro do Carmo Rosa detail big data analytics and visualization practices and tools used by FINRA to support machine learning and other surveillance activities that the Market Regulation Department conducts in the AWS cloud.