Presented By O'Reilly and Cloudera
Make Data Work
31 May–1 June 2016: Training
1 June–3 June 2016: Conference
London, UK

Using Spark and Hadoop in high-speed trading environments

Fergal Toomey (Corvil), Pierre Lacave (Corvil Ltd.)
14:05–14:45 Thursday, 2/06/2016
Hadoop use cases
Location: Capital Suite 10/11 Level: Intermediate
Average rating: **...
(2.14, 21 ratings)

Prerequisite knowledge

Attendees should have a basic understanding of Spark application design and Hadoop and HBase architecture.


High-speed, machine-automated trading is now responsible for more than half of all US equity trades, and its continued growth across security markets seems inevitable. As human agents become less involved, and machines drive these transactions to a greater extent, it is becoming increasingly important to closely monitor these automated transactions at both a high level and a granular level to help inform future strategic decisions, control risk, and ensure regulatory compliance.

When it comes to processing and analyzing large-scale data for the trading of hundreds of millions of dollars, every packet matters. Strategically, the insights derived from analysis in these environments are highly valuable to the financial audience. Fergal Toomey and Pierre Lacave demonstrate how to effectively use Spark and Hadoop to analyze trading activity in the world’s largest financial institutions.

Topics include:

  • Why Spark and Hadoop are the right tools for analysis in these intense environments
  • Optimizing Spark and Hadoop applications to work with high-volume, real-time trading data
  • How to source authoritative, accurate monitoring data without impacting trading system performance
Photo of Fergal Toomey

Fergal Toomey


Fergal Toomey is a specialist in network data analytics and a founder of Corvil, where he has been intensively involved in developing key product innovations directly applicable to managing IT system performance. Fergal has been involved in the design and development of innovative measurement and analysis algorithms for the past 12 years. Previously, he was an assistant professor at the Dublin Institute for Advanced Studies, where he was a member of the Applied Probability Group, which also included Raymond Russell, Corvil’s CTO. Fergal holds an MSc in physics and a PhD in applied probability theory, both from Trinity College, Dublin.

Photo of Pierre Lacave

Pierre Lacave

Corvil Ltd.

Pierre Lacave is a senior software engineer at Corvil. Pierre has contributed to more than 200 Corvil analytics plugins to decode, analyze, and enrich network data in enterprise and electronic trading infrastructure, and he is also lead developer in the integration of the Corvil flagship product with the Hadoop/Spark ecosystem. Pierre studied computer science in Paris, France, and Chongqing, China, and holds a double MSC in software engineering.