Skip to main content

Real-Time Analytical Processing (RTAP) using Spark and Shark

Hadoop & Beyond Gramercy Suite
Average rating: ***..
(3.00, 11 ratings)
Slides:   1-PDF 

Hadoop brought MapReduce and big data to mainstream; however, as requirements and usage models expand, new big data analysis paradigms beyond MapReduce have, inevitably, emerged. In particular, there is increasing demand from organizations to discover and explore data iteratively and interactively for real-time insights; these new paradigms can be characterized by the following four salient properties, which we lump together under the term Real-Time Analytical Processing (RTAP):

  1. data ingested & processed in a real-time, streaming fashion
  2. real-time data queried and presented in an online fashion
  3. real-time and history data combined and mined interactively
  4. predominantly RAM-based processing

In this talk, we will present our efforts and experience on building real-time analytical processing framework for several large
websites, leveraging Spark and Shark (the in-memory cluster computing research) from UC Berkeley.

Photo of Jason (Jinquan) Dai

Jason (Jinquan) Dai

Intel

Dai is currently an Engineering Director and Principal Engineer in Intel SSG (Software and Services Group), leading the SW engineering efforts on advanced big data technology development in Intel. Prior to that, he was the lead architect and engineering manager for building the 1st auto-partitioning and parallelizing compiler product for many-core many-thread processors (Intel Network Processor) in the industry. He received M.S. from National University of Singapore, and BSc from Fudan University, both in computer science.

Sponsors

Sponsorship Opportunities

For exhibition and sponsorship opportunities, contact Susan Stewart at sstewart@oreilly.com

Media Partner Opportunities

For information on trade opportunities with O'Reilly conferences email mediapartners
@oreilly.com

Press & Media

For media-related inquiries, contact Maureen Jennings at maureen@oreilly.com

Contact Us

View a complete list of Strata + Hadoop World 2013 contacts