Skip to main content
Make Data Work
Oct 15–17, 2014 • New York, NY

Open Source Real Time BI using Storm, Hadoop, Titan, Druid & D3

Anil Madan (PayPal)
2:35pm–3:15pm Friday, 10/17/2014
Hadoop & Beyond
Location: 1 E20/1 E21
Average rating: ***..
(3.71, 14 ratings)

The talk will illustrate how PayPal has built its behavioral
analytics using a combination of open source projects. It will first
provide an overview of our Real Time Analytics pipeline which
extensively leverages Storm & Hadoop. Next it will go deep into how
we have deployed Druid as a real time distributed OLAP
metrics store scaling it to 50+ dimensions with billions of data points
and sub-second response times. This opens up a world of exploratory
analytics with ability to do real time drill-downs, roll-ups, and
segmentation. Further, it will outline how we integrated the richness of D3 visualization framework with Druid backend to provide self-servicereporting and business intelligence.
Last but not the least it will provide an
overview on how we have solved visitor pathing and funnel analytics
using a combination of Apache Titan & Gremlin. Along the way it will
highlight the importance of Metadata – consistency of instrumentation,
event model, tagging & standardization of metrics which powers the
end to end data integrity, quality and precision.

Photo of Anil Madan

Anil Madan


Anil is the Sr. Director of Engineering at PayPal running several of their Online & Offline systems around PayPal Behavioral analytics, Personalization and Marketing. Prior to this he built out eBay’s Big Data Hadoop platform with an underlying infrastructure handling petabytes of data to power search relevance and other efforts around enhancing Trust, Value & Selection. His prior work at eBay also included building scalable, highly available real time systems of Experimentation & Segmentation. He has spent last 10 years in building distributed online big data systems. Anil has a Masters in Computers Science from Pune University and Bachelors in Physics from St Stephens College Delhi, India.