Anomaly detection is the process of identifying data points that do not conform to normal behavior, and it is used ubiquitously at Netflix. For example, real-time systems detect, and raise, outliers when internal systems do not meet some service level agreement. In data warehousing applications, traditional outlier detection methods (e.g. some number of standard deviations) will work for low cardinality dimensions that are normally distributed, but typically dimensions of interest are neither normally distributed nor have low cardinality. In these settings the number of false positives/negatives create an unnecessary overhead and limit the end-user’s ability to respond.
In this session, we present a case study at Netflix where we deployed a variant of the Singular Value Decomposition for anomaly detection in high cardinality dimensions. We then wrapped this in a Business Intelligence tool to present actionable insights for business use.
We will then discuss a specific application centered on payment processing. With more than 50 Million customers worldwide, Netflix has to ensure that the payment methods provided by customers do not fail due to processing problems in the payment network. A typical payment transaction goes through at least 4 external participants (issuers, acquirers, payment gateways, processors, etc…) in addition to Netflix’s systems. The wide array of banks that customers use to pay for Netflix creates this high cardinality dimension, and the complexity of the payment transaction necessitates the need for a different solution than the common methods mentioned above. We will also present the decoupled architecture in the cloud that enables us to provide a highly performing, scalable solution.
Shankar Vederaman leads the Payment Analytics Data science and Engineering team at Netflix. His team is responsible for providing analytical solutions for Payments, Fraud and Retail gift analytics. The solutions include data engineering, BI engineering and analytical story telling. Shankar is highly passionate about creating data products that utilize the power of data science for better business benefits.
Christopher Colburn is just another data scientist at Netflix.
Comments on this page are now closed.
For exhibition and sponsorship opportunities, email firstname.lastname@example.org
For information on trade opportunities with O'Reilly conferences, email email@example.com
For media-related inquiries, contact Maureen Jennings at firstname.lastname@example.org
View a complete list of Strata + Hadoop World contacts
©2015, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com
Apache Hadoop, Hadoop, Apache Spark, Spark, and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries, and are used with permission. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event, which is managed by O'Reilly Media and/or Cloudera.