Presented By O'Reilly and Cloudera
Make Data Work
September 26–27, 2016: Training
September 27–29, 2016: Tutorials & Conference
New York, NY

Swipe, dip, and hover: Managing card payment data at Visa

Nandu Jayakumar (Oracle)
2:05pm–2:45pm Wednesday, 09/28/2016
Enterprise adoption
Location: 3D 12 Level: Intermediate
Average rating: ***..
(3.00, 1 rating)

Prerequisite knowledge

  • A general understanding of databases and Hadoop/MR, traditional data warehousing, and BI (and how they are implemented at enterprises)
  • Basic experience with big data technologies and practices
  • Familiarity with the workflows of typical data analysts at enterprises
  • What you'll learn

  • Gain insight into how a very risk-averse financial enterprise is adopting big data practices and technologies
  • Learn how to apply some of these big data practices and technologies in your own enterprise
  • Description

    Billions of Visa cards are used around the world to make payments. Each payment transaction has a story. Getting payments from point A to point B is complex, and the resulting data Visa captures reflects this. The scale and complexity of that data is a direct manifestation of the number, variety, and complexity of payment transactions processed by the Visa network.

    As enterprises go, Visa is among the more cautious. Visa systems have stringent availability requirements to ensure payments never fail, designed around proven ideas and technologies. Visa is almost never an early adopter of technologies. Instead, it waits for technologies to harden and prove themselves.

    Nandu Jayakumar explores the adoption of big data practices at Visa and explains how Visa is transforming the way it manages data: database appliances are giving way to Hadoop and HBase; proprietary ETL technologies are being replaced by Spark; and enterprise warehouse data models will be complemented by flexible data schemas. Due to the nature of its business, regulatory compliance and secure management of data are central tenets at Visa. Many open source and less mature technologies tend to fall short in these aspects. Embracing big data technologies has also required a culture change in the engineering teams and the ultimate users of data in the company. Nandu also discusses Visa’s experience with Impala supporting a large reporting setup.

    Topics include:

    • Choosing commodity hardware and scale-out architectures as a way to achieve cost-efficient scaling
    • Trading off agility of data publishing and usage against security and compliance requirements
    • Moving beyond traditional EDW and BI to data science and near-real time data applications at scale
    • Introduction of agile data that is defined less strictly as an important technique for handling rapidly evolving enterprise data needs
    • Visa’s decision to build a hosted, multitenant data lake across the company to reduce the time and effort it takes to build data applications
    • The decision matrix that was used to pick particular technologies, including Impala, Spark, HBase, Chef, and Kafka
    Photo of Nandu Jayakumar

    Nandu Jayakumar

    Oracle

    Nandu Jayakumar is a software architect and engineering leader at Oracle. Before that he was responsible for the long-term architecture of data systems and was Senior Director of data platform development at Visa. Previously, as a senior leader of Yahoo’s well-regarded data team, Nandu built key pieces of Yahoo’s data processing tools and platforms over several iterations, which were used to improve user engagement on Yahoo websites and mobile apps. He also designed large-scale advertising systems and contributed code to Shark (SQL on Spark) during his time there. Nandu holds a bachelor’s degree in electronics engineering from Bangalore University and a master’s degree in computer science from Stanford University, where he focused on databases and distributed systems.