Presented By O'Reilly and Cloudera
Make Data Work
March 13–14, 2017: Training
March 14–16, 2017: Tutorials & Conference
San Jose, CA

Swipe, dip, and hover: Managing card payment data at Visa

4:20pm5:00pm Thursday, March 16, 2017
Enterprise adoption
Location: LL20 C Level: Intermediate
Average rating: *****
(5.00, 2 ratings)

Who is this presentation for?

  • CIOs, IT managers, software developers, fintech developers, and software vendors

Prerequisite knowledge

  • A general understanding of databases and Hadoop/MR, traditional data warehousing, and BI (and how they are implemented at enterprises)
  • Basic experience with big data technologies and practices
  • Familiarity with the workflows of typical data analysts at enterprises

What you'll learn

  • Gain insight into how a very risk-averse financial enterprise is adopting big data practices and technologies
  • Learn how to apply some of these big data practices and technologies in your own enterprise

Description

It seems strange to talk about adopting Hadoop 10 successful years after its release. But to put this in context from a Visa perspective, IBM celebrated 50 years of mainframes in 2014 and held up Visa as one of its most successful current users. So what did it take for a risk-averse enterprise like Visa to adopt the big data ecosystem?

Billions of Visa cards are used to make payments around the world. Each payment transaction has a story. Getting payments from point A to point B is complex, and the resulting data Visa captures reflects this. The scale and complexity of that data is a direct manifestation of the number, variety, and complexity of payment transactions processed by the Visa network.

As enterprises go, Visa is among the more cautious. Visa systems have stringent availability requirements to ensure payments never fail, designed around proven ideas and technologies. Due to the nature of its business, regulatory compliance and secure management of data are central tenets at Visa. Many open source and less mature technologies tend to fall short in these aspects. Visa tends to wait for technologies to harden and prove themselves before widely adopting them within the enterprise.

Nandu Jayakumar and Rajesh Bhargava explore the adoption of big data practices at Visa and explains how Visa is transforming the way it manages data: database appliances are giving way to Hadoop and HBase; proprietary ETL technologies are being replaced by Spark; and enterprise warehouse data models will be complemented by flexible data schemas. Nandu and Rajesh also discuss the culture change in the engineering teams and the ultimate users of data in the company that embracing big data technologies has required and explain how Visa uses Hive and Impala to support a large enterprise reporting setup and offloading the enterprise data warehouse.

Topics include:

  • Choosing commodity hardware and scale-out architectures as a way to achieve cost-efficient scaling
  • Balancing agility of data publishing and usage against security and compliance requirements
  • Moving beyond traditional EDW and BI to data science and near-real time data applications at scale
  • Visa’s decision to build a hosted, multitenant data lake across the company to reduce the time and effort it takes to build data applications
  • The decision matrix that was used to pick particular technologies, including Impala, Spark, HBase, Chef, and Kafka
Photo of Nandu Jayakumar

Nandu Jayakumar

Visa

Nandu Jayakumar is a software architect and engineering leader at Visa, where he is currently responsible for the long-term architecture of data systems and leads the data platform development organization. Previously, as a senior leader of Yahoo’s well-regarded data team, Nandu built key pieces of Yahoo’s data processing tools and platforms over several iterations, which were used to improve user engagement on Yahoo websites and mobile apps. He also designed large-scale advertising systems and contributed code to Shark (SQL on Spark) during his time there. Nandu holds a bachelor’s degree in electronics engineering from Bangalore University and a master’s degree in computer science from Stanford University, where he focused on databases and distributed systems.

Rajesh Bhargava

Visa

Rajesh Bhargava is an engineering leader at Visa, where he currently leads the effort to offer the Hadoop Platform as a Service (PaaS). Prior to that, as a Director and founding member of a predictive analytics startup, he built scalable platforms for predictive analytics, segmentation & real-time recommendations. At Yahoo, as an engineering lead, he helped define and design a number of audience and advertiser analytics solutions. Rajesh holds a bachelor’s degree in computer science and a master’s in computer applications.

Comments on this page are now closed.

Comments

Aditya Verma | SENIOR SOLUTIONS DELIVERY MANAGER
03/28/2017 12:33am PDT

Hello Nandu,

Thanks for the great session. Would you be posting the material used for presentation?