It seems strange to talk about adopting Hadoop 10 successful years after its release. But to put this in context from a Visa perspective, IBM celebrated 50 years of mainframes in 2014 and held up Visa as one of its most successful current users. So what did it take for a risk-averse enterprise like Visa to adopt the big data ecosystem?
Billions of Visa cards are used to make payments around the world. Each payment transaction has a story. Getting payments from point A to point B is complex, and the resulting data Visa captures reflects this. The scale and complexity of that data is a direct manifestation of the number, variety, and complexity of payment transactions processed by the Visa network.
As enterprises go, Visa is among the more cautious. Visa systems have stringent availability requirements to ensure payments never fail, designed around proven ideas and technologies. Due to the nature of its business, regulatory compliance and secure management of data are central tenets at Visa. Many open source and less mature technologies tend to fall short in these aspects. Visa tends to wait for technologies to harden and prove themselves before widely adopting them within the enterprise.
Nandu Jayakumar and Rajesh Bhargava explore the adoption of big data practices at Visa and explains how Visa is transforming the way it manages data: database appliances are giving way to Hadoop and HBase; proprietary ETL technologies are being replaced by Spark; and enterprise warehouse data models will be complemented by flexible data schemas. Nandu and Rajesh also discuss the culture change in the engineering teams and the ultimate users of data in the company that embracing big data technologies has required and explain how Visa uses Hive and Impala to support a large enterprise reporting setup and offloading the enterprise data warehouse.
Nandu Jayakumar is a software architect and engineering leader at Visa, where he is currently responsible for the long-term architecture of data systems and leads the data platform development organization. Previously, as a senior leader of Yahoo’s well-regarded data team, Nandu built key pieces of Yahoo’s data processing tools and platforms over several iterations, which were used to improve user engagement on Yahoo websites and mobile apps. He also designed large-scale advertising systems and contributed code to Shark (SQL on Spark) during his time there. Nandu holds a bachelor’s degree in electronics engineering from Bangalore University and a master’s degree in computer science from Stanford University, where he focused on databases and distributed systems.
Rajesh Bhargava is an engineering leader at Visa, where he currently leads the effort to offer the Hadoop Platform as a Service (PaaS). Prior to that, as a Director and founding member of a predictive analytics startup, he built scalable platforms for predictive analytics, segmentation & real-time recommendations. At Yahoo, as an engineering lead, he helped define and design a number of audience and advertiser analytics solutions. Rajesh holds a bachelor’s degree in computer science and a master’s in computer applications.
Comments on this page are now closed.
©2017, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com
Apache Hadoop, Hadoop, Apache Spark, Spark, and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries, and are used with permission. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event, which is managed by O'Reilly Media and/or Cloudera.