FireEye has a consolidated RDBMS data platform used for research and analytics. It’s currently 25 TB in size with more than 200 tables, and it’s growing at rate of 25 to 30 GB per day. The platform, which holds transaction tables that are terabytes in size each with a billion+ rows, has seen exponential growth (from 1 TB in 2013 to 25 TB in 2016) in the last three years. FireEye realized that it had to start exploring new avenues for the platform since a traditional RDBMS platform was unable to scale to the current volume of data.
Ganesh Prabhu, Alex Rivlin, and Vivek Agate present FireEye’s journey migrating data from RDBMS to Cloudera’s big data Hadoop platform. The migration was completed with a very lean team of two engineers and a single administrator in a very short period of time. Ganesh, Alex, and Vivek cover the plan’s details, migration approach, and tools implemented to help with the migration and explore the challenges faced along the way. If you are are looking at embarking on this RDBMS to Hadoop journey, join in to learn from FireEye’s experience.
Ganesh Prabhu is a staff software engineer at FireEye with 20+ years of RDBMS and engineering experience.
Vivek Agate is a staff software engineer at FireEye with 8+ years of experience in software design and development in various Java technologies.
Alex Rivlin leads the team responsible for dynamic threat intelligence at FireEye, which includes a crowdsourced malware exchange across FireEye customers and a malware analytics platform supporting FireEye’s research. For two decades, Alex developed novel analytical capabilities for high-tech companies in projects such as semiconductor failure analysis, supply chain optimization, pricing, and cybersecurity. Previously, Alex worked at Altera (currently part of Intel), where he was hired to develop analytical platform for semiconductor test. His very first project earned a US patent for optimization of bulk loading of data to RDBMS. He later joined supply chain optimization project, where he was responsible for operational analytics. One of his developments allowed real-time reallocation of materials, a feature not available in any commercial packages. Alex also spent time at Flextronics, where he was in charge of project management for implementation of global procurement solution and master data management.
Comments on this page are now closed.
©2017, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org
Apache Hadoop, Hadoop, Apache Spark, Spark, and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries, and are used with permission. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event, which is managed by O'Reilly Media and/or Cloudera.