Traditionally, big data applications teams have relied upon the power of MapReduce to process large amounts of data. The current generation of AI and big data applications offer interactive rather than batch processing. These AI workloads are no longer constrained by the performance of MapReduce and HDFS but instead achieve massive levels of performance by processing data in memory. In effect, these applications have become stateless in order to circumvent the limitations of the data infrastructure that feeds them.
Modern data lakes, built on disaggregated infrastructure, are able to provide the performance and flexibility required to run cloud native applications, such as AI and ML workloads. By separating compute from the storage layer, we’re no longer constrained by an infrastructure that binds both together. Now these applications can run with a data infrastructure that is based on object storage, which provides both performance and scaling advantages for AI workloads.
Recently, Scott Mcclellan’s team—which analyzes over six petabytes of data using Hadoop technology—created a high-performance data lake using object storage for consumption by big data workloads. Scott shares his experience deploying object storage for AI workloads.
This session is sponsored by Minio.
Scott Mcclellan is CTO at PRGX. A creative, results-driven technology leader, Scott is a change agent and problem solver with a passion for technology. He’s skilled in grasping and explaining the big picture and conceptualizing, developing, and implementing solutions. Scott has substantial experience working with business leaders and C-level executives. Previously, he was chief technologist and VP of engineering for Hewlett-Packard’s cloud services and for scalable computing, where he set technical direction for the company’s scalable computing business and introduced new products focused on cloud service providers and high-performance computing customers.
©2019, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com