Data platforms are being asked to support an ever-increasing range of workloads and compute environments, including machine learning and elastic cloud platforms. These new workloads and environments introduce challenges for both end users and cluster administrators. Data scientists want the flexibility to run interactive and batch analysis with on-demand compute, data engineers want to ensure the scalability and reliability of production workloads, and cluster administrators want to maintain governance and control over cluster resources and costs. Cloudera has built a machine learning platform, optimized for the cloud, that seeks to balance these competing objectives.
Tristan Zajonc and Tim Chen discuss emerging capabilities in the industry and some of the key design choices they made, including leveraging Kubernetes as a cloud-optimized compute layer and running Spark with a serverless experience. They also share their vision for the road ahead for enterprise machine learning and AI in the cloud.
Tristan Zajonc is CTO for machine learning at Cloudera. Previously, Tristan led engineering for Cloudera Data Science Workbench and was the cofounder and CEO of enterprise data science platform Sense (acquired by Cloudera in 2016). He has over 15 years’ experience in applied data science, machine learning, and machine learning systems development across academia and industry and holds a PhD from Harvard University.
Tim Chen is a software engineer at Cloudera leading cloud initiatives for the company’s enterprise machine learning platform. Previously, he was cofounder and CEO and cofounder of Hyperpilot, a startup focused on applying machine learning to improve performance and cost efficiency of container clusters and big data workloads, led containerization development and design at Mesosphere, and worked at VMware and Microsoft. He’s an Apache PMC member and committer on Apache Drill and Apache Mesos and helped initiate the Spark-on-Kubernetes project and led development of Mesos support for Spark.
©2019, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org