Over the past decade, the industry has invested heavily in hardware accelerators like GPUs and TPUs. While these investments are essential in solving the hard machine learning problems, they aren’t sufficient to close the gap created by increasing data sizes and the death of Moore’s law. In their Turing lecture, John Hennessy and Dave Paterson, two legends of computer science, claim that it’s indeed possible to speed up computations purely by software optimizations to the order of 61K times. However, in practice it’s a hard task. Even a 10x improvement would be a great win. RocketML has done exactly that.
Vinay Rao and Santi Adavani explain why software architectures will lead next generation of machine learning approaches. They take you through the history of distributed machine learning and hardware architectures, detailing the breakthroughs that have been made in software, and demonstrate how software-only approaches can ultimately scale better.
Vinay and Santi share a parallelized implementation of the L-BFGS algorithm on a distributed system that includes a cluster of commodity computing machines. RocketML uses the open source HPCC (high-performance computing cluster) Systems platform as the underlying distributed system to implement the L-BFGS algorithm. They offer an overview of the HPCC Systems framework and explain how it allows for the parallel and distributed computations important for big data analytics; they then detail their own implementation of the L-BFGS algorithm on this platform. Experimental results show that this large-scale implementation of the L-BFGS algorithm can easily scale from training models with millions of parameters to models with billions of parameters by simply increasing the number of commodity computational nodes.
Vinay Rao is the founder and CEO of RocketML, a machine learning platform on a mission to lead and enable transformation of the world toward artificial intelligence.
Santi Adavani is a cofounder at RocketML, where he and his team are building a superfast engine for building machine learning models. Previously, Santi was a product manager and software development lead in the Technology and Manufacturing Group at Intel. He holds a PhD in computational sciences from the University of Pennsylvania. His areas of expertise include high-performance computing, nonlinear optimization, partial differential equations, machine learning, and big data.
©2019, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com