A software accelerator for machine learning

Description

Over the past decade, the industry has invested heavily in hardware accelerators like GPUs and TPUs. While these investments are essential in solving the hard machine learning problems, they aren’t sufficient to close the gap created by increasing data sizes and the death of Moore’s law. In their Turing lecture, John Hennessy and Dave Paterson, two legends of computer science, claim that it’s indeed possible to speed up computations purely by software optimizations to the order of 61K times. However, in practice it’s a hard task. Even a 10x improvement would be a great win. RocketML has done exactly that.

Vinay Rao and Santi Adavani explain why software architectures will lead next generation of machine learning approaches. They take you through the history of distributed machine learning and hardware architectures, detailing the breakthroughs that have been made in software, and demonstrate how software-only approaches can ultimately scale better.

Vinay and Santi share a parallelized implementation of the L-BFGS algorithm on a distributed system that includes a cluster of commodity computing machines. RocketML uses the open source HPCC (high-performance computing cluster) Systems platform as the underlying distributed system to implement the L-BFGS algorithm. They offer an overview of the HPCC Systems framework and explain how it allows for the parallel and distributed computations important for big data analytics; they then detail their own implementation of the L-BFGS algorithm on this platform. Experimental results show that this large-scale implementation of the L-BFGS algorithm can easily scale from training models with millions of parameters to models with billions of parameters by simply increasing the number of commodity computational nodes.