Join in to explore MLPerf, a common benchmark suite for training and inference for systems from workstations through large-scale servers. In addition to ML metrics like quality and accuracy, MLPerf evaluate metrics such as execution time, power, and cost to run the suite.
Drawing on lessons from the 40-year history of computing benchmarks, MLPerf’s primary goals include:
The seed proposal comes from a collaboration of four teams from two universities and two companies with experience in benchmarking ML systems, the plan is to expand quickly into a much wider collaboration that spans many groups in academia, industry, and other organizations.
David Patterson is a professor emeritus at UC Berkeley, a distinguished engineer in Google Brain, and vice chair of the board of the RISC-V Foundation. His most successful research projects are reduced instruction set computers (RISC), redundant arrays of inexpensive disks (RAID), and network of workstations, which together led to multibillion-dollar industries, seven books, and about 40 honors, including election to the National Academy of Engineering, the National Academy of Sciences, and the Silicon Valley Engineering Hall of Fame. He also shared the ACM Turing award, the IEEE von Neumann Medal, and NEC C&C prize with John Hennessy, past president of Stanford University and coauthor of two of his books. David holds an AB, MS, and PhD, all from UCLA.
Greg Diamos leads computer systems research at Baidu’s Silicon Valley AI Lab (SVAIL), where he helped develop the Deep Speech and Deep Voice systems. Previously, Greg contributed to the design of compiler and microarchitecture technologies used in the Volta GPU at NVIDIA. Greg holds a PhD from the Georgia Institute of Technology, where he led the development of the GPU-Ocelot dynamic compiler, which targeted CPUs and GPUs from the same program representation.
Sharan Narang is a senior researcher on the systems team at Baidu’s Silicon Valley AI Lab (SVAIL), where he leads the effort to benchmark deep learning applications. He released DeepBench in 2016, an open-source benchmark that measures the performance of deep learning workloads. Sharan also focuses on research to improve the performance of deep learning models by reducing their memory and compute requirements. He has explored techniques like pruning neural network weights and reduced precision to achieve this goal. Previously, Sharan worked on next-generation mobile processors at NVIDIA.
Cliff Young is a data scientist on the Google Brain team, where he works on codesign for deep learning accelerators. He is one of the designers of Google’s Tensor Processing Unit (TPU), which is used in production applications including Search, Maps, Photos, and Translate. TPUs also powered AlphaGo’s historic 4-1 victory over Go champion Lee Sedol. Previously, Cliff built special-purpose supercomputers for molecular dynamics at D. E. Shaw Research and worked at Bell Labs. A member of ACM and IEEE, he holds AB, MS, and PhD degrees in computer science from Harvard University.
Peter Mattson is a staff engineer at Google Brain, where he originated and coordinates the multi-organization MLPerf benchmarking effort. Previously, he led the Programming Systems and Applications Group at NVIDIA Research, was VP of software infrastructure for Stream Processors Inc (SPI), and was a managing engineer at Reservoir Labs. He has authored more than a dozen technical papers as well as four patents. His research focuses on accelerating and understanding the behavior of machine learning systems by applying novel benchmarks and analysis tools. Peter holds a PhD and MS from Stanford University and a BS from the University of Washington.
Peter Bailis is an assistant professor of computer science at Stanford University. Peter’s research in the Future Data Systems group and DAWN project focuses on the design and implementation of postdatabase data-intensive systems. He is the recipient of the ACM SIGMOD Jim Gray Doctoral Dissertation Award, an NSF Graduate Research Fellowship, a Berkeley Fellowship for Graduate Study, best-of-conference citations for research appearing in both SIGMOD and VLDB, and the CRA Outstanding Undergraduate Researcher Award. He holds a PhD from UC Berkeley and an AB from Harvard College, both in computer science.
Gu-Yeon Wei is the Robert and Suzanne Case Professor of Electrical Engineering and Computer Science at Harvard University. Gu’s research spans a broad range of areas from mixed-signal I/O circuits to power electronics to accelerator-centric SoC architectures and chip design. To name just one project, his group developed a “BrainSoC” to control Harvard’s RoboBees. Gu’s recent efforts have focused on developing energy-efficient neural network accelerators leveraging codesign opportunities from algorithms through architecture to circuit design. He holds BS, MS, and PhD degrees in electrical engineering from Stanford University.
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
©2018, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com