Presented By O’Reilly and Intel AI
Put AI to Work
April 29-30, 2018: Training
April 30-May 2, 2018: Tutorials & Conference
New York, NY

MLPerf: A benchmark suite for machine learning from an academic-industry cooperative

David Patterson (UC Berkeley), Greg Diamos (Baidu), Sharan Narang (Baidu), Cliff Young (Google), Peter Mattson (Google), Peter Bailis (Stanford University), Gu-Yeon Wei (Harvard University)
11:55am–12:35pm Wednesday, May 2, 2018
Location: Grand Ballroom East

What you'll learn

  • Explore MLPerf, a common benchmark suite for training and inference for systems from workstations through large-scale servers


Join in to explore MLPerf, a common benchmark suite for training and inference for systems from workstations through large-scale servers. In addition to ML metrics like quality and accuracy, MLPerf evaluate metrics such as execution time, power, and cost to run the suite.

Drawing on lessons from the 40-year history of computing benchmarks, MLPerf’s primary goals include:

  • Enabling a fair comparison of competing systems while encouraging innovation that improves the state-of-the-art of ML;
  • Rapidly iterating the benchmark suite to remain relevant in the fast moving ML field and correct mistakes in the formulation of the benchmarks or their ground rules;
  • Enforcing replicability to ensure reliable results;
  • Serving both the commercial and research communities; and
  • Keeping benchmarking effort affordable.

The seed proposal comes from a collaboration of four teams from two universities and two companies with experience in benchmarking ML systems, the plan is to expand quickly into a much wider collaboration that spans many groups in academia, industry, and other organizations.

Photo of David Patterson

David Patterson

UC Berkeley

David Patterson is a professor emeritus at UC Berkeley, a distinguished engineer in Google Brain, and vice chair of the board of the RISC-V Foundation. His most successful research projects are reduced instruction set computers (RISC), redundant arrays of inexpensive disks (RAID), and network of workstations, which together led to multibillion-dollar industries, seven books, and about 40 honors, including election to the National Academy of Engineering, the National Academy of Sciences, and the Silicon Valley Engineering Hall of Fame. He also shared the ACM Turing award, the IEEE von Neumann Medal, and NEC C&C prize with John Hennessy, past president of Stanford University and coauthor of two of his books. David holds an AB, MS, and PhD, all from UCLA.

Photo of Greg Diamos

Greg Diamos


Greg Diamos leads computer systems research at Baidu’s Silicon Valley AI Lab (SVAIL), where he helped develop the Deep Speech and Deep Voice systems. Previously, Greg contributed to the design of compiler and microarchitecture technologies used in the Volta GPU at NVIDIA. Greg holds a PhD from the Georgia Institute of Technology, where he led the development of the GPU-Ocelot dynamic compiler, which targeted CPUs and GPUs from the same program representation.

Photo of Sharan Narang

Sharan Narang


Sharan Narang is a senior researcher on the systems team at Baidu’s Silicon Valley AI Lab (SVAIL), where he leads the effort to benchmark deep learning applications. He released DeepBench in 2016, an open-source benchmark that measures the performance of deep learning workloads. Sharan also focuses on research to improve the performance of deep learning models by reducing their memory and compute requirements. He has explored techniques like pruning neural network weights and reduced precision to achieve this goal. Previously, Sharan worked on next-generation mobile processors at NVIDIA.

Photo of Cliff Young

Cliff Young


Cliff Young is a data scientist on the Google Brain team, where he works on codesign for deep learning accelerators. He is one of the designers of Google’s Tensor Processing Unit (TPU), which is used in production applications including Search, Maps, Photos, and Translate. TPUs also powered AlphaGo’s historic 4-1 victory over Go champion Lee Sedol. Previously, Cliff built special-purpose supercomputers for molecular dynamics at D. E. Shaw Research and worked at Bell Labs. A member of ACM and IEEE, he holds AB, MS, and PhD degrees in computer science from Harvard University.

Photo of Peter Mattson

Peter Mattson


Peter Mattson is a staff engineer at Google Brain, where he originated and coordinates the multi-organization MLPerf benchmarking effort. Previously, he led the Programming Systems and Applications Group at NVIDIA Research, was VP of software infrastructure for Stream Processors Inc (SPI), and was a managing engineer at Reservoir Labs. He has authored more than a dozen technical papers as well as four patents. His research focuses on accelerating and understanding the behavior of machine learning systems by applying novel benchmarks and analysis tools. Peter holds a PhD and MS from Stanford University and a BS from the University of Washington.

Photo of Peter Bailis

Peter Bailis

Stanford University

Peter Bailis is an assistant professor of computer science at Stanford University. Peter’s research in the Future Data Systems group and DAWN project focuses on the design and implementation of postdatabase data-intensive systems. He is the recipient of the ACM SIGMOD Jim Gray Doctoral Dissertation Award, an NSF Graduate Research Fellowship, a Berkeley Fellowship for Graduate Study, best-of-conference citations for research appearing in both SIGMOD and VLDB, and the CRA Outstanding Undergraduate Researcher Award. He holds a PhD from UC Berkeley and an AB from Harvard College, both in computer science.

Photo of Gu-Yeon Wei

Gu-Yeon Wei

Harvard University

Gu-Yeon Wei is the Robert and Suzanne Case Professor of Electrical Engineering and Computer Science at Harvard University. Gu’s research spans a broad range of areas from mixed-signal I/O circuits to power electronics to accelerator-centric SoC architectures and chip design. To name just one project, his group developed a “BrainSoC” to control Harvard’s RoboBees. Gu’s recent efforts have focused on developing energy-efficient neural network accelerators leveraging codesign opportunities from algorithms through architecture to circuit design. He holds BS, MS, and PhD degrees in electrical engineering from Stanford University.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)