September 26-27, 2016
New York, NY

End-to-end learning for autonomous driving

Urs Muller (NVIDIA)
3:45pm–4:25pm Monday, 09/26/2016
Verticals and applications
Location: River Pavilion B Level: Beginner

What you'll learn

  • Understand how deep learning is used in autonomous driving, with lessons applicable to everyday enterprises
  • Description

    Urs Muller presents the architecture and training methods used to build an autonomous road-following system. A key aspect of the approach is eliminating the need for hand-programmed rules and procedures such as finding lane markings, guardrails, or other cars, thereby avoiding the creation of a large number of “if, then, else” statements.

    Urs and his team trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands, using an NVIDIA DevBox and Torch 7 for training and an NVIDIA DRIVE PX self-driving car computer also running Torch 7 for determining where to drive. The system operates at 30 frames per second (FPS). This end-to-end approach proved surprisingly powerful. With minimal training data from humans, the system learns to drive in traffic on local roads with or without lane markings and on highways. It also operates in areas with unclear visual guidance, such as in parking lots and on unpaved roads. The system automatically learns internal representations of the necessary processing steps such as detecting useful road features with only the human steering angle as the training signal. It was never explicitly trained to detect, for example, the outline of roads.

    Compared to explicit decomposition of the problem, such as lane marking detection, path planning, and control, this end-to-end system optimizes all processing steps simultaneously. This will eventually lead to better performance—because the internal components self-optimize to maximize overall system performance instead of optimizing human-selected intermediate criteria (e.g., lane detection)—and smaller systems made possible because the system learns to solve the problem with the minimal number of processing steps.

    Photo of Urs Muller

    Urs Muller


    Urs Muller is a developer at NVIDIA, where he built and leads an autonomous driving team that creates novel deep-learning solutions for self-driving cars on NVIDIA’s high-performance DRIVE PX platform. Previously, Urs worked at Bell Labs and later founded Net-Scale Technologies, Inc., a prime contractor on several robotics and machine-learning DARPA programs.