Fueling innovative software
July 15-18, 2019
Portland, OR

Build a self-driving car... without a car! AI-problem solving with Unity and TensorFlow

Paris Buttfield-Addison (Secret Lab Pty. Ltd.), Mars Geldard (University of Tasmania), Tim Nugent (lonely.coffee)
1:30pm5:00pm Monday, July 15, 2019
Incorporating Artificial Intelligence
Location: Portland 256
Secondary topics:  AI Enhanced

Who is this presentation for?

Programmers, team leaders, and those wanting to do AI projects that can't afford to (or don't want to commit to) building a whole car.



Prerequisite knowledge

Nothing special. A basic, basic understanding of ML-concepts would be useful, but nowhere near essential! Basic Python would probably ease progress in the tutorial, too, but is _not_ essential at all. Unity knowledge would make it all easier, but isn't needed.

Materials or downloads needed in advance

We'll supply a download in advance, and simple instructions. Unity is a free download, and TensorFlow is available from common package managers.

What you'll learn

Basic Unity. Basic TensorFlow. Combining the two of them for amazing ML- and AI-powered explorations, training, and testing. How to use a game engine (tutorial will explore Unity + Python TensorFlow, but principles are generally applicable) to test out and explore ML- and AI-problems that have a visual component (e.g. self-driving cars, real-world interactions, etc.)


Are you a scientist who wants to test a research problem without building costly and complicated real-world rigs? A self-driving car engineer who wants to test their AI logic in a constrained virtual world? A data scientist who needs to solve a thorny real-world problem without touching a production environment? Have you considered AI problem solving using game engines?

No? This workshop will teach you how to solve AI and ML problems using the Unity game engine, and Google’s TensorFlow for Python.

In this workshop, we’ll teach you ML and AI problem solving with game engines. Learn how you could use a game engine to train, explore, and manipulate intelligence agents that learn.

Game engines are a great place to explore ML and AI. They’re wonderful constrained problem spaces, tiny little ecosystems for you to explore a problem in. Here you can learn how to use them even though you’re not a game developer.

In this session, we’ll look at:

  • how video game engines are a perfect environment to constrain a problem and train an agent
  • how easy it is to get started, using the Unity engine and Google’s TensorFlow for Python
  • how to build up a model, and use it in the engine, to explore a particular idea or problem
  • PPO (proximal policy optimisation) for generic but useful machine learning
  • deep reinforcement learning, and how it lets you explore and study complex behaviours

Specifically, this session will:

  • teach the very basics of the Unity game engine
  • explore how to setup a scene in Unity for both training and use of a ML model
  • show how to train a model, using TensorFlow (and Docker), using the Unity scene
  • discuss the use of the trained model, and potential applications
  • show you how to train AI agents in complicated scenarios and make the real world better by leveraging the virtual

We’ll explore fun, engaging scenarios, including virtual self-driving cars, bipedal human-like walking robots, and disembodied hands that can play tennis.

This workshop is for non-game developers to learn how they can use game technologies to further their understanding of machine learning fundamentals, and solve problems using a combination of open source tools and (sadly often not open source) game engines. Deep reinforcement learning using virtual environments is the beginning of an exciting new wave of AI.

It’s a bit technical, a bit creative.

Photo of Paris Buttfield-Addison

Paris Buttfield-Addison

Secret Lab Pty. Ltd.

Dr Paris Buttfield-Addison is co-founder of Secret Lab, a game development studio based in beautiful Hobart, Australia. Secret Lab builds games and game development tools, including the multi-award-winning ABC Play School iPad games, the BAFTA- and IGF-winning Night in the Woods, the Qantas airlines Joey Playbox games, and the open source Yarn Spinner narrative game framework. Previously, Paris was mobile product manager for Meebo (acquired by Google). Paris particularly enjoys game design, statistics, the blockchain, machine learning, and human-centered technology research and writes technical books on mobile and game development (more than 20 so far) for O’Reilly Media. He holds a degree in medieval history and a PhD in computing. Find him online at http://paris.id.au and @parisba

Photo of Mars Geldard

Mars Geldard

University of Tasmania

Marina Rose Geldard, more commonly known as Mars, is a technologist from Down Under in Tasmania. Entering the world of technology relatively late as a mature-age student, she has found her place in the world: an industry where she can apply her lifelong love of mathematics and optimization. She compulsively volunteers at industry events, dabbles in research, and serves on the executive committee for her state’s branch of the Australian Computer Society (ACS) as well as the AUC (http://auc.edu.au). She is currently writing ‘Practical Artificial Intelligence with Swift’, for O’Reilly Media, and working on machine learning projects to improve public safety through public CCTV cameras in her home town of Hobart.

Photo of Tim Nugent

Tim Nugent


Dr Tim Nugent pretends to be a mobile app developer, game designer, tools builder, researcher, and tech author. When he isn’t busy avoiding being found out as a fraud, Tim spends most of his time designing and creating little apps and games he won’t let anyone see. He also spent a disproportionately long time writing this tiny little bio, most of which was taken up trying to stick a witty sci-fi reference in. . .before he simply gave up. Tim is currently writing ‘Practical AI with Swift’ for O’Reilly Media.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)