Fueling innovative software
July 15-18, 2019
Portland, OR

Build a self-driving car. . .without a car: AI problem solving with Unity and TensorFlow

Paris Buttfield-Addison (Secret Lab), Mars Geldard (University of Tasmania), Tim Nugent (Lonely Coffee)
1:30pm5:00pm Monday, July 15, 2019
Incorporating Artificial Intelligence
Location: Portland 256
Secondary topics:  AI Enhanced
Average rating: ***..
(3.80, 20 ratings)

Who is this presentation for?

  • Programmers, team leaders, and those wanting to do AI projects but can't afford to build a whole car




Whether you’re a scientist who wants to test a research problem without building costly and complicated real-world rigs, a self-driving car engineer who wants to test your AI logic in a constrained virtual world, or a data scientist who needs to solve a thorny real-world problem without touching a production environment, AI problem solving using game engines might just be for you.

Games are wonderful contained problem spaces, making them great places to explore AI—even if you’re not a game developer. Paris Buttfield-Addison, Mars Geldard, and Tim Nugent teach you how to solve AI and ML problems using the Unity game engine and Google’s TensorFlow for Python to train, explore, and manipulate intelligent agents that learn.


  • How video game engines are a perfect environment to constrain a problem and train an agent
  • How easy it is to get started using the Unity engine and Google’s TensorFlow for Python
  • How to build up a model and use it in the engine to explore a particular idea or problem
  • Proximal policy optimization (PPO) for generic but useful machine learning
  • Deep reinforcement learning and how it lets you explore and study complex behaviors

Specifically, Paris, Mars, and Tim:

  • Teach the very basics of the Unity game engine
  • Explore a scene setup in Unity for both training and use of a ML model
  • Show how to train a model with TensorFlow (and Docker) using the Unity scene
  • Discuss the use of the trained model and potential applications
  • Show you how to train AI agents in complicated scenarios and make the real world better by leveraging the virtual

They’ll explore fun, engaging scenarios, including virtual self-driving cars, bipedal human-like walking robots, and disembodied hands that can play tennis, for nongame developers to learn how to use game technologies to further their understanding of machine learning fundamentals and solve problems using a combination of open source tools and (sadly, often not open source) game engines. Deep reinforcement learning using virtual environments is the beginning of an exciting new wave of AI.

It’s a bit technical, a bit creative.

Prerequisite knowledge

  • A basic understanding of ML concepts, Python, and Unity (useful but not required)

Materials or downloads needed in advance

  • A laptop with TensorFlow, Unity, and Docker installed (Downloads will be supplied in advance with simple instructions.)
  • Check out the installation instructions.

What you'll learn

  • Learn how to use a game engine to test out and explore ML and AI problems that have a visual component (e.g., self-driving cars, real-world interactions, etc.)
Photo of Paris Buttfield-Addison

Paris Buttfield-Addison

Secret Lab

Paris Buttfield-Addison is a cofounder of Secret Lab, a game development studio based in beautiful Hobart, Australia. Secret Lab builds games and game development tools, including the multi-award-winning ABC Play School iPad games, the BAFTA- and IGF-winning Night in the Woods, the Qantas airlines Joey Playbox games, and the Yarn Spinner narrative game framework. Previously, Paris was a mobile product manager for Meebo (acquired by Google). Paris particularly enjoys game design, statistics, blockchain, machine learning, and human-centered technology. He researches and writes technical books on mobile and game development (more than 20 so far) for O’Reilly; he recently finished writing Practical AI with Swift and is currently working on Head First Swift. He holds a degree in medieval history and a PhD in computing. Paris loves to bring machine learning into the world of practical and useful. You can find him on Twitter as @parisba.

Photo of Mars Geldard

Mars Geldard

University of Tasmania

Marina Rose Geldard (Mars) is a technologist from Down Under in Tasmania. Entering the world of technology relatively late as a mature-age student, she has found her place in the world: an industry where she can apply her lifelong love of mathematics and optimization. She compulsively volunteers at industry events, dabbles in research, and serves on the executive committee for her state’s branch of the Australian Computer Society (ACS) as well as the AUC. She’s writing Practical Artificial Intelligence with Swift for O’Reilly and working on machine learning projects to improve public safety through public CCTV cameras in her hometown of Hobart.

Photo of Tim Nugent

Tim Nugent

Lonely Coffee

Tim Nugent pretends to be a mobile app developer, game designer, tools builder, researcher, and tech author. When he isn’t busy avoiding being found out as a fraud, Tim spends most of his time designing and creating little apps and games he won’t let anyone see. He also spent a disproportionately long time writing his tiny little bio, most of which was taken up trying to stick a witty sci-fi reference in…before he simply gave up. He’s writing Practical Artificial Intelligence with Swift for O’Reilly and building a game for a power transmission company about a naughty quoll. (A quoll is an Australian animal.)

Comments on this page are now closed.


Picture of Paris Buttfield-Addison
Paris Buttfield-Addison | Cofounder
07/14/2019 9:42am PDT

We’re excited to see everyone tomorrow!

Just a reminder to install the perquisites if you have chance! It will mean you can get straight into it at OSCON!

Details are here: https://secretlab.institute/2019/07/11/installing-unity-ml-agents/

Please post if you have any questions.

If you don’t have time, or can’t manage to make it work, please still come along and we’ll figure it out with you on site!

(@parisba on Twitter)

Picture of Mars Geldard
Mars Geldard | Tutor
07/13/2019 7:46am PDT

Hi Stephen!

The documentation was corrected shortly after original posting to read ‘pip install tensorflow==1.7.1’ on that line. If you run into any further troubles, we’re continuing to update the post—as we double-check the activities on different machines and setups and with different people—to include any troubles or ambiguities we encounter.

Apologies for the inconvenience.

Stephen Smith | Java Programmer
07/13/2019 7:39am PDT

Installing on a MacBook Pro using the instructions athttps://secretlab.institute/2019/07/11/installing-unity-ml-agents/ Running into some trouble:

(UnityML) HISJMEN36:~ ssmith$ pip install tensorflow1.71 Collecting tensorflow1.71
ERROR: Could not find a version that satisfies the requirement tensorflow1.71 (from versions: 0.12.1, 1.0.0, 1.1.0rc0, 1.1.0rc1, 1.1.0rc2, 1.1.0, 1.2.0rc0, 1.2.0rc1, 1.2.0rc2, 1.2.0, 1.2.1, 1.3.0rc0, 1.3.0rc1, 1.3.0rc2, 1.3.0, 1.4.0rc0, 1.4.0rc1, 1.4.0, 1.4.1, 1.5.0rc0, 1.5.0rc1, 1.5.0, 1.5.1, 1.6.0rc0, 1.6.0rc1, 1.6.0, 1.7.0rc0, 1.7.0rc1, 1.7.0, 1.7.1, 1.8.0rc0, 1.8.0rc1, 1.8.0, 1.9.0rc0, 1.9.0rc1, 1.9.0rc2, 1.9.0, 1.10.0rc0, 1.10.0rc1, 1.10.0, 1.10.1, 1.11.0rc0, 1.11.0, 1.12.0rc0, 1.12.0rc1, 1.12.0rc2, 1.12.0, 1.12.2, 1.12.3, 1.13.0rc0, 1.13.0rc1, 1.13.0rc2, 1.13.1, 1.14.0rc0, 1.14.0rc1, 1.14.0, 2.0.0a0, 2.0.0b0, 2.0.0b1) ERROR: No matching distribution found for tensorflow1.71

If not 1.71 which version should we use? I tried not specifying the version and it loaded the latest. That was fine but pip install mlagents expects tensorflow 1.71:
ERROR: Could not find a version that satisfies the requirement tensorflow1.7.1 (from mlagents) (from versions: 1.13.0rc1, 1.13.0rc2, 1.13.1, 1.14.0rc0, 1.14.0rc1, 1.14.0, 2.0.0a0, 2.0.0b0, 2.0.0b1) ERROR: No matching distribution found for tensorflow1.7.1 (from mlagents)


Picture of Paris Buttfield-Addison
Paris Buttfield-Addison | Cofounder
07/10/2019 1:41pm PDT


If you’re attending this session, please check out this guide to installing the prerequisites:


Post here if you have any questions.