Sep 23–26, 2019
Please log in

Handtrack.js: Building gesture-based interactions in the browser using TensorFlow

Victor Dibia (Cloudera Fast Forward Labs)
1:15pm1:55pm Thursday, September 26, 2019
Location: 3B - Expo Hall
Average rating: ****.
(4.00, 1 rating)

Who is this presentation for?

  • Data scientists, frontend web (JavaScript developers) developers, and user experience designers




While JavaScript continues to be the most-used programming language, until recently, there’s been limited frameworks for machine learning that cater to this audience. With the advent of TensorFlow—a library for developing and training ML models in JavaScript for deployment in browser or on Node.js—things are changing. When running inside the browser, TensorFlow uses the GPU of the device via WebGL to enable fast parallelized floating point computation. In Node.js, TensorFlow binds to the TensorFlow C library, enabling full access to TensorFlow.

In the browser, ML can enable truly novel forms of interactions while reaping the benefits associated with on-device computation such as reduced latency for interactive applications, reduced model distribution costs, and enhanced privacy, as data is no longer sent to remote servers for analysis.

Victor Dibia provides an overview of the TensorFlow library, benchmarks performance results for image tasks in the browser, and covers an end-to-end example on his experience building Handtrack.js—a library for prototyping real time hand tracking interactions. Handtrack.js is powered by an object detection neural network (MobileNetV2, SSD) and allows users predict the location (bounding box) of human hands in an image, video, or canvas HTML tag. You’ll learn the steps and best practices for deploying a neural network model in the browser from data collection, model training, and model conversion to TensorFlow WebModel format, model hosting, and inference. Victor also shares a live demo prototyped using Handtrack.js.

Prerequisite knowledge

  • Familiarity with ML and Javascript

What you'll learn

  • Understand the state for ML in the browser using TensorFlow 1.0
  • See a walkthrough of steps and best practices for deploying a neural network model in the browser
  • Discover examples of hand tracking interactions prototyped in the browser, UX concerns, and future possibilities
Photo of Victor Dibia

Victor Dibia

Cloudera Fast Forward Labs

Victor Dibia is a research engineer at Cloudera’s Fast Forward Labs, where his work focuses on prototyping state-of-the-art machine learning algorithms and advising clients. He’s passionate about community work and serves as a Google Developer Expert in machine learning. Previously, he was a research staff member at the IBM TJ Watson Research Center. His research interests are at the intersection of human-computer interaction, computational social science, and applied AI. He’s a senior member of IEEE and has published research papers at conferences such as AAAI Conference on Artificial Intelligence and ACM Conference on Human Factors in Computing Systems. His work has been featured in outlets such as the Wall Street Journal and VentureBeat. He holds an MS from Carnegie Mellon University and a PhD from City University of Hong Kong.

  • Cloudera
  • O'Reilly
  • Google Cloud
  • IBM
  • Cisco
  • Dataiku
  • Intel
  • Io-Tahoe
  • MemSQL
  • Microsoft Azure
  • Oracle Cloud Infrastructure
  • SAS
  • Arcadia Data
  • BMC Software
  • Hazelcast
  • SAP
  • Amazon Web Services
  • Anaconda
  • Esri
  •, Inc.
  • Kyligence
  • Pitney Bowes
  • Talend
  • Google Cloud
  • Confluent
  • DataStax
  • Dremio
  • Immuta
  • Impetus Technologies Inc.
  • Keyence
  • Kyvos Insights
  • StreamSets
  • Striim
  • Syncsort
  • SK holdings C&C

    Contact us

    For conference registration information and customer service

    For more information on community discounts and trade opportunities with O’Reilly conferences

    For information on exhibiting or sponsoring a conference

    For media/analyst press inquires