October 28–31, 2019

Handtrack.js: Building Gesture-based Interactions in the Browser Using Tensorflow.js

Victor Dibia (Cloudera Fast Forward Labs)
1:40pm2:20pm Thursday, October 31, 2019
Location: Grand Ballroom E

Who is this presentation for?

Front-end web (javascript) developers, User experience designers, Data scientists

Level

Intermediate

Description

While JavaScript continues to be the most-used programming language, until recently, there have been limited frameworks for machine learning that cater to this audience. With the advent of Tensorflow.js – a library for developing and training ML models in JavaScript (browser and Node.js) – things are changing. When running inside the browser, Tensorflow.js utilizes the GPU of the device via WebGL to enable fast parallelized floating point computation. When run in Node.js, TensorFlow.js binds to the TensorFlow C library, enabling full access to TensorFlow.

In the browser, Machine Learning can enable truly novel forms of interactions, while reaping the benefits associated with on-device computation, such as reduced latency for interactive applications, reduced model distribution costs, and enhanced privacy, as data is no longer sent to remote servers for analysis.

In this talk, I provide an overview of the Tensorflow.js library, benchmark performance results for image tasks in the browser, and provide a live demonstration of Handtrack.js – a library for prototyping real time hand tracking interactions. Handtrack.js is powered by an object detection neural network (MobilenetV2, SSD) and allows users predict the location (bounding box) of human hands in an image, video, or canvas html tag. The talk will also cover steps and best practices for deploying a neural network model in the browser – from data collection, model training, model conversion to Tensorflow.js webmodel format, model hosting, and inference.

Prerequisite knowledge

None, come as you are.

What you'll learn

- Understand the state for Machine Learning in the Browser using Tensorflow.js 1.0 - Learn steps and best practices for deploying a neural network model in the browser - from data collection, model training, model conversion to Tensorflow.js webmodel format, model hosting, and inference. - View examples of hand-tracking interactions prototyped in the browser, to help in understanding UX concerns and future possibilities.
Photo of Victor Dibia

Victor Dibia

Cloudera Fast Forward Labs

Victor Dibia is a Research Engineer with Cloudera’s Fast Forward Labs where his work focuses on prototyping state of the art machine learning algorithms and advising clients. Prior to this, he was a Research Staff Member at the IBM TJ Watson Research Center, New York. His research interests are at the intersection of human computer interaction, computational social science, and applied AI. A senior member of IEEE, Victor has published work at venues like the AAAI Conference on Artificial Intelligence and ACM Conference on Human Factors in Computing Systems. His work has been featured in outlets such as the Wall Street Journal and VentureBeat. He holds an M.S. from Carnegie Mellon University and a Ph.D. from City University of Hong Kong.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Contact us

confreg@oreilly.com

For conference registration information and customer service

partners@oreilly.com

For more information on community discounts and trade opportunities with O’Reilly conferences

sponsorships@oreilly.com

For information on exhibiting or sponsoring a conference

Contact list

View a complete list of TensorFlow World Conference contacts