Handtrack.js: Building gesture-based interactions in the browser using TensorFlow.js
Who is this presentation for?
In the browser, ML can enable truly novel forms of interactions while reaping the benefits associated with on-device computation such as reduced latency for interactive applications, reduced model distribution costs, and enhanced privacy, as data is no longer sent to remote servers for analysis.
Victor Dibia provides an overview of the TensorFlow.js library, benchmarks performance results for image tasks in the browser, and provides a live demonstration of Handtrack.js—a library for prototyping real-time hand-tracking interactions. Handtrack.js is powered by an object detection neural network (MobileNetV2, SSD) and allows you to predict the location (bounding box) of human hands in an image, video, or canvas HTML tag. You’ll learn the steps and best practices for deploying a neural network model in the browser from data collection, model training, and model conversion to TensorFlow.js WebModel format, model hosting, and inference.
What you'll learn
- Understand the state for machine learning in the browser using TensorFlow.js 1.0
- Learn the steps and best practices for deploying a neural network model in the browser from data collection, model training, and model conversion to TensorFlow.js WebModel format, model hosting, and inference
- See examples of hand-tracking interactions prototyped in the browser to help understand UX concerns and future possibilities
Cloudera Fast Forward Labs
Victor Dibia is a research engineer at Cloudera’s Fast Forward Labs where his work focuses on prototyping state-of-the-art machine learning algorithms and advising clients. He’s passionate about community work and serves as a Google Developer Expert in machine learning. Previously, he was a research staff member at the IBM TJ Watson Research Center. His research interests are at the intersection of human-computer interaction, computational social science, and applied AI. He’s a senior member of IEEE and has published research papers at conferences such as AAAI Conference on Artificial Intelligence and ACM Conference on Human Factors in Computing Systems. His work has been featured in outlets such as the Wall Street Journal and VentureBeat. He holds an MS from Carnegie Mellon University and a PhD from City University of Hong Kong.
Leave a Comment or Question
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
For conference registration information and customer service
For more information on community discounts and trade opportunities with O’Reilly conferences
For information on exhibiting or sponsoring a conference
For media/analyst press inquires