Sep 9–12, 2019

Using Deep Learning Models to Extract the Most Value From 360-Degree Images

Shourabh Rawat (Trulia)
4:00pm4:40pm Thursday, September 12, 2019
Location: 231

Who is this presentation for?

This session is intended for an audience with a basic understanding of AI and deep learning models.

Level

Intermediate

Description

Recent camera advances enabling automatic panorama generation have made 360-degree images ubiquitous in industries ranging from real estate to e-commerce and travel. These panoramic views enable an immersive experience that benefits consumers. Trulia’s parent company, Zillow Group, uses this technology to create 3D home views that allow users to see a complete view of a room and find the perfect home. But 360-degree images can create a challenge for businesses: how do you direct viewers to the most important parts of the scene?

The wide field of view created by panoramas means that businesses must ensure viewers see the most engaging part of the image first. This need becomes paramount when panoramas need to be represented as static 2D images. The key here is to identify a salient thumbnail specifically chosen to give the most informative view of each panorama to help drive engagement.

In order to compute a saliency score, Trulia relies on three different Deep Convolutional Neural Networks:
-Scene Model helps capture the representativeness of a viewpoint to ensure the most relevant photos are chosen for a real estate listing (i.e. a kitchen or living room versus a blank wall or window)
-Attractiveness Model penalizes low visual quality such as blurry or dark photos and rewards aesthetically pleasing photos with a high score. However, they also train a deep learning model to label properties as either “Luxury” or “Fixer Upper” as home location and listing price tends to affect the photo quality as well
-Appropriateness Model helps differentiate between relevant viewpoints like views of a bedroom from irrelevant views like walls or humans

In this session, Shourabh Rawat, manager of data science at Trulia, will demonstrate how to utilize and train saliency score models, deep learning techniques, and algorithms to identify and extract the most visually informative and pleasing viewpoints to create this salient thumbnail.

Audience members will walk away with specific insights and actions they can use to:
-Create a saliency model that defines criteria for salient thumbnails, ensuring they are representative, attractive and diverse
-Extract salient thumbnails while maintaining important aspects like specific field of view, 3D orientation, aspect ratio and viewport size
-Create an algorithm to rank all potential thumbnails extracted from the panorama based on a defined saliency criteria using the three models: Scene, Attractiveness, and Appropriateness
-Deploy these images within their own organization’s practices

Prerequisite knowledge

A basic understanding of deep learning algorithms and models.

What you'll learn

Audience members will walk away with specific insights and actions they can use to: -Create a saliency model that defines criteria for salient thumbnails, ensuring they are representative, attractive and diverse -Extract salient thumbnails while maintaining important aspects like specific field of view, 3D orientation, aspect ratio and viewport size -Create an algorithm to rank all potential thumbnails extracted from the panorama based on a defined saliency criteria using the three models: Scene, Attractiveness, and Appropriateness -Deploy these images within their own organization’s practices
Photo of Shourabh Rawat

Shourabh Rawat

Trulia

Shourabh is a Manager of Data Science in the data engineering organization at Trulia (Zillow Group). He has over 5 years of industry experience working in AI, deep learning, computer vision and personalization, deploying these systems to production at scale. Shourabh and his team focus on developing data science solutions to gain a better understanding of Trulia’s customers, specifically how they engage with content and property recommendations. Shourabh completed his Master’s degree from Carnegie Mellon University where he did research on “Event Detection in Consumer Videos,” applying deep learning on multi-modal (audio/images) data.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Contact us

confreg@oreilly.com

For conference registration information and customer service

partners@oreilly.com

For more information on community discounts and trade opportunities with O’Reilly conferences

Become a sponsor

For information on exhibiting or sponsoring a conference

Contact list

View a complete list of O'Reilly AI contacts