Presented By O’Reilly and Cloudera
Make Data Work
September 11, 2018: Training & Tutorials
September 12–13, 2018: Keynotes & Sessions
New York, NY

When Tiramisu Meets Online Fashion Retail

Patty Ryan (Microsoft), CY Yam (Microsoft), Elena Terenzi (Microsoft)
4:35pm–5:15pm Wednesday, 09/12/2018
Data science and machine learning
Location: 1A 15/16 Level: Intermediate
Secondary topics:  Media, Marketing, Advertising, Retail and e-commerce

Who is this presentation for?

Data Scientists

Prerequisite knowledge

General understanding of applied machine learning.

What you'll learn

Application of the latest deep learning architecture for image segmentation and background removal. How to apply both tradition ML and deep learning to mitigate the issues, especially related to human error, associated with maintaining very large catalogue for online retail.



Large online fashion retailers face the problem of efficiently maintaining catalogue of millions of items. Due to human error, it is not unusual that some items have duplicate entries. To trawl along such a large catalogue manually is near to impossible. To prevent such unintentional human error, one could take a snapshot of newly arrived item with a mobile phone, and have algorithm to automatically check if such an item is already registered, based on its visual appearance.
The challenge now is that the query image taken by a mobile phone has a busy background, while the target image is of studio quality, see figure below. When applying any content-based image retrieval, it is highly likely that the performance will be hindered by the difference of their visual content, mainly due to the busy background of a mobile image and the clean studio image, not to mention inconsistent folding/creases, lighting, scale and point of view angle.

To increase the success rate, it is prudent to remove the background of the query image before applying any retrieval algorithms. Therefore, we set off to develop a specialised segmentation tool for garments.


We have developed a specialised segmentation model for background removal, or garment (foreground) segmentation using one of the most recent deep learning architecture, Tiramisu, achieving remarkable segmentation accuracy of 94% with 200 training images. In this case, we applied a simplified version of Tiramisu. This garment segmentation tool is then used in a specific case of content-based image retrieval, where the query image is a mobile snapshot of a garment with busy background, while the target is a clean catalogue image of studio quality. It has been proved to significantly improve the content-based
image retrieval performance.

We first discussed a very successful foreground segmentation tool which is based on computer vision technique – GrabCut, how it is being used to create labelled data, and how we develop a deep learning-based specialised segmentation tool – Tiramisu. Then, we show results of segmentation where the model performs well, and where it’s performance is less satisfactory. It is interesting to note a few examples where the original label (human-labelled) was not perfect, still the model partially reproduces the same error. It seems like the network has not only ‘learnt’ to segment, but it could may have also ‘learnt’ the human labelling error. Finally, we demonstrate how this can be applied to help to prevent the issue of duplicate entries into a already very large catalogue maintained by online fashion retailers.

Photo of Patty Ryan

Patty Ryan


Patty Ryan is an applied data scientist for Microsoft. She codes with our partners and customers to tackle tough problems using machine learning approaches, with sensor, text and vision data. She’s a graduate of University of Michigan. On Twitter: @singingdata

Photo of CY Yam

CY Yam


Invented new ways to recognise people by the way the move in the early days, now, main focus is to apply machine learning techniques into solving various problems in daily life.

Photo of Elena Terenzi

Elena Terenzi


Elena Terenzi is a software development engineer at Microsoft, where she brings business intelligence solutions to Microsoft Enterprise customers and advocates for business analytics and big data solutions for the manufacturing sector in Western Europe, such as helping big automotive customers implement telemetry analytics solutions with IoT flavor in their enterprises. She started her career with data as a database administrator and data analyst for an investment bank in Italy. Elena holds a master’s degree in AI and NLP from the University of Illinois at Chicago.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)