Lessons learned from scaling the tech stack of a modern analytics platform

Scott Castle (Sisense)

3:45pm–4:25pm Thursday, September 26, 2019

Location: 1E 07/08

Data Engineering and Architecture

Secondary topics: Data, Analytics, and AI Architecture

Average rating:

(4.00, 1 rating)

Who is this presentation for?

Data engineers, data architects, data scientists, and data analysts

Level

Intermediate

Description

In a given day, analysts write more than 300,000 new lines of SQL in Periscope Data, on top of the millions of lines already executing on the platform. As the average analyst on the platform has increased their workload to use it for more than 20+ hours per week, Periscope has scaled up dramatically with dozens of terabytes of RAM supporting its 2,000+ teams as part of Sisense.

In this session, Scott Castle, General Manager at Sisense and former VP of Product at Periscope Data, will discuss lessons learned from scaling up Periscope Data to support incredibly large volumes of data and queries from its data teams. He’ll highlight the process of migrating from Heroku to Kubernetes and discovering new ways to leverage its power, plus other developments and changes to the tech stack, including warehouse and data processing tools and partnerships, that have allowed Periscope users to delve deeper into new data science and machine learning analysis.

Prerequisite knowledge

Familiarity with or interest in ETL concepts, data warehousing, and programming languages like SQL, Python, and R

What you'll learn

Explore the challenges behind building a platform to support data science and analysis at massive scale

Scott Castle

Sisense

Scott Castle is GM of the Data Business at Sisense and served as VP of Product for Periscope Data prior to its merger with Sisense in 2019. At Periscope Data, Scott oversaw product strategy, planning, design and delivery. He brings over 20 years of experience in software development and product management at leading technology companies including Adobe, Electric Cloud, and FileNet. Scott has a B.A. degree from the University of Massachusetts, Amherst, and a master’s degree from UC Irvine, both in computer science.