Sep 23–26, 2019

Using Spark for crunching astronomical data on the LSST scale

Petar Zecevic (SV Group d.o.o.)
11:20am12:00pm Thursday, September 26, 2019
Location: 1E 07/08
Secondary topics:  Data Integration and Data Processing

Who is this presentation for?

Data Engineers, Software Architects, Big Data & Astronomy enthusiasts

Level

Beginner

Description

The slew of upcoming large-scale astronomical surveys promises exciting times for both astronomy and computer science. One of the most important future surveys is Large Scale Survey Telescope, or LSST. Its unique design and excellent location allow it to go both “wide” and “deep”, at the same time covering large regions of the sky and obtaining images of the faintest objects. LSST will produce one 3.2 giga-pixel image every 20 seconds and will do that every night for 10 years. That will result in the first “video” of the deep sky in history and (according to some estimates) about 80 PB of data. Furthermore, worldwide scientific community will receive real-time alerts triggered by changes in the sky, within 60 seconds from their detection.
In this talk I will describe how LSST image processing pipeline uses acquired images to produce catalogs of astronomical objects. Together with colleagues from University of Washington, I built AXS (Astronomy Extensions for Spark), a system for processing and quickly cross-matching catalog data, based on Apache Spark. I will explain its architecture and what is behind its great performance.

Prerequisite knowledge

Basics of distributed data processing and SQL

What you'll learn

Introduction to LSST data processing pipelines; details of a distributed spatial cross-matching approach
Photo of Petar Zecevic

Petar Zecevic

SV Group d.o.o.

Petar started out as a Java developer almost 20 years ago, and worked as a Software Architect, Team Leader and IBM software consultant. After switching to the exciting new field of Big Data technologies, he wrote the Spark in Action book (Manning 2016) and these days primarily works on Apache Spark and Big Data projects. Today he is CTO of SV Group in Zagreb, Croatia, while also pursuing his PhD at the University of Zagreb. He is collaborating with Astronomy Department at the University of Washington on building new methods for processing images and data from future astronomical surveys.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Contact us

confreg@oreilly.com

For conference registration information and customer service

partners@oreilly.com

For more information on community discounts and trade opportunities with O’Reilly conferences

strataconf@oreilly.com

For information on exhibiting or sponsoring a conference

Contact list

View a complete list of Strata Data Conference contacts