Using Spark for crunching astronomical data on the LSST scale
Who is this presentation for?Data Engineers, Software Architects, Big Data & Astronomy enthusiasts
The slew of upcoming large-scale astronomical surveys promises exciting times for both astronomy and computer science. One of the most important future surveys is Large Scale Survey Telescope, or LSST. Its unique design and excellent location allow it to go both “wide” and “deep”, at the same time covering large regions of the sky and obtaining images of the faintest objects. LSST will produce one 3.2 giga-pixel image every 20 seconds and will do that every night for 10 years. That will result in the first “video” of the deep sky in history and (according to some estimates) about 80 PB of data. Furthermore, worldwide scientific community will receive real-time alerts triggered by changes in the sky, within 60 seconds from their detection.
In this talk I will describe how LSST image processing pipeline uses acquired images to produce catalogs of astronomical objects. Together with colleagues from University of Washington, I built AXS (Astronomy Extensions for Spark), a system for processing and quickly cross-matching catalog data, based on Apache Spark. I will explain its architecture and what is behind its great performance.
Prerequisite knowledgeBasics of distributed data processing and SQL
What you'll learn
SV Group d.o.o.
Petar started out as a Java developer almost 20 years ago, and worked as a Software Architect, Team Leader and IBM software consultant. After switching to the exciting new field of Big Data technologies, he wrote the Spark in Action book (Manning 2016) and these days primarily works on Apache Spark and Big Data projects. Today he is CTO of SV Group in Zagreb, Croatia, while also pursuing his PhD at the University of Zagreb. He is collaborating with Astronomy Department at the University of Washington on building new methods for processing images and data from future astronomical surveys.
Leave a Comment or Question
Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?
Join the conversation here (requires login)
For conference registration information and customer service
For more information on community discounts and trade opportunities with O’Reilly conferences
For information on exhibiting or sponsoring a conference
View a complete list of Strata Data Conference contacts