Presented By
O’Reilly + Cloudera
Make Data Work
March 25-28, 2019
San Francisco, CA
James Taylor

James Taylor
Software Engineer, Lyft

@JamesPlusPlus

James Taylor is a software engineer in the Data Infrastructure Group at Lyft, where he works on big data systems. Previously, he was an architect at Salesforce, where he founded the Apache Phoenix project and led its development, and worked on federated query processing systems and event-driven programming platforms at BEA Systems.

Sessions

2:40pm–3:20pm Wednesday, March 27, 2019
Adaptive ETL to optimize query performance at Lyft
Data Engineering & Architecture
 Location: 2001
Secondary topics:  Data Integration and Data Pipelines, Transportation and Logistics
James Taylor (Lyft)
Average rating: ***..
(3.56, 9 ratings)
James Taylor offers an overview of an automated feedback loop at Lyft to adapt ETL based on the aggregate cost of queries run across the cluster. He also discusses future work to enhance the system through the use of materialized views to reduce the number of ad hoc joins and sorting performed by the most expensive queries by transparently rewriting queries when possible. Read more.