How to avoid drowning in logs: Streaming 80 billion events and batch processing 40 TB/hour (sponsored by Pure Storage)

Description

Continuous integration (CI) pipelines generate massive amounts of messy log data. Pure Storage engineering runs over 70,000 tests per day creating a large triage problem that would require at least 20 triage engineers. Instead, Spark’s flexible computing platform allows the company to write a single application for both streaming and batch jobs so that a team of only three triage engineers can understand the state of the company’s CI pipeline. Spark indexes log data for real-time reporting (streaming), uses machine learning for performance modeling and prediction (batch job), and reindexes old data for newly encoded patterns (batch job). Ivan Jibaja discusses the use case for big data analytics technologies, the architecture of the solution, and lessons learned.

This session is sponsored by Pure Storage.