Presented By
O’Reilly + Cloudera
Make Data Work
March 25-28, 2019
San Francisco, CA

Enabling insights and analytics with data streaming architectures and pipelines using Kafka and Hadoop

11:50am12:30pm Wednesday, March 27, 2019
Average rating: ****.
(4.60, 5 ratings)

Who is this presentation for?

  • Engineers and architects

Level

Intermediate

Prerequisite knowledge

  • Familiarity with Hadoop and Kafka

What you'll learn

  • Learn the importance of streaming data architectures

Description

In a large global health services company, streaming data for processing and sharing comes with its own challenges. Data science and analytics platforms need data fast, from relevant sources, to act on this data quickly and share the insights with consumers with the same speed and urgency.

Join Mohammad Quraishi to learn why streaming data architectures are a necessity—Kafka and Hadoop are key. Mohammad outlines architectures centered around the Hadoop Platform and Kafka that were implemented to support a variety of integration and analytics requirements.

Topics include:

  • Enabling streaming to and from relational sources and files using custom frameworks that automate and speed up workflows
  • Combining the polyglot techniques with Kafka API to support various streaming solutions
  • Combining data driven techniques to support consumers through a simple streaming architecture and microservices
  • How HBase, Kudu, and Kafka Streams are used to reduce latency between these microservices and frontend application APIs
  • Enabling the consumption and sharing of data sources and results using streams
  • Enabling Spark Structured Streaming, Flink, and Spark ML on these streams
  • Enabling data sync between on-premises data lakes and the cloud
  • Supporting cloud native architectures that enable machine learning in the cloud
Photo of Mohammad Quraishi

Mohammad Quraishi

Cigna

Mohammad Quraishi is a senior principal technologist within the Data and Analytics Organization at Cigna, where he’s the lead engineer in the Big Data Guild; his primary focus is on Hadoop and streaming architectures. Mohammad has worked in the healthcare industry for 23 years. He holds a BS in computer science and engineering from the University of Connecticut at Storrs.