Engineering the Future of Software
April 10–11, 2016: Training
April 11–13, 2016: Conference
New York, NY

Designing a reactive data platform: Challenges, patterns, and antipatterns

Alex Silva (Pluralsight)
3:35pm–4:25pm Wednesday, 04/13/2016
Reactive and its variants
Location: Regent Parlor Level: Intermediate
Average rating: ****.
(4.00, 2 ratings)

Prerequisite knowledge

Attendees should have some familiarity with Scala and Java, reactive principles, RESTful and hypermedia concepts, message-driven protocols, and JSON.

Description

The last few years have seen a tremendous surge in data volume, along with an unparalleled explosion of toolsets and solutions aimed at extracting the most value from this deluge. Integrating these different technologies in a way that makes sense to the organization is a real challenge that has trampled many experienced engineering teams. Alex Silva discusses these challenges—their definition, mitigation, and potential solutions—and explains what makes a good design pattern (and what doesn’t) when architecting an integrated data platform.

Alex covers the key architectural decisions Pluralsight made as it moved from a blank slate to a fully reactive self-service platform that is able to fulfill several business use cases, ranging from data ingestion to analysis, at both real-time and batch scales. Pluralsight’s current implementation consists of discrete microservices running on top of the JVM, using a mixture of Scala, Akka, and Java. Alex shares Pluralsight’s event-driven, reactive design that leverages REST, Hypermedia, and several open source frameworks and platforms, including Spring, Hadoop, YARN, Kafka, Storm, and Spark.

Topics include:

  • The major architectural components and decisions of a data platform
  • Ways to abstract the heterogenous nature of a data ecosystem using services
  • Real-time and batch data pipelines
  • Designing event-driven protocols in Scala
  • Asynchronous communication between microservices
  • Akka and actor systems in general
  • Data-integration challenges
Photo of Alex Silva

Alex Silva

Pluralsight

Alex Silva is a chief data architect at Pluralsight, where he leads the development of the company’s data infrastructure and services. He has been instrumental in establishing Pluralsight’s data initiative by architecting a platform that is used to capture valuable insights on real-time video analytics while integrating several data sources within the business. Before joining Pluralsight, Alex was a principal data engineer at Rackspace, leading a team of developers responsible for building the company’s data initiative. Alex has built a reputation as a passionate and pragmatic data evangelist. He was a key asset in establishing Rackspace’s big data platform by helping architect a solution used to drive actionable insight on consumer behavior and product-usage trends. He has also designed several analytical models, APIs, and frameworks that enable Rackspace to deliver its award-winning Fanatical Support, including a computational linguistics library used to analyze and classify support chat logs.

Alex was also a principal software engineer at ESPN Emerging Technologies. In that role, he architected and developed a distributed application to help basketball operators collect play-by-play records for data warehousing and modeling purposes. Alex has also held several senior-level engineering positions at Walt Disney World Internet Group, Pentaho, OutStart, and Travelatro.com, all based in Orlando, Florida. Alex is Sun Certified as an Enterprise Architect for the J2EE platform and is also a web component developer and a Java 2 programmer. He earned a bachelor’s degree in molecular biology and an MBA from the University of Central Florida in Orlando. When Alex is not programming, you’ll probably catch him with an athletic bag on his shoulders. He is a little bit of a sports junkie, particularly a CrossFit addict, who has been known to create an epidemic of fitness recovery, smoking cessation, and weight loss around him.