Schedule: Building Data-Intensive Applications sessions

Join Tim Berglund to learn how to produce and consume a Kafka topic, integrate Kafka with a database using Kafka Connect, and perform real-time stream processing on Kafka data. Read more.

Today’s world generates different kinds of data at unbelievably rapid rates. Grishma Jena explains data science, the data science pipeline, and algorithms using real-life examples. Read more.

Fires caused by electric grid failures are increasing at an alarming rate. Michael Enescu and Peter Enescu examine how the energy adaptive networks technology built on open source and used to monitor and control power grids forms a planetary skin that can be used to predict and avoid such disasters as the Napa and Paradise Fires. Read more.

Applications from social media to healthcare to media are increasingly focused on humans and their relationships. But people do not lend themselves easily to being reduced to small slots. Christophe Pettus draws on his experience in data modeling and application design to examine how to successfully approach modeling humans and their relationships and legal compliance issues in a GDPR world. Read more.

Streaming analytics is a popular subject in enterprise organizations because customers want real-time experiences, such as notifications and advice based on online behavior and other users’ actions. Bas Geerdink details an open source reference solution for streaming analytics that covers many use cases that follow a "pipes and filters" pattern, built with Scala, Flink, Kafka, and Cassandra. Read more.

When designing APIs such as the new GCP Firestore real-time database and Google Assistant, how did Google decide which trade-offs to make? Wenbo Zhu dives deep into the challenges faced while deploying a real-time streaming API designed for clients from data centers to the internet and details the trade-offs API developers need be aware of when designing such an API. Read more.

Cassandra is one of the most popular datastores in big data and ML applications. Data analysis at scale with fast query response is critical for business needs, and while Cassandra with Spark integration allows running an analytical workload, it can be slow. Shradha Ambekar dives into the challenges faced at Intuit and the solutions her team implemented to improve performance by 100x. Read more.

Graphs provide a method to store and analyze the relationships within the data. Algorithms deepen our understanding of data through aggregation and perspectives to help developers make valuable business decisions for the future based on existing scenarios. Amy Hodler and Mark Needham lead you through a crash course in how to use graph algorithms as part of your big data toolkit. Read more.

Managing large stateful applications is tough. Matt Schallert outlines the challenges of automating stateful systems at scale and details how embracing a declarative approach can ease operation and automation of these systems on orchestrators such as Kubernetes. He then demonstrates how to apply this methodology to different types of stateful workloads. Read more.

Data-intensive applications, with many layers of transformations and movement from different data sources, can often be challenging to maintain and iterate even after they are initially built and validated. Jiaqi Liu explores how to factor in monitoring, alerting, and tracing data lineage when building data applications that move and transform data across multiple dependencies. Read more.

Alex Silva outlines lessons learned, common pitfalls, and design traps when designing your streaming data infrastructure, and he shares 21 best practices and standards used at Pluralsight. Read more.

With so many moving parts, it's hard for the average database administrator (DBA) or database developer to come up with a good explanation for why the optimizer chooses certain query plans. Vicențiu Ciorbaru dives deep into how a modern database query optimizer works to optimize your queries and how you can help it work for you. Read more.