Skip to main content
Make Data Work
Oct 15–17, 2014 • New York, NY

Super Simple Internet of Things Backend: Persistence Post Hadoop with Crate Data

Jodok Batlogg (CRATE Technology GmbH)
2:35pm–3:15pm Friday, 10/17/2014
Machine Data
Location: 1 E12/1 E13
Tags: iot
Average rating: *****
(5.00, 2 ratings)

Super Simple Internet of Things Backend: Persistence post Hadoop with Crate Data

In the connected home, modular systems with lots of different pluggable sensors connect people with their home and actively help to enhance everyday life. Among others, this is one of the causes for the explosive growth of data being stored and processed in the cloud. Sensors permanently measure temperature, humidity, and other parameters and continuously forward them to the Backend API. The backend has not only to deal with a constant parallel inflow of data, but even more important: to process lots of ad-hoc events being forwarded from sensors as they happen.

Unfortunately traditional infrastructure isn’t designed to handle this kind of workload:

  • Massive parallel inflow of small writes
  • Lots of read queries in parallel
  • Large number user generated queries (e.g. API for mobile phone apps) that need to be handled in sub-seconds
  • Advanced search capabilities (geo-spatial, histograms,…) over large datasets

In this talk we’ll describe how we solved this challenge with two horizontally scaling components – Crate Data (Persistence Layer) and Twitter Storm (Event processing). Crate Data is a developer friendly, distributed data store. It is Open Source and makes real time SQL queries on large data sets super simple.

We’ll cover all the pitfalls and learnings on our way to the productive system. The first scaling issues, the architectural decisions at the beginning, but also about the development process with test-driven development but also interesting code snippets and the outcome of the loadtests.

Photo of Jodok Batlogg

Jodok Batlogg

CRATE Technology GmbH

Jodok is Co-Founder and CEO of Crate and has a decade of experience with Big Data and Open Source. He successfully did a few startups and as CTO led the tech teams of big web companies, for example the by far biggest german-speaking website in 2010, StudiVZ. Jodok is an entertaining and experienced speaker at many venues, including high-profile conferences in USA and EU, such as Nokia World, PyCon, Berlin Buzzwords or Burda’s DLD event.

Jodok enjoys wide recognition for his vision and expertise for innovative approaches. He served for several years as director of the Plone & Zope foundations. Loves his skis, his chainsaw and is an adventurous eater.

With Crate Jodok opens a new chapter – the post-Hadoop area of realtime BigData.