Presented By O’Reilly and Cloudera
Make Data Work
21–22 May 2018: Training
22–24 May 2018: Tutorials & Conference
London, UK

A heretical monitoring view: Using PostgreSQL to store Prometheus metrics and visualizing them in Grafana

Erik Nordström (Timescale)
12:05–12:45 Thursday, 24 May 2018
Data science and machine learning, Expo Hall, Streaming systems and real-time applications, Visualization and user experience
 Location: Expo Hall Level: Intermediate
Secondary topics:  Time Series and Graphs

Who is this presentation for?

  • DevOps engineers, SREs, software engineers, and software architects

What you'll learn

  • Explore a new monitoring stack—Grafana + Prometheus + TimescaleDB—that allows for metrics in Prometheus to be visualized in Grafana and stored in TimescaleDB, to be analyzed with more complex queries using full SQL
  • Learn about pg_prometheus, a custom Prometheus datatype, prometheus-postgresql-adapter, a remote storage adaptor for PostgreSQL, and the new PostgreSQL connector released by Grafana

Description

Grafana and Prometheus have become a popular duo for collecting, querying and graphing metrics, giving teams greater clarity on their operations. Until recently, the databases supported by Grafana have been NoSQL systems, offering SQL-like or custom query languages that were limited in scope (i.e., in comparison to ANSI SQL) and designed for specific data model and architectures in mind. One of these systems is Prometheus. While Prometheus has its own time series storage subsystem specifically for metrics monitoring, sometimes you need something more to allow you to ask more complex questions of your data. With the recent addition of the MySQL and PostgreSQL data sources, full SQL is now available to Grafana users, and more powerful systems are available as persistent backends for storing larger volumes of Prometheus metrics.

Erik Nordström explains how and why to use PostgreSQL as a Prometheus backend to support complex questions (and get a proper SQL interface). Along the way, Erik offers an overview of pg_prometheus, a custom Prometheus datatype, and prometheus-postgresql-adapter, a remote storage adaptor for PostgreSQL, and shares his experience with TimescaleDB, an open source time series database optimized for scalable data ingest and complex query performance, which enables PostgreSQL to scale for classic monitoring volumes.

Photo of Erik Nordström

Erik Nordström

Timescale

Erik Nordström is a senior software engineer at Timescale, where he focuses on both core database and infrastructure services. Previously, he worked on Spotify’s backend service infrastructure and was a postdoc and research scientist at Princeton, where he focused on networking and distributed systems, including a new end-host network stack for service-centric networking. Erik holds an MSc and PhD from Uppsala in Sweden.