Build resilient systems at scale
28–30 October 2015 • Amsterdam, The Netherlands

Service instrumentation, monitoring, and alerting with Prometheus

Björn Rabenstein (SoundCloud), Julius Volz (SoundCloud)
14:00–15:30 Wednesday, 28/10/2015
Tutorial
Location: Auditorium
Average rating: ***..
(3.59, 17 ratings)
Slides:   1-PDF 

Prerequisite Knowledge

  • Basic knowledge of monitoring and alerting
  • Experience with distributed services
  • Programming skills in Python or Go

Materials or downloads needed in advance

  • Familiarize yourself with Prometheus. A good starter is the SoundCloud blog post.
  • Next, you should look at the Week of Monitoring blog posts by Boxever.
  • Finally, you find comprehensive documentation at Prometheus.io
  • Bring a Linux or Mac laptop. Optionally, have Linux running in a VM on a Mac or on any other platform. Install the most recent binaries for your platform:
  • The Prometheus server

    The Node Exporter. If you have Linux running in a VM, install the node exporter there (the node exporter works much better on Linux).

  • An example application to be instrumented will be offered in Go and Python. If you prefer Python, make sure you have a working Python development environment on your laptop and install the Python client library by running "pip install prometheus_client"
  • If you prefer Go, make sure you have a working Go1.4.x development on your machine and clone https://github.com/prometheus/client_golang into $GOPATH/src/github.com/prometheus/client_golang.
  • Make sure the following works:
  • cd $GOPATH/src/github.com/prometheus/client_golang/examples/simple

    go get

    go build

Description

Running a multitude of highly scalable services in large clusters provides a challenge to monitoring. Prometheus is a next-generation monitoring system built to cope with that challenge. Over the last three years, it has been developed as an open-source project at SoundCloud, where it has become the standard monitoring system. Other early adopters and contributors are Boxever and Docker. Since its wider announcement in January 2015, the project has rapidly gained attention, including support by third-party tools like Google’s cAdvisor or CoreOS’s etcd.

This tutorial will start with an introduction into the fundamental concepts of Prometheus and the various components of its ecosystem:

  1. The core collection server with its time series database
  2. The various client libraries
  3. The various exporters to export metrics from third-party systems into the Prometheus ecosystem
  4. The alerting component Alertmanager
  5. The dashboard builder Promdash
  6. The Pushgateway for metrics of short-lived jobs

The introduction is followed by a hands-on workshop, where participants will instrument the code of a toy service and create a dashboard and alerts for the service and the hosts it is running on.

Photo of Björn Rabenstein

Björn Rabenstein

SoundCloud

Björn Rabenstein is a production engineer at SoundCloud and one of the main Prometheus developers. Previously, Björn was a site reliability engineer at Google and a number cruncher for science.

Photo of Julius Volz

Julius Volz

SoundCloud

Julius Volz is a production engineer at SoundCloud and co-founder of the Prometheus project. In the past, he worked as a site reliability engineer in Google’s production offline storage team to back up the internet and more.

Comments on this page are now closed.

Comments

Picture of Björn Rabenstein
Björn Rabenstein
30/10/2015 23:35 CET

Thanks everybody for the huge turn-out. Here are the slides and the instructions .

Aleksei Markov
27/10/2015 14:51 CET

Wow, so many options!
Thank you for clarification,
see you tomorrow!

Picture of Björn Rabenstein
Björn Rabenstein
27/10/2015 12:46 CET

OK. Weird that it doesn’t work on CentOS in VirtualBox. The binaries are statically linked and have minimal requirements for the system they run on.

Pairing during the tutorial is definitely a viable Plan B.

Plan C would be to try out the native MS-Windows binaries we have for 0.16.0 https://github.com/prometheus/prometheus/releases/tag/0.16.0 .

Plan D is to just follow along during the tutorial. Due to the large number of participants and the setup of the room, we’ll also demo everything via the projection for those that cannot work on their laptops.

Aleksei Markov
27/10/2015 12:38 CET

I will try to find time to download another image of linux (like Ubuntu).
I hope we will have ‘plan B’ in case I didn’t make it work (like spare laptops with installed linux, or I will just pair with somebody)

Aleksei Markov
27/10/2015 12:32 CET

Hi, thank you for the answer.
I unpacked prometheus-*.tar.gz files, of course, and I try to launch "prometheus "binary file (on the same level there are promtool binary file and two directories consoles and console_libraries). And that is where I get an error “can not execute binary file”.
I tried the same on my working machine (redHat enterprise 6.6 x86_64) and the binary works fine.
So there is definitely a problem with incompatibility of the system I have in my VirtualBox on laptop and prometheus.

Picture of Björn Rabenstein
Björn Rabenstein
26/10/2015 1:13 CET

Hi Aleksei,

It seems you have forgotten to unpack the tar-balls. But you don’t have to do that right now. Just make sure you have the downloaded *.tar.gz files available in your virtualBox on Wednesday. Unpacking them will be part of the workshop.

Your Go installation is probably fine.

Aleksei Markov
26/10/2015 0:39 CET

hi, I am having troubles with installing prometheus.
I have windows7 laptop with virtualBox vm centOS7 (uname -r : 3.10.0-123.9.2.el7.×86-64). I downloaded both prometheus-0.16.1.linux-amd64.tar.gz and prometheus-0.16.1.linux-386.tar.gz – neither of them runs (error: cannot execute binary file)
I tried to build prometheus from sources, but on “make” step I receive error import cycle not allowed …. package github.com/prometheus/prometheus/cmd/prometheus… probably go1.5 not properly installed, however go version works (didn’t try helloWorld though).
I spent already half of my Sunday on it and a bit frustrated.
Do you have by chance working image for VirtualBox with a system with all preinstalled software? If you had one – all participants will need just download image and run it – that’s it.
And currently I am a bit confused – I will not have time before Wednesday to clean everything and install everything from scratch.
Any ideas or advice will be very appriciated. Thank you!

Picture of Björn Rabenstein
Björn Rabenstein
31/08/2015 19:20 CEST

Prometheus is a fast moving project. It probably makes sense to perform (or update) the installs mentioned above just a week or so before the conference start.