Uncovering Web Performance Trends with HTTP Archive and Google Cloud Dataflow
HTTP Archive crawls over 300K sites and archives millions of response bodies that are used to construct the page. In this lightning demo we’ll take a look at the performance trends and insights hidden in this data with the help of Google Cloud Dataflow and BigQuery.
Webpagetest-automation 2.0 – OpenSpeedMonitor
Nils Kuhn and Uwe Beßle
WebPagetest is a great tool for ad hoc measurement and analysis of single web-page performance. Our goal however was a continuous monitoring of web-applications. With WebPagetest-Monitor there is an extension to WebPagetest for this purpose. Unfortunately WebPagetest-Monitor had it’s last release on 2010/12/20 with version 0.3.0. Many new features and metrics from WebPagetest are not supported by the old WebPagetest-Monitor. Therefore we developed a new tool to leverage private webpagetest-infrastructures: OpenSpeedMonitor. In providing the following features it perfectly fits the need for automation of webpagetest and trending its results.
Measurement-Jobs can be scheduled with cron-like expressions. Thus scheduling is much more flexible than configuration by frequency. You can schedule jobs to only run over the day (omit the nightly maintenance window) or on weekends.
Every setting of WebPagetest can be configured per job within OpenSpeedMonitor (bandwidth characteristics, capturing a tcpdump, emulate mobile browser etc.). Every result WebPagetest provides OpenSpeedMoonitor visualizes in trends over time. Correlations between different kind of measurands, like document complete-time and the bytes loaded until that time, can be represented by a diagram. Additionally, all measured results can be sent to Graphite, to include page performance measurements in central monitoring from OPS.
The individual pages of a web application typically share some static content. Other content is page-specific. Each web-application has its typical use cases, with sequences of pages typically visited by customers in the same order. Thus, the pageload-behaviour users normally experience differs significantly from measured page load times with both WebPagetests first view as well as the repeated view.
WebPagetest can handle scripts with multiple steps, in order to measure page load times correctly. But the handling of scripts with WebPagetest is cumbersome. With WebPagetest only one measurement can be made per job, containing either the whole script as one measurement or only a certain part of the script. This leads to very inefficient WebPagetest measurements executing scripts with ten steps and only measuring the last page load. Furthermore there is no assistance for editing scripts with the scripting API. In order to measure the impact of the browser cache in correct state we adapted WebPagetest to save and visualize separate results for each single page in the sequence of a script.
OpenSpeedMonitor is free, open source and will be available on github soon.
Ilya Grigorik is a web performance engineer and developer advocate at Google, where his focus is on making the web fast and driving adoption of performance best practices.
Nils was born in 1977 in Hamburg city in Germany. From 1999 to 2005 he studied biology in Hamburg and finished with master degree.
From end of study to first years of work he moved from ecology and geoinformatics to software- and database-development. He worked as a software-developer in many different projects with both, web- and desktop-technologies.
Since 2011 he is senior software engineer at iteratec with a focus on web-applications and the monitoring and optimization of web-performance.
Uwe was born 1965 in Berlin. He studied in Budapest from 1983 to 1988 and finished with a master degree in electrical engineering.
From 1988 until now he worked for different software companies in different roles.
Since 2003 he is senior IT consultant at iteratec with a focus on IT architecture and non-functional requirements.
For exhibition and sponsorship opportunities, contact Gloria Lombardo at email@example.com
For information on trade opportunities with O'Reilly conferences contact firstname.lastname@example.org
For media-related inquiries, contact Maureen Jennings at email@example.com
View a complete list of Velocity contacts