Data Science with Hadoop at Opower

Hadoop: Case Studies, Gramercy Suite (NY Hilton)
Average rating: ***..
(3.56, 9 ratings)

Opower works with utility companies to provide engaging, relevant, and personalized content about home energy use to millions of households. We have found that simply providing data is not enough to change behavior; data are meaningful only to the extent that people can relate to it. Opower makes energy use data relatable and meaningful by using normative comparisons and personalized insights to help reduce energy use. This simple framework has enabled us to reduce consumer’s energy use by over one terawatt hour, or about 25% of the entire output of the US solar industry in 2011. Currently, Opower works with over 60 utilities domestically and internationally and houses energy data on over 30 million households. And our big data is becoming bigger — many households measure energy use monthly, but more and more homes have smart meters which keep energy usage reads at hourly or half hourly intervals.

This talk will discuss how we interact with the Hadoop ecosystem to transform raw data into contextualized information that drives behavior change. This starts with our Hadoop setup and how it has changed how we store data and think about problems. We then interface Hadoop with tools like R and Python to extract features, visualize, and validate data. On top of this setup, we have extended our set of addressable problems by utilizing curated crowdsourcing to classify patterns that humans can easily recognize. Lastly, we use statistical models and machine learning algorithms including nearest neighbors, self organizing maps, and regularized regressions to create models of characteristics that drive energy use.

The inputs and outputs of each step are stored in Hadoop for ease of use, transparency, and to facilitate collaboration. Throughout the talk I’ll give examples of data science and efficiency problems this infrastructure has solved and where we expect it to take us in the future.

Erik Shilts


OpowerAnalytics – Senior Quantitative Analyst

  • Provide custom insights into energy use
  • Design experiments
  • Develop models to measure, predict, and segment energy savings

Freddie MacMission Division – Quantitative Researcher

  • Determine the effects of policy changes on the market, regulatory goals, fair lending metrics, and affordability.
  • Analyze automated underwriting system’s fair lending impacts and ability to predict risk.

Johns HopkinsM.A. Applied Economics

Virginia TechB.A. Economics


Sponsorship Opportunities

For information on exhibition and sponsorship opportunities, contact Susan Stewart at

Media Partner Opportunities

For information on trade opportunities contact Kathy Yu at mediapartners

Press and Media

For media-related inquiries, contact Maureen Jennings at

Contact Us

View a complete list of Strata contacts.