Optimizing MySQL for High Volume Data Logging Applications

Architecture and Technology, Java, Replication and Scale-Out
Location: Ballroom D Level: Intermediate

Hyperic is an open source provider of enterprise systems management software, specializing in collecting tens of thousands of metrics per minute and performing complex event and performance analysis in real time.

Last year, due to popular demand, the Hyperic engineering team undertook the task of adding support for MySQL as a backend database for Hyperic’s flagship product, Hyperic HQ. In order to meet the massive throughput requirements with a minimum of overhead, the development team re-architected the database schema, optimized queries, and utilized several MySQL-specific features.

This session is intended to share Hyperic’s lessons learned and best practices with those who are developing Java applications and have similar transactional and storage requirements and need to optimize support for MySQL to perform at such demanding scale. Attendees will also learn about a Hyperic customer who benefited from a MySQL-backed HQ. This use case will focus on best practices for using MySQL as HQ’s database.

The promise of abstraction layers in MySQL for developers was that database-independent coding would be easier and more automated. Unfortunately, this promise is not yet realized, as any move towards a database-independent codebase still requires monumental effort and dedication to performance enhancement and debugging.

This talk will focus on the significant changes Hyperic’s developers made to to HQ’s underlying database code in order to add support for a MySQL backend, and will provide tips for other developers engaged in similar projects. This includes its treatment of indices which differs greatly from PostgreSQL and Oracle (HQ’s other supported backend databases). Audience members will learn the following:

1. The challenge faced by HQ developers

2. First task – Hibernate

3. 2nd task – Adding MySQL support

3a. Engineering around some challenges

4. The last-mile problem – the final product of re-engineering

4a. A faster, better database backend

5. Real-life use case from a customer with a large MySQL scale-out

Conclusion: MySQL can perform very well as a database backend for enterprise-class, data-intensive software.

Charles Lee

Hyperic, Inc.

Charles Lee is co-founder and vice-president of engineering of Hyperic. Prior to co-founding Hyperic, Lee was a senior software engineer at Covalent. There, he built Covalent’s configuration management product for Apache (CMP), and he spearheaded and architected the application management software (CAM). Before Covalent, Lee developed a document management system for retail store build-outs based on open-source technology at WiseConnect. Lee also held senior engineering position at Hewlett-Packard, where he was instrumental in developing print drivers for network LaserJets for the Asian market, as well as developing the UI framework used for LaserJets for all markets. Lee also developed the first GUI printer configuration framework for AutoCAD while a senior engineer at Autodesk. Lee was an early engineer at Backflip, where he created the document publishing system for the website based on mod_perl.

Lee received his BS in Computer Science and BA in Chemistry with honors from the University of Washington.

  • Kickfire
  • Zmanda, Inc.
  • Continuent
  • EDS
  • JasperSoft
  • Sun Microsystems
  • Symantec Anti-Virus Software
  • XAware
  • Data Direct Technologies
  • Dolphin Interconnect Solutions
  • Hewlett Packard
  • Infobright, Inc
  • Linagora
  • Microsoft
  • OpSource
  • Oracle
  • Pentaho
  • R1Soft
  • Red Hat
  • Ticketmaster
  • TechRepublic

Contact Us

View a complete list of MySQL contacts.