Scaling Heavy Concurrent Writes In Real Time

Average rating: ****.
(4.00, 1 rating)

Flickr implemented a real-time collection of referrers to Photos, streams, sets and collections. Currently the data is gathered real time for every single user on Flickr. The data is then cooked and reported with a 24 hour lag for filtering out referrer SPAM. MySQL myISAM/INNODB, curl, Java are the only component used in the setup and it scales linearly. This talk is about building a model for capacity planning, scaling for triple the request rate, and scaling linearly for an intensive application.

  • Disk I/O is not an issue capacity is based off of Disk consumption rate.
  • Java is used to buffer the writes to INNODB and myISAM
  • The data is federated across many servers
  • The goal is to keep unlimited stats for the lifetime of the pro account
  • Transactions are used to keep the data insync among multiple table views.
  • Detail data is kept on a per day, week, month, year for every view class recorded
  • Distributed locks across the cluster using mySQL’s GET_LOCK

Dathan Pattishall


Former Database Architect of Various Inc, Friendster, Flickr and current Architect at Rockyou. I’m an expert at scaling all application teirs but with a special emphasis in data layout, millisecond data retrieval on multi-terabyte data stores all using mySQL.
I share my tips on

  • Kickfire
  • Zmanda, Inc.
  • Continuent
  • EDS
  • JasperSoft
  • Sun Microsystems
  • Symantec Anti-Virus Software
  • XAware
  • Data Direct Technologies
  • Dolphin Interconnect Solutions
  • Hewlett Packard
  • Infobright, Inc
  • Linagora
  • Microsoft
  • OpSource
  • Oracle
  • Pentaho
  • R1Soft
  • Red Hat
  • Ticketmaster
  • TechRepublic

Contact Us

View a complete list of MySQL contacts.