Mining for Insight Within Two Billion Visits

Running a globally popular virtual world with over 10 million monthly users (Oct 2008) in more than 30 countries and 12 languages is one thing — sharding, scale-out, massive caching, and dozens of database servers can do that.

Collecting all that data back to one place and analysing what do users enjoy doing, figuring out how to improve the service, keeping the in-world economy in balance, and providing next-day business results back to the management is another. We’ve tried a few things over the years — in-house reporting tools, reporting “in the cloud” with tools like Google Analytics, data mining, etc. All have their uses, as well as their weaknesses. MySQL’s strong point hasn’t really been the capability to deal with datasets in the millions to billions of rows, and data warehousing solutions built on the technology that has been so familiar to us has had it’s issues.

However, with the introduction to Infobright’s Brighthouse storage engine, we gained control of the process again. Loading speeds increased enough to not just keep up with the torrent of new data, but to catch up to with everything else waiting for analysis, and to provide a view across the entire history of the business. Query speeds for ad-hoc analysis increased even further — and suddenly we don’t have to try to anticipate every possible future requirement!

I will describe what our main issues were prior to the implementation, overall steps through the deployment, results and a few things we have in mind as next steps.