Relevant and Real-Time: Building a bi-directional recommendation system for a massive online game

Simon Worgan (Jagex Ltd), Samuel Kerrien (RESEREC)
Data Science
Location: 113
Average rating: ***..
(3.29, 14 ratings)
Slides:   1-PDF 

Online recommendation systems are a well-known and common part of Data Science but turning complex in-game interactions into personalised real-time recommendations presents a whole new set of challenges. In this presentation we will detail how we developed and deployed a bi-directional event stream recommendation system in RuneScape, the world’s largest free-to-play massively multiplayer online game. By capturing a feature rich bi-directional relationship between player and content we were able to train different ‘flavours’ of recommendation, each of which is tailored towards a different business target. Through the careful delivery of these different ‘flavours’ we balance engagement, monetisation and enjoyment according to shifting business requirements. With over 200 million registered players, our system scales to deliver personalised in-game content at the right moment. This approach drives engagement with our content and produces a significant uplift in both the financial and ‘enjoyment’ metrics of the game.

With millions of players the multi-terabyte daily event stream generated by RuneScape presents massive value when these ‘raw’ events can be transformed into personalised in-game content recommendations. While developing this system we overcame a number of unique challenges, specifically:

  • How to tie complex behavioural patterns to multiple business outcomes, enabling the business to strike a balance between engagement, enjoyment and monetisation
  • How to infer player enjoyment and subsequent purchase behaviour from observed interactions. How to select recommendations that are both suitable and enjoyable.
  • How to deliver in-game content recommendations, at the right moment, in response to current in-game activity.
  • How to maintain scalability across millions of players and terabytes of daily log data.

We will answer these questions by detailing two key components, firstly, a big data processing pipeline that produces personalised recommendations and in-game trigger conditions for real-time delivery and, secondly, one of the world’s first commercial deployments of a feature based bi-directional recommendation model 1. Current recommendation approaches are constrained by a User-Item Vs. Item-User design decision, by comparison our in-game content recommendation system is bi-directional and will only recommend content to a player if both content-player and player-content dependencies are satisfied. This ensures that recommended content is both desirable and suitable, avoiding player frustration through the recommendation of desirable but unobtainable items. Furthermore, by characterising each player and item by their interaction frequencies we can link content interaction to subsequent behavioural outcomes, e.g. linking a certain piece of content to subsequent purchase behaviour or increase in game time.

This presentation will be relevant to three groups of people; those who are facing inherently bi-directional recommendation problems (e.g. dating websites), those who wish to extract value from a user’s ‘raw’ event sequences and those who wish to have the flexibility to produce recommendations that optimise multiple (potentially completing) business metrics.

References

1 Gorla, J 2014, A Bi-directional Unified Model for Information Retrieval. Ph.D. thesis, University College London.

Photo of Simon Worgan

Simon Worgan

Jagex Ltd

Simon Worgan is the Senior Data Scientist at Jagex Games Studio. In this role he has applied his machine learning expertise to a variety of challenges within the games industry, these include in-game sentiment analysis, behavioural player clustering and predictive modelling. With a Computer Science Ph.D. from the University of Southampton he has over 6 years’ experience in the real world application of Data Science.

Samuel Kerrien

RESEREC

Sam is a Data Engineer at Jagex, the largest independent games developer and publisher in the UK. Sam has a wealth of experience on leveraging the Hadoop stack to build business Intelligence and analytics data products.

Comments on this page are now closed.

Comments

Picture of Simon Worgan
Simon Worgan
24-11-2014 15:37 CET

Hi Cindy,

For the sequence mining we used PrefixSpan, the details of which can be found here:

https://www.cs.sfu.ca/~jpei/publications/span.pdf

All the best,
Simon

Samuel Kerrien
24-11-2014 14:28 CET

@Cindy Thanks for you interest !

Here is the URL of the other co-founder of RESEREC that created this algorithm, please get in touch with him: http://www0.cs.ucl.ac.uk/staff/J.Gorla/

As for the sequence mining, I will leave it to Simon to answer this question as this is a Jagex specific matter.

Picture of Cindy Lamm
Cindy Lamm
23-11-2014 22:25 CET

Hey there, I really liked you talk! I couldn’t catch you afterwards though but I do have 2 questions:
1) Samuel mentioned publicly available papers about this new recommendation model – which are those?
2) Which algorithm(s) did you use for the sequence mining to come up with recommendable entities for RuneScape?

Best,
Cindy