There was a fascinating, well shared Quartz piece the other week that described how Spotify's algorithms curate their brilliant Discover Weekly playlists which have become uncannily good at recommending great music that we haven't yet heard. Spotify uses a combination of the composition of millions of user's playlists alongside a categorisation of tracks into nuanced genres to help build a 'taste profile' for each user and surface tracks that they haven't yet heard. It made for a compelling read.
More recently I came across some interesting insight into how Amazon's famous recommendation algorithm actually works in this paper (PDF) from Greg Linden, Brent Smith and Jeremy York who all work there. Amazon's algorithm enables an unprecedented level of content personalisation served up at lightning speed, contributing billions of dollars through enhanced click through and conversion rates that vastly out-perform averages from banner ads or top-seller lists.
The most common approaches to algorithmic recommendation are collaborative filtering, cluster models, and search-based methods.
An algorithm based on traditional collaborative filtering works by looking at the purchase and/or ratings history of the user and generates recommendations based on a few customers who are most similar to the user, disregarding any items in the set that they may have already purchased.
Cluster models work by dividing the customer base into a number of segments and then classifying users into segments that contain the most similar customers. The purchase and ratings history of the users in the segment are then used to generate recommendations.
Search-based models build keyword, category, and author indexes but don't work well in recommendation at scale when you have customers with large numbers of purchases and/or ratings.
The challenge with these approaches is that they are 'computationally expensive'. Amazon has over 270 million monthly active users worldwide who generate huge amounts of customer data. Some of those customers will have built up a purchase history of thousands of items over time. Combine that with a massive product catalogue and you run into real scalability issues. As the number of potential combinations quickly scales to the point where it becomes immense and unwieldy the speed of decision-making and performance of the algorithm is impacted. Using methods to be selective about the number of inputs and so reduce the data set often simply reduces the quality of output. Add in a significant degree of volatility in customer preferences and the fact that new users have very little data on which to base decisions and these methods fall short.
So Amazon developed a process called 'item to item collaborative filtering'. This involves making recommendations on the product level rather than the user level. The algorithm works to recognise products that are generally bought together. Rather than simplistically rely on looking at items that might obviously go together, it looks at similarities between one specific product and all related products and finds items that customers tend to purchase together. So for each of the user's purchased items, it selects and aggregates similar products and then forms a recommendations list. Add to this the customer's ability to change their preferences manually and you have something that scales to massive data sets without sacrificing speed or accuracy. The algorithm can react immediately to changes in customer data, generate high quality recommendations at huge scale for all users regardless of their number of purchases or ratings.
What's interesting about this I think is that both Spotify and Amazon take inputs from both item combinations (for Spotify the tracks that people combine into playlists, for Amazon the product combinations that people buy) and a user's 'taste profile' (in Spotify's case categorising the genres of tracks you've played, in Amazon's case using the products you've bought). Marketing automation and personalisation are really hot topics right now. It's going to be really interesting to see how these kinds of more three dimensional recommendation algorithms might have broader application.
