One single sentence describes personalization in e-commerce: Everyone talks about it, everyone claims to have it, and it rarely works.

E-commerce Web sites use recommendation algorithms to personalize the experience of each customer. But most Web sites end up recommending bestsellers or category bestsellers. So if you buy a pair of sunglasses, you are recommended five additional pairs of sunglasses.

The two main reasons most recommendation algorithms fail are Harry Potter and Paris Hilton.

Harry Potter

The Harry Potter effect is relatively well known, thanks to Amazon. The name was coined in the mid-2000s, when the Harry Potter books popped up as suggested items in the Amazon “customers who bought this also bought” widget, even for completely unrelated books and products.

Why?

The “customers who bought this also bought” widget is based on similarities. The recommendations are based on the items that the customer has previously purchased. The algorithm matches the user’s purchased items to similar items and then combines the similar items into a recommendation list. By definition, a very popular item is often purchased with other items, and therefore the popular item is suggested to be similar to several other items.

Enter Kanye

Kanye West spends so much money that he claims to be broken. He buys a great deal of merchandise; he is likely to consumed two items together. Compare his behavior to a casual shopper who buys just one item. This nonobvious problem makes Kanye West more dangerous than the little magician—and more subtle.

How Big Are These Problems?

The bigger the ,the more skewed the data. The Internet is much more skewed than the off-line world is. Anyone running an e-commerce Web site with thousands of customers knows that there is no such thing as an “average order” or “average customer spending.” Few customers spend a great deal of money; most customers spend a small amount of money.

No Easy Way Out

Easy solutions call for a normalization of the data, specifically for normalizing similarities for the number of times an item has been consumed. Some of these solutions are even worse than the original problem was. If we were to divide the similarity of a Harry Potter book by the number of books sold to create a relative measure of similarity, then an item with one single sale would end up being considered extremely similar to every item that customer consumed.

At Muse, we developed a proprietary system that not only solves both the Harry Potter and Kanye West effects but also avoids the grave dangers of commonly suggested “naïve” solutions.

Our item-to-item collaborative filtering algorithm scales independently from the number of items and users. And unlike traditional collaborative filtering, the algorithm generates high-quality recommendations, even with limited user data and even with as few as three or four items.