Update: We’ve given away the results of this work to the community for free.
As a platform for working with data, we’ve seen users tackle lots of interesting use-cases: log analysis, natural language processing, pattern detection, and many more.
However, perhaps no use-case is in greater demand than recommender systems. If you have more “inventory” than your users can easily find (whether it’s news, jobs, videos, restaurants, vacations, recipes, apps, etc.), a great recommender is crucial to driving engagement.
The problem is that recommender systems are really hard to implement, so most companies either don’t have one or aren’t happy with what they have.
What makes recommenders so tough?
- It’s painful to Obtain, Scrub, and Explore. These systems often require a lot of data, and the “OSE” steps from “A Taxonomy of Data Science” can take 75% of the effort in building a recommender system.
- Processing data at scale is difficult (at best). Those who have ever attempted setting up and maintaining a Hadoop cluster know how challenging it can be.
- Data scientists are nearly impossible to find. Looking at every user and everything that could be recommended to them typically requires a very good data scientist. Unfortunately, there is a massive data scientist shortage, and the problem is only getting worse.
Although our users were finding Mortar hugely helpful for OSE and processing their data at scale, they still found they needed extra help to build the recommenders they wanted due to the scarcity of data science talent available to them.
To remedy this problem, we’re custom-building recommenders for 10 companies by partnering with
three four world-class data scientists:
- Hilary Mason – Chief Scientist at Bitly
- Drew Conway – Author of Machine Learning for Hackers, Scientist in Residence at IA Ventures and Co-founder of DataKind
- Max Shron – Data strategy consultant (clients include Amazon.com, the Guardian, and Warby Parker), formerly lead data scientist at OkCupid
- Eric Colson [added 6/20/13] – Former Netflix VP of Data Science & Engineering, now Chief Algorithms & Analytics Officer at Stitch Fix
We’re going to select 10 companies to work with, building each of them a custom recommender system created with open technologies (Pig, Python, Java), for free. We are open sourcing the generic components and giving the custom pieces to their respective companies to keep.
Why would we do this?
We want to provide future users reusable, open source components that work on real problems, at scale. So while 10 companies will get custom-built solutions, all of our customers benefit by being able to more easily build recommenders on Mortar.
We’ve already accepted two companies to our program during our beta rollout, and now we’re looking for eight more. Want a free recommender custom-built for you? Enter your email address here.