The Curve | An online learning blog for professionals, from MIT

Investment and Advancement is Key: MIT Professor Devavrat Shah on the Recommendation Engine Industry of the Future

Written by MIT xPRO | Nov 2, 2020 1:30:00 PM

The market for recommendation engines is predicted to grow by USD $3.57B in the next four years, according to a recent report from Technavio. What’s the story behind these growth numbers? We talked with MIT Professor of Electrical Engineering & Computer Science and co-founder of machine learning startup Celect, Devavrat Shah, about the findings from this recent report and to discuss his predictions for recommendation systems technology.

We’re all familiar with Amazon recommendations or the carousel of curated shows and movies that appear on your Netflix profile. You have likely interacted with a recommendation engine if you researched a school, searched for a new dentist, or visited a dating site. And there are a multitude of opportunities beyond consumer applications, for everything from government policy and supply chain to market dynamics and competitive landscape.

"What gets more meaningful results is offering options, like an optician with lenses. Which is better, this one or that one? And it’s not just important what you buy, but what you don’t buy.”

-- Professor Devavrat Shah

 

The implementation of artificial intelligence (AI) is one factor driving recommendation engine growth. Recommendation systems apply AI to analyze big data stores and present options tuned to buyer preferences. Shah observes that data acquisition is an art more than a science. It’s incredibly difficult to understand the intent and the interest of a person looking for something.

“How humans think is different from how machines think. How can you ask for information in a way that gets you what you need? A 1-10 rating scale often doesn’t work. What gets more meaningful results is offering options, like an optician with lenses. Which is better, this one or that one? And it’s not just important what you buy, but what you don’t buy.”

He says the industry needs more refined models that allow for real-time updates as information is learned. This requires investment and advancement of the technology.

 

An Industry Dominated By a Few Large Players

“Companies don’t build cloud servers from scratch, and they don’t build databases from scratch. Tech companies buy and resell them. Recommendations engines could be sold the same way, but first they need standardized interfaces. This is one of several interesting technology problems that needs to be solved.”

-- Professor Devevrat Shah

Today this industry is dominated by companies that have recommendation engines embedded within their larger systems like Amazon and Netflix. These players have all the influence.

“Selling of goods used to be through a bazaar,” says Shah. “Every seller had equal opportunity with the buyers, and vendors were able to build relationships with their customers. That’s completely gone today."

Online platforms like Amazon and Etsy dominate the access of consumers to small businesses. Finding ways to get this technology into the hands of more companies would democratize the process.”

Several recommendation engine companies came on the scene in the early 2000s, and still operate today. Yet they are not growing to the degree the industry had expected. With such limited success, VCs aren’t investing in the next wave of recommendation systems technology.

Shah sees this as a missed opportunity. He believes these smaller players have the opportunity to advance by packaging their technology for purchase by many companies, similar to how Intel, Oracle, and many others have sold their technology. 

“Companies don’t build cloud servers from scratch, and they don’t build databases from scratch,” observes Shah. “Tech companies buy and resell them. Recommendations engines could be sold the same way, but first they need standardized interfaces. This is one of several interesting technology problems that needs to be solved.”

 

Recommendation Technology Challenges to Solve

What does Shah predict as critical for the advancement of recommendation systems in the coming decade? He expects and hopes to see recommendations engines:

  • interact with people to narrow down too many options. Similar to the way a sales clerk would help an in-person customer, the search engine would ask a series of questions that build on each other to narrow down the options and flag the three best pairs of jeans for you.
  • that take into account social and ethical concerns. This includes reducing bias, considering the context of different audiences, and making recommendations that are appropriate to the moment and the individual. It also includes careful use and protection of personal data. 
  • able to analyze and match from both sides of an interaction in a meaningful way. Matching an Uber driver with a rider is one example. Another is analyzing a buyer’s past purchase patterns and demographic indicators together with a company’s inventory and supply chain before making a recommendation. 
  • designed to require and solicit meaningful, honest feedback from an individual. This step would inform future recommendations, but is often missed entirely. Today the engines are working with you ahead of a purchase, but rarely check back to see if what you bought really worked for you.

In 2016 “The Netflix Prize” invited coders around the world to improve the company’s recommendation engine by 10%. While this challenge generated a lot of buzz and yielded results, Shah thinks it has done the industry a disservice in the long run.

“It was a good start, but people took the results as the end product,” he says. “That work should be viewed as just a beginning. Rather than stylized solutions, we need to design systems that are closer to real-world scenarios and will yield better results.” 

“The recommendation engine sector has huge growth potential,” concludes Shah. “Intellectually, recommendation systems provide a terrific sandbox to bring together machine learning and social sciences -- it's what I would call the canonical problem for ‘social data processing’ where it is essential to utilize social sciences to understand the appropriate models, while using machine learning, statistics, and AI to come up with better algorithms. There are technological and business challenges data scientists can step up to solve, and it’s worth investors taking on the risk.”

Want more insights from Professor Shah and MIT xPRO? Learn to create your own recommendation engine. Download your free data science case study here.