More than a century ago, the department store magnate John Wanamaker famously complained about his inability to gauge the effectiveness of the money he spent on advertising. Since then, technologies such as radio, television, and the internet have given companies new venues for self-promotion, but the age-old problem persists: How to tell whether ad dollars are really boosting sales?
That question is one factor driving firms to shift ad money to digital media. Not only are people spending more time online, but advertisers believe that companies such as Facebook and Google, which track people’s online habits, can put the right ads in front of the people most likely to buy (and the companies can measure what results). According to data from Accenture, digital media now account for 41% of large companies’ ad spending, and forecasters expect the amount to exceed 50% by 2018.
But the issue of effectiveness nags here, too. Although most advertisers have come to believe that ads delivered when a customer is searching specific terms are more effective than the static banner ads that once dominated the web, recent research has cast doubt on that. A 2015 study found that when eBay started and then stopped advertising on a large search engine, the company saw no difference in traffic. “That paper brought into question whether these kinds of ads do anything or not,” says Michael Luca, an assistant professor at Harvard Business School. A subsequent study found that some advertisers are decreasing their spending on search ads.
The studies piqued Luca’s curiosity. Since graduate school he’s been interested in how data, rankings, and reviews influence consumer behavior. Over the past five years he has published papers on the dynamics of college rankings and book reviews. He’s also conducted several studies of Yelp, including a widely publicized paper concluding that 16% of the restaurant reviews he examined were fake. As Luca’s research began appearing, Yelp reached out to discuss how it could work with academics on a range of research questions. As a result of those conversations, during the summer of 2015 Luca and his colleague Daisy Dai (now a professor at Lehigh University) moved into cubicles at Yelp headquarters.
The question they sought to answer goes to the core of Yelp’s business model: Do the ads that Yelp sells to small businesses, which give those firms’ listings prime position atop search results, deliver more customers? To answer it, the researchers designed a series of rigorous experiments and obtained Yelp’s agreement to allow them to publish the findings no matter what the experiments revealed.
Luca and Dai created a randomized sample of 18,295 U.S. restaurants, selected 7,210 that had never advertised on Yelp, and designed free ad packages for each one in that group. (The restaurants weren’t told about the ads or the experiment.) For the next three months they closely tracked user engagement with all the restaurants. Then they took the ads down to see what would happen.
They found that while the ads were up, the restaurants in them got more page views than the others—22% more on desktop browsers, 30% more on mobile devices, and 25% more overall. Users requested directions to them 18% more often, made 13% more calls to them, and clicked through to their websites 9% more often. The differences disappeared as soon as the ads were taken down. “This was a big effect,” Luca says. “It looks like Yelp ads are a positive investment, even for a business that doesn’t ordinarily advertise. The value Yelp ads seem to provide is in surfacing brands to customers.”
What if the study had shown Yelp ads to be worthless? Luca says that although those results would have been damaging to Yelp’s current strategy, they would have uncovered a need for the firm to focus more on alternative revenue models. “Platforms have to decide how to make money and what they can do to help customers who are using them,” he says. “If ads weren’t working for Yelp, maybe it would put more emphasis on charging companies to facilitate transactions or selling analytics packages.”
Luca and Dai’s findings contrast with the results involving eBay, but Luca sees an important difference between the recent study and the earlier one. EBay is a well-known brand whose name people are likely to type into a search engine; it makes sense that touting something consumers are already searching for would have little effect. Many Yelp advertisers are local businesses that few people have heard of; for unfamiliar brands like these, ads that propel them to the top of a list and create awareness can pay off.
This isn’t to say that big brands should never invest in search advertising, Luca adds—but they should bear in mind that search ads work best when they alert consumers to something they’re not already aware of. For instance, Gap might forgo search ads that would pop up when users search the company’s name or its best-known categories, such as jeans, and instead pay to appear in results for categories it isn’t commonly associated with, such as shoes. “Bigger brands should use search ads to promote things about the brand that people wouldn’t otherwise discover,” Luca says.