Note added May 21, 2009: Harvard Business published a slightly updated version of this post at http://blogs.harvardbusiness.org/now-new-next/2009/05/the-social-data-revolution.html
A previous post discussed how free communication has changed the world, including the expectations and work of individuals, business, and society. This post discusses how two data revolutions (the first about passively collected clicks on Amazon.com, the second about actively contributed data), and the ensuing change in consumer expectation make an astute, coherent data strategy critical.
The first data revolution came from the dream of collecting data from consumer decision-making. With the advent of the web, firms pondered whether it might worth saving the vast amounts of data that customers were generating through their clicks and searches. For consumers, there was no hiding-after all, there is no online equivalent of discreetly checking out a magazine while a bookstore employee is looking the other way. Amazon.com has pretty much saved all user data from its beginning.Back then, customers had no choice but to share their intentions with firms. If a technology enthusiast wanted to find out if a website sold a particular surveillance device, there was no shortcut but to type some keywords into a search box and therefore give the company a valuable intention stream. Companies, therefore, had all the power. Many tried often too hard to push products and advertisements. The consumer had no voice.During the first data revolution, successful companies gained power by collecting, aggregating, and analyzing the customer data they collected. However, most companies did not know what to do and ended up burying their data in tombs.
The second data revolution brought about a new dimension to data creation: users started to actively contribute explicit data such as information about themselves, their friends, or about the items they purchased. These data went far beyond the clicks and search data that characterized the first decade of the web.
An early example of user-generated content was Amazon.com’s reviews system. The firm realized that users often trusted recommendations by other users more than promotional material found elsewhere on the web. By enabling users to actively contribute such explicit data, Amazon.com succeeded to leverage knowledge dormant in its large customer base to help customers with their purchasing decisions.
Later, Wikipedia increased transparency even more by allowing online collaboration. By allowing users to interact and build on top of each other, the site relinquished control over their space. The benefit of allowing such user interaction today is obvious-why spend time on hold with a customer service representative if we can just Google the cryptic error codes to see if someone else has already solved the same problem? People learned that by sheer large numbers, an online user community was likely to be more helpful than a representative employed by the company.
Today, the online world has shifted to a model of collaboration and explicit data creation. Successful firms develop systematic ways to encourage and reward users who contribute honest data. A good system does not try to trick customers into revealing demographics or contact information that is useful for the company. Rather, it rewards users with information that is useful to them.
Netflix, for example, allows users to contribute ratings for movies that they have seen. Users have an incentive to contribute accurate data because this will give them better recommendations for new movies. The 1999 “Web 2.0 company” MoodLogic (acquired by All Media Guide, in turn acquired by Macrovision) enabled users to create metadata about their favorite music. Why on earth would they do that? Because they got back playlists, which made it easier for them to discover new music they enjoyed. Such successful companies realized the key feature of a good incentive system: people need to see that they profit from the outcome in some way if they are willing to put in the effort to contribute truthfully.
In the last few years, users have gone a lot further than contributing metadata to movies and music: in fact, they have taken center stage. The center of the universe has shifted from e-business to me-business. Customers are also starting to discover each other, and to interact with each other. Knowing that they are not alone has shifted the balance of power from companies back to consumers. And they have begun to demand transparency. Customers are beginning to have a voice. They are realizing that the data they voluntarily contribute can help them and others with making decisions, providing true value. In turn, they want to be treated fairly as individuals by the companies they pay attention and money to.
Successful interactions have become genuine communication with near-instantaneous feedback. For example, PayScale allows users to retrieve real-time salary reports based on their job title, location, education, and experience-but only after they have contributed their own data. As the expectations of users change, firms must spend more time developing incentive systems that will entice users to participate.Indeed, the online world is beginning to be ruled by the expectations of the users. No longer is it sufficient for a search engine to cough up some hotels across the world when a weary traveler is looking for a good deal in Bangkok! As these consumer expectations shift, companies that want to stay relevant have no choice but to accept the ideas of the consumer revolution as swiftly as possible. For users, switching costs are cheap-firms can no longer think of “customer relationship management” as providing stickiness for thecustomer (just like fly paper provides stickiness to the fly). Industries such as real estate and automobiles whose business models are built on information asymmetries will quickly lose their revenues to those who increase transparency using data contributed by consumers.
This leaves us several deep questions to ponder, including what the implications are on customer expectations, and what companies can do to address these expectations. This is the social data revolution (SDR).
Way back in time, communication seemed simple: people were home in the evening, and you could just swing by for a chat. But then the printing press was invented, greatly increasing the scope and reducing the cost of communication. Print, often complemented by services such as mail delivery, enabled firms to reach a huge number of people inexpensively.
Sears, for example, sent its catalog to millions of US households twice a year from 1896 until 1993. It was a slow world—products and prices remained valid until the next issue came out. Relevant dates, such as the delivery date, were hard to predict and rarely communicated to customers. But the customers did not expect much transparency from the firms, either—they were happy as long as the toaster they ordered eventually arrived.
Shifting the focus from transaction to relationship
In this era of limited communication, the firm only knew about the final orders, not the process of decision making. The focus was on transactions, not on relationships.
And now? The Internet allows us to reach anyone, anywhere, instantaneously. The reach of communication has increased from the people in the sender’s town to the entire world. People are social—they want to listen, comment, and be heard. But now that everyone can have a voice, who actually gets heard?
In the old world, senders bore the main cost of communication. Buying stamps and mailing out physical letters limited the number of messages generated. But in electronic communication, the marginal cost of another message is essentially zero. The bottleneck has moved from the sender to the receivers: they are becoming inundated with more requests for attention than they can deal with. The problem is that we are hard-wired to attend to new stimuli. We need to make these new technologies work for people and not against them.
The new currency: May I have your attention, please?
With all these demands on our time, how should we allocate our attention? Randomly? Perhaps—a former colleague’s strategy was to sporadically delete the messages in his inbox as his way of coping with information overload. Needless to say, though, his typical excuse (“I guess your email must have been in the batch I deleted”) was not particularly popular.
Right now, for most of us, that long-awaited love letter arrives the same way as yet another credit card solicitation. Can we do better than allocating our attention randomly? The answer comes in two parts: data, and more data.
Meta-data, data about the message, can help guide our decisions: how important it is for senders that their message gets read, and what is the message’s expected value for the reader?
Well, the simplest way to get this data is just to ask! Mr. Sender, tell us on a scale from 0 to 10: how important is it to you that the reader actually reads your message? And how much do you think the reader will get out of it? These two numbers can help us prioritize our attention.
But taking these values by themselves won’t do the trick. Just as in the physical world, slimy marketers will try to game the system by creating the impression that their message is of utmost importance to us. They’ll try to whet our appetites and get us to open that spam.
To solve this problem, we’ll need to introduce a direct feedback mechanism by getting some data from the message’s recipient. Obviously, this wouldn’t work for physical mail—our junk mail just finds its way to the shredder. This non-response is a very weak learning signal since the sender has no way of gauging the recipient’s response to the message. It could be that the recipient was an early adopter of the sender’s product and is very happy with it. Or, he could be getting very annoyed with all these messages, to the degree that he is actually starting to hate the company!
In the world of cheap, bi-directional communication, we can do better. The receiver can directly indicate the actual value the message has for him—if he actually does enjoy receiving lots of updates, for example, he can express positive feedback. By indicating the actual relevance for him, the receiver can increase or decrease the relevance of future messages from the same sender. That is, he directly benefits from his actions in the immediate future.
Senders, on the other hand, can benefit as well. There is a new term in the cost function of mass communication—the cost of sending unwanted messages, as expressed by the rising voice of the consumer. Being aware of their recipients’ feedback helps them maintain their pristine reputation—senders will not benefit by becoming attention offenders.
Cheap communication allows us to calibrate senders’ predictions with the actual value perceived by the recipient. As we build up a history of direct feedback, our relevance functions will improve and allow us to prioritize our attention effectively. With free bi-directional communication, the era of the con-artist is coming to an end—only companies that respect their customers will be able to get through to them. Since everybody has an incentive to make as accurate relevance predictions as possible, we can use the power of the community to build a good system.
To sum up, two data sources allow us to harness the power of the community: relevance predictions from the senders, and relevance assessments of the recipients.
The communication revolution is a meta-data revolution
With communication being free and instantaneous, attention is increasingly scarce. Economics is the science of scarcity. So, that’s why we need to develop an economic model of communication. Before, scarcity was on the side of the senders (time, money). It was impossible for firms to communicate effectively with large numbers of people at once, and communication/coordination between customers was even more difficult. There was no way for an individual to effectively reach a broad audience beyond a very limited radius. But the communication revolution has brought about many changes. At first glance, this seemed to be great for companies—it’s now almost free to bury customers in ad campaigns! However, now that the scarcity has shifted to the recipients (time, attention), communication needs to go beyond transactions and move to relationships. In fact, the value of relationships is greater than the value of transactions. Truly customer-centric companies like Zappos understand the value of long-term relationships and bidirectional communication. Unfortunately, though, these companies are the exception. There are many more companies that are moving in the wrong direction by cutting costs in customer service. In general, communication between individuals and firms has not become any easier even though it’s now easier than ever for individuals to communicate with each other. When will the communication revolution allow us to easily reach all companies we want to talk to?
ANDREAS WEIGEND SPEAKS AT SAP
Wednesday, April 16, 2008
3:30pm Networking Mixer (refreshments will be provided)
4:00pm – 6:00pm Presentation and Discussion
Building D, Southern Cross Room
SAP Labs, 3410 Hillview Avenue, Palo Alto, CA 94304
The production, aggregation, distribution, and consumption of data is changing dramatically. Traditionally, paid specialists actively collected data for a specific purpose. Now, we are flooded with data streams of intention, attention, situation, and location of individuals, plus data about personal relationships. In addition to these implicit traces of behavior, people contribute data explicitly on platforms for mapping, housing, automotive, and salary data.
The money is where data influence decisions. Most firms believe in internal transparency, basing decisions on data they collect. Few understand how they can benefit by extending this transparency to the outside. What data should the firm share with its customers so some of them can actually help the firm? A sound data strategy has become central to most firms.
Barriers to data business used to be high, including expensive infrastructure and complex business relationships. Infrastructure has now been commoditized; information asymmetries are being reduced by those companies that understand the new consumer data revolution. Relevant questions in today’s marketplace include: How can we set up a system (including incentives) so that people actually do contribute useful and truthful data? What properties does the market need to have so that collective intelligence emerges? What value can the firm create for the contributors to drive participation? It is their perception of this value that will decide whether they demand to be paid for their contributions, or whether they are willing to pay themselves? Finally, what should be the currency of the payments: money, attention, or even more data?