A hundred years ago, the only data a shopkeeper had to work with was the inventory on the shelf, and the money in the till at the end of the day. That data was recorded with a fountain pen. The consumer based her purchases on pretty pictures on the box or on anecdotes from her friends.
Fifty years ago, mail order companies knew where you lived and what you ordered. In addition, they could buy some basic demographic information about you. That was it for personal data pre-web.
With the advent of e-commerce, retailers could track every click and purchase, and capture every abandoned shopping cart.
In the 1990s, Amazon pioneered the use of data to help its customers make better decisions. First, implicit data: Clicks and purchases of all users are aggregated to suggest items to a shopper in response to their most recent click. Second, explicit data: Customers have the opportunity to publish reviews that potentially influence the purchasing decisions of other customers. User-generated content turned marketing–previously viewed as carefully controlled and released information–on its head.
I think of Amazon as a data refinery: Amazon takes the data that people create, refines the data, and returns results, allowing people to make better decisions. Amazon now influences how a billion people shop.
This article looks at three common questions that many people ask every day: (1) Who should I work with? (2) Which route should I take? (3) Where should I stay on my next trip? The answers to these questions, their decisions, are now influenced by the personal data of a billion people.
A startup I am advising recently hired a star engineer. How did they find him? Not through referrals or a headhunter, but through a post of his on Quora, a question-and-answer site. Like the shopkeeper, employers now have vastly more data resources. And like Amazon, job and professional sites now refine data that people create to help both individuals and companies make better decisions.
For example, LinkedIn provides tools for individuals to both refine their own personal data, creating a work identity that transcends a specific job, and to find others by acting as a refinery for other people’s data. Similar to e-commerce, the asymmetry between buyer and seller is fading away.
This does not only apply to full-time jobs. The number of marketplaces with different mechanisms to match talent and tasks is exploding. Underlying the future of work is identity that persists across tasks and jobs where reputation is a key output of the data refinery.
Within firms, data refineries are used to create teams and track interactions. A hedge fund with more than 100 billion dollars under management captures video and audio of its meetings and other data sources and correlates them to the outcomes of trading decisions. And Google’s “People Analytics” has reinvented HR.
In the future, what kinds of jobs will still require full-time employment, and what outputs of personal data refineries will be needed to power the human cloud?
In the 1990s, at Xerox PARC, we used a Thinking Machines supercomputer to analyze automobile traffic patterns in order to predict when the flow would change from laminar to turbulent. Little data, and many assumptions, went into those models.
Twenty years later, a complicated prediction problem has turned into simple observations, in real time, of how cars are moving, or not. Microsoft spin-off Inrix refines geo-location data from more than 100 million individuals a day. In turn, it provides them with crowd-sourced traffic information. You may be sharing your location data without even knowing it.
The company, which sells to Garmin, MapQuest, Ford, BMW and others, collects data from mobile carriers about when a phone switches between cell towers, in addition to GPS and other data. Besides helping drivers make better decisions on which route to take, Inrix also helps cities with their planning decisions, from how to time traffic lights to where to build bridges.
As a byproduct, Inrix provides hedge funds with shopping mall traffic data to help them place bets. For example, data collected on Black Friday 2012 correctly predicted a major bump in sales for the entire holiday season.
We are what we eat, we are what we search for, we are where we were, and we are who we were with. Location history is amongst the most sensitive data about a person. Or, as Yogi Berra said, “No matter where you go, there you are.”
In 2005, Marriott announced a breakthrough in customer service: Guests would now be able to specify their pillow preference when making their reservation! This pillow personalization represented a shift in what had become gold standard in hospitality: personality-free lodging.
While hotels can capture personal data ranging from real-time minibar and video consumption, to card key accesses to room and gym, their goal still seems to be the sanitized experience.
In the meantime, their market has been threatened from a completely different direction. Airbnb offers a rich set of data to both guest and hosts enabling them to make their decision: Love pets? Want to share a hot tub? We’ve got the match for you.
However, staying in a stranger’s guest room requires a much deeper level of trust than staying in a hotel. To address this need, Airbnb verifies online identity on Facebook or LinkedIn by matching it with offline identity via Jumio.
Travel and tourism is ten percent of the world’s GDP. Beyond accommodation, matching and trust based on refining personal data now also extend to other areas from ridesharing to renting out your car.
A hundred years ago, data got recorded with a fountain pen. The data deteriorated over time, whereas the fountain pen got better with consistent use. In the information age, the central question for companies is: Will their product or service get better over time, or worse? Data refineries such as Amazon, Google and Facebook get better.
Like the story of the genie in the bottle, the personal data servant can wield its power for good or evil. What it cannot do, however, is go back into the bottle. The new opportunities in this abundant data ecosystem will come from new ideas about how to refine this data.
The sign has flipped, like that of the shopkeeper in the morning, from CLOSED to OPEN.
Here are the Eight Rules discussed in the Master Classes on “Big Data, Smart Metrics and Customer Centricity” in Shanghai, Singapore, London, and at Stanford in January 2012:
By The Social Data Lab (socialdatalab.stanford.edu)
In today’s increasingly digitized world, we are creating data in unprecedented ways. Ubiquitous and conspicuous social and mobile connections have empowered social consumers to broadcast their locations, opinions, thoughts and emotions to the world in real-time with nothing more than a few clicks, many times a day, on multiple platforms.
by Jason Lee, Evelyn Larrubia, Sameh El Amawy and Michael Marcotte
In 1993, The New Yorker published a cartoon by Peter Steiner of two dogs at a computer that became an instant classic, tacked up on bulletin boards everywhere. The caption: “On the internet, no one knows you’re a dog.”
A recent survey of a group of 98 tech-savvy Stanford students shows that the world Steiner depicted may be behind us. Asked about how they behave—or would behave—in a variety of situations online, students painted a picture that looked surprisingly like real life.
Hi there, here is the 20-minute audio of the keynote on “The State of the Social Data Revolution” at the 2011 Predictive Analytics World in San Francisco.
Would love to get your comments. Thanks!
Hello, you can do four things with the speech I gave at the 2010 World Innovation Forum in New York:
It was exciting to be part of the World Innovation Forum, an event packed with insights and a turnout of more than 800 thought leaders and a fantastic line-up of speakers. I had great company on stage, speaking between Chip Heath (who I went to grad school with) and Biz Stone (who co-founded Twitter).
I have put up the audio of my talk [mp3, 35 min, 32MB], the transcript [pdf | docx], and the slides [pdf | pptx]. And in terms of press commentary, check out what The Huffington Post, FastCompany, HSM, OnInnovation, and Steve Todd write about it, and please add your own thoughts via the comment box at the bottom of this post.
I am fortunate to present the insights on WIF2010 and the Social Data Revolution by two guest writers: Noah Burbank, a student in Stanford’s Social Data Revolution class this Spring, and Ted Shelton, the CEO of Open-First. And, as always, please do tell us what you think by leaving a comment below. Thanks!
by Chuanyang Chee, Ron Chung, and Andreas Weigend
Curious about the best response to the question from IDEAS PROJECT last week?
Here is what Ron said about a “friend” in the era of the social data revolution:
There are two types of ‘friends’, (i) real ‘close-to-heart’ personal friendships, and (ii) online social friendships.
(i) In real personal friendships you more carefully screen and maintain that relationships. In these situations, you provide more physical and emotional attention compared to online relationships.
(ii) Online social friendships form to maintain touchpoints with people we interact with (sort of like a large addressbook). In the context of consumer internet and social networks/media, an online ‘friend’ is someone you form a weak connection through some form of engagement. This engagement can occur through real world meeting or simply an online exchange (e.g. blog comments, Twitter message, etc).
Also, in these online friendships, there is ambiguity around bilateral versus unilateral ‘friendships’. For example, Twitter uses ‘followers’ & Facebook uses ‘fans’ to represent unidirection relationships and Facebook uses ‘friends’ to denote bilateral friendships. However, some Facebook ‘friendships’ are not truly bilateral. They are simply ways for one side to collect ‘friends’ for the sake of amassing a large audience. All of this points to a desire for people maintain touchpoints with people through online medium should they ever want to re-engage them.
In the end, online social friendships give us ambient awareness of what is going on with people, giving us a type of “reality-TV news” channel.
Daphna Oyserman suggested:
My favorite one-liner came from Jason Wei in my Stanford class:
My own points (to the degree anyone can have their own points after reading through hundreds of responses) would be, that a friend to me is:
Please use the comment box below for your comments. Thanks!
This survey on the Social Data Revolution was developed by Chuanyang and Andreas and taken by Spring 2010 students at Stanford’s The Social Data Revolution, and Tsinghua’s The Digital Networked Economy.
Finding that long-lost best friend from elementary school has become trivial ever since Facebook hit a total subscription of 400 million active users. But having not kept in touch for a couple of years or decades, what is the point of connecting now? Does he even still consider me a friend? Remember me? Read the rest of this entry »
Listen to the Conversation with Kai Ryssdal (Marketplace) on the Social Data Revolution: Companies get smart on Digital Data. Produced by American Public Media. Broadcast by NPR and Public Radio International on November 18, 200.
And please share what you think… Comment (via Facebook Connect) below!
Transcript (from http://bit.ly/dataNPR)
KAI RYSSDAL: The data trail that we create every day is only growing. Every time we go online, every time we use our cell phones, companies log our preferences. They make suggestions, and they remember what we do. Even though a lot of consumers have gotten used to that, a lot of businesses are still trying to figure out how to use our data to the best effect. One of the first companies to realize the social potential of consumer data was Amazon.com. And Andreas Weigend used to be the chief scientist there. Welcome to the program. Read the rest of this entry »
Download the mp3 of the World Marketing Forum keynote (45MB, 50 minutes, Mexico City, July 1, 2009).
Ladies and gentlemen, it is an honor for me to be here and to talk to you about what I think it the most interesting, the most exciting thing I can talk to you about. Read the rest of this entry »