A hundred years ago, the only data a shopkeeper had to work with was the inventory on the shelf, and the money in the till at the end of the day. That data was recorded with a fountain pen. The consumer based her purchases on pretty pictures on the box or on anecdotes from her friends.
Fifty years ago, mail order companies knew where you lived and what you ordered. In addition, they could buy some basic demographic information about you. That was it for personal data pre-web.
With the advent of e-commerce, retailers could track every click and purchase, and capture every abandoned shopping cart.
In the 1990s, Amazon pioneered the use of data to help its customers make better decisions. First, implicit data: Clicks and purchases of all users are aggregated to suggest items to a shopper in response to their most recent click. Second, explicit data: Customers have the opportunity to publish reviews that potentially influence the purchasing decisions of other customers. User-generated content turned marketing–previously viewed as carefully controlled and released information–on its head.
I think of Amazon as a data refinery: Amazon takes the data that people create, refines the data, and returns results, allowing people to make better decisions. Amazon now influences how a billion people shop.
This article looks at three common questions that many people ask every day: (1) Who should I work with? (2) Which route should I take? (3) Where should I stay on my next trip? The answers to these questions, their decisions, are now influenced by the personal data of a billion people.
A startup I am advising recently hired a star engineer. How did they find him? Not through referrals or a headhunter, but through a post of his on Quora, a question-and-answer site. Like the shopkeeper, employers now have vastly more data resources. And like Amazon, job and professional sites now refine data that people create to help both individuals and companies make better decisions.
For example, LinkedIn provides tools for individuals to both refine their own personal data, creating a work identity that transcends a specific job, and to find others by acting as a refinery for other people’s data. Similar to e-commerce, the asymmetry between buyer and seller is fading away.
This does not only apply to full-time jobs. The number of marketplaces with different mechanisms to match talent and tasks is exploding. Underlying the future of work is identity that persists across tasks and jobs where reputation is a key output of the data refinery.
Within firms, data refineries are used to create teams and track interactions. A hedge fund with more than 100 billion dollars under management captures video and audio of its meetings and other data sources and correlates them to the outcomes of trading decisions. And Google’s “People Analytics” has reinvented HR.
In the future, what kinds of jobs will still require full-time employment, and what outputs of personal data refineries will be needed to power the human cloud?
In the 1990s, at Xerox PARC, we used a Thinking Machines supercomputer to analyze automobile traffic patterns in order to predict when the flow would change from laminar to turbulent. Little data, and many assumptions, went into those models.
Twenty years later, a complicated prediction problem has turned into simple observations, in real time, of how cars are moving, or not. Microsoft spin-off Inrix refines geo-location data from more than 100 million individuals a day. In turn, it provides them with crowd-sourced traffic information. You may be sharing your location data without even knowing it.
The company, which sells to Garmin, MapQuest, Ford, BMW and others, collects data from mobile carriers about when a phone switches between cell towers, in addition to GPS and other data. Besides helping drivers make better decisions on which route to take, Inrix also helps cities with their planning decisions, from how to time traffic lights to where to build bridges.
As a byproduct, Inrix provides hedge funds with shopping mall traffic data to help them place bets. For example, data collected on Black Friday 2012 correctly predicted a major bump in sales for the entire holiday season.
We are what we eat, we are what we search for, we are where we were, and we are who we were with. Location history is amongst the most sensitive data about a person. Or, as Yogi Berra said, “No matter where you go, there you are.”
In 2005, Marriott announced a breakthrough in customer service: Guests would now be able to specify their pillow preference when making their reservation! This pillow personalization represented a shift in what had become gold standard in hospitality: personality-free lodging.
While hotels can capture personal data ranging from real-time minibar and video consumption, to card key accesses to room and gym, their goal still seems to be the sanitized experience.
In the meantime, their market has been threatened from a completely different direction. Airbnb offers a rich set of data to both guest and hosts enabling them to make their decision: Love pets? Want to share a hot tub? We’ve got the match for you.
However, staying in a stranger’s guest room requires a much deeper level of trust than staying in a hotel. To address this need, Airbnb verifies online identity on Facebook or LinkedIn by matching it with offline identity via Jumio.
Travel and tourism is ten percent of the world’s GDP. Beyond accommodation, matching and trust based on refining personal data now also extend to other areas from ridesharing to renting out your car.
A hundred years ago, data got recorded with a fountain pen. The data deteriorated over time, whereas the fountain pen got better with consistent use. In the information age, the central question for companies is: Will their product or service get better over time, or worse? Data refineries such as Amazon, Google and Facebook get better.
Like the story of the genie in the bottle, the personal data servant can wield its power for good or evil. What it cannot do, however, is go back into the bottle. The new opportunities in this abundant data ecosystem will come from new ideas about how to refine this data.
The sign has flipped, like that of the shopkeeper in the morning, from CLOSED to OPEN.
By The Social Data Lab (socialdatalab.stanford.edu)
In today’s increasingly digitized world, we are creating data in unprecedented ways. Ubiquitous and conspicuous social and mobile connections have empowered social consumers to broadcast their locations, opinions, thoughts and emotions to the world in real-time with nothing more than a few clicks, many times a day, on multiple platforms.
Hi there, here is the 20-minute audio of the keynote on “The State of the Social Data Revolution” at the 2011 Predictive Analytics World in San Francisco.
Would love to get your comments. Thanks!
Shanghai, China. Quite early, 3:30am local time. (Or maybe very late? Actually just only barely back to my condo from a relaxing foot massage.) My US mobile rings. Austin Carr calling from New York. Austin Carr? Sounds like a superposition of two friends, Austin Ku who took me to see CHINGLISH by David Henry Hwang in New York last month, and Jeremy Carr, my Stanford TA who kept the class in shape last quarter. But we right away started having a fascinating conversation… which made it into Fast Company very fast (and served as starting point for a great article Are these Nobodies the New Somebodies? with the London-based Evening Standard)! Here you go:
by AUSTIN CARR Wed Jul 14, 2010
Andreas Weigend knows how to influence people. As the former chief scientist at Amazon, Weigend helped implement a series of ingenious tools to help customers “make better decisions,” from recommended purchases and one-click checkouts, to wish lists and book-interest sharing. With our recent launch of the Influence Project, we spoke with Weigend about what “influence” means on the Web. Weigend, a professor at Stanford, approached the subject philosophically, picking apart the complicated concept of influence by each attribute and nuance. Read the rest of this entry »
Hello, you can do four things with the speech I gave at the 2010 World Innovation Forum in New York:
It was exciting to be part of the World Innovation Forum, an event packed with insights and a turnout of more than 800 thought leaders and a fantastic line-up of speakers. I had great company on stage, speaking between Chip Heath (who I went to grad school with) and Biz Stone (who co-founded Twitter).
I have put up the audio of my talk [mp3, 35 min, 32MB], the transcript [pdf | docx], and the slides [pdf | pptx]. And in terms of press commentary, check out what The Huffington Post, FastCompany, HSM, OnInnovation, and Steve Todd write about it, and please add your own thoughts via the comment box at the bottom of this post.
I am fortunate to present the insights on WIF2010 and the Social Data Revolution by two guest writers: Noah Burbank, a student in Stanford’s Social Data Revolution class this Spring, and Ted Shelton, the CEO of Open-First. And, as always, please do tell us what you think by leaving a comment below. Thanks!
by Adrian Chan and Andreas Weigend
The social data revolution
We live in an age in which social data has become the air we live and breathe. As individuals, our actions, preferences, habits, and even friendships, leave behind a wake of data. Not only data about us, but data that captures our communication and connections. Even our conversations are now data. Conversations that can be captured, stored, and re-distributed as data. Data that connects to us, and is shared with companies and brands with whom we have relationships. Like it or not, the social data revolution is the new business environment. Smart analysis of this social data demands a new mindset.
Business in this new environment has already been profoundly affected by the new datascape. Adaptation is an imperative. But for those who will do more than survive and actually thrive in this environment, the question is not one of adaptation. It is a matter of how best to respond to the world of social data, how to metabolize it, and incorporate it as if it belonged to the very company DNA. Read the rest of this entry »
Listen to the Conversation with Kai Ryssdal (Marketplace) on the Social Data Revolution: Companies get smart on Digital Data. Produced by American Public Media. Broadcast by NPR and Public Radio International on November 18, 200.
And please share what you think… Comment (via Facebook Connect) below!
Transcript (from http://bit.ly/dataNPR)
KAI RYSSDAL: The data trail that we create every day is only growing. Every time we go online, every time we use our cell phones, companies log our preferences. They make suggestions, and they remember what we do. Even though a lot of consumers have gotten used to that, a lot of businesses are still trying to figure out how to use our data to the best effect. One of the first companies to realize the social potential of consumer data was Amazon.com. And Andreas Weigend used to be the chief scientist there. Welcome to the program. Read the rest of this entry »
In 2009, more data will be generated by individuals than in the entire history of mankind through 2008. Information overload is more serious than ever. What are the implications for marketing?
Check out this article at http://blogs.harvardbusiness.org/now-new-next/2009/05/the-social-data-revolution.html
by Ray Bradford and Andreas Weigend. Ray Bradford, currently a student at Stanford University’s Graduate School of Business, is taking Data Mining and E-Business (Stats 252)
You’re working on that big project when momentum stalls at 9:06 PM and you find yourself on Facebook staring at the news feed. You are confronted by a stream of updates from that melodramatic train wreck of a former high school classmate, whose friend request you accepted out of guilt last week. You couldn’t care less Read the rest of this entry »