A hundred years ago, the only data a shopkeeper had to work with was the inventory on the shelf, and the money in the till at the end of the day. That data was recorded with a fountain pen. The consumer based her purchases on pretty pictures on the box or on anecdotes from her friends.
Fifty years ago, mail order companies knew where you lived and what you ordered. In addition, they could buy some basic demographic information about you. That was it for personal data pre-web.
With the advent of e-commerce, retailers could track every click and purchase, and capture every abandoned shopping cart.
In the 1990s, Amazon pioneered the use of data to help its customers make better decisions. First, implicit data: Clicks and purchases of all users are aggregated to suggest items to a shopper in response to their most recent click. Second, explicit data: Customers have the opportunity to publish reviews that potentially influence the purchasing decisions of other customers. User-generated content turned marketing–previously viewed as carefully controlled and released information–on its head.
I think of Amazon as a data refinery: Amazon takes the data that people create, refines the data, and returns results, allowing people to make better decisions. Amazon now influences how a billion people shop.
This article looks at three common questions that many people ask every day: (1) Who should I work with? (2) Which route should I take? (3) Where should I stay on my next trip? The answers to these questions, their decisions, are now influenced by the personal data of a billion people.
A startup I am advising recently hired a star engineer. How did they find him? Not through referrals or a headhunter, but through a post of his on Quora, a question-and-answer site. Like the shopkeeper, employers now have vastly more data resources. And like Amazon, job and professional sites now refine data that people create to help both individuals and companies make better decisions.
For example, LinkedIn provides tools for individuals to both refine their own personal data, creating a work identity that transcends a specific job, and to find others by acting as a refinery for other people’s data. Similar to e-commerce, the asymmetry between buyer and seller is fading away.
This does not only apply to full-time jobs. The number of marketplaces with different mechanisms to match talent and tasks is exploding. Underlying the future of work is identity that persists across tasks and jobs where reputation is a key output of the data refinery.
Within firms, data refineries are used to create teams and track interactions. A hedge fund with more than 100 billion dollars under management captures video and audio of its meetings and other data sources and correlates them to the outcomes of trading decisions. And Google’s “People Analytics” has reinvented HR.
In the future, what kinds of jobs will still require full-time employment, and what outputs of personal data refineries will be needed to power the human cloud?
In the 1990s, at Xerox PARC, we used a Thinking Machines supercomputer to analyze automobile traffic patterns in order to predict when the flow would change from laminar to turbulent. Little data, and many assumptions, went into those models.
Twenty years later, a complicated prediction problem has turned into simple observations, in real time, of how cars are moving, or not. Microsoft spin-off Inrix refines geo-location data from more than 100 million individuals a day. In turn, it provides them with crowd-sourced traffic information. You may be sharing your location data without even knowing it.
The company, which sells to Garmin, MapQuest, Ford, BMW and others, collects data from mobile carriers about when a phone switches between cell towers, in addition to GPS and other data. Besides helping drivers make better decisions on which route to take, Inrix also helps cities with their planning decisions, from how to time traffic lights to where to build bridges.
As a byproduct, Inrix provides hedge funds with shopping mall traffic data to help them place bets. For example, data collected on Black Friday 2012 correctly predicted a major bump in sales for the entire holiday season.
We are what we eat, we are what we search for, we are where we were, and we are who we were with. Location history is amongst the most sensitive data about a person. Or, as Yogi Berra said, “No matter where you go, there you are.”
In 2005, Marriott announced a breakthrough in customer service: Guests would now be able to specify their pillow preference when making their reservation! This pillow personalization represented a shift in what had become gold standard in hospitality: personality-free lodging.
While hotels can capture personal data ranging from real-time minibar and video consumption, to card key accesses to room and gym, their goal still seems to be the sanitized experience.
In the meantime, their market has been threatened from a completely different direction. Airbnb offers a rich set of data to both guest and hosts enabling them to make their decision: Love pets? Want to share a hot tub? We’ve got the match for you.
However, staying in a stranger’s guest room requires a much deeper level of trust than staying in a hotel. To address this need, Airbnb verifies online identity on Facebook or LinkedIn by matching it with offline identity via Jumio.
Travel and tourism is ten percent of the world’s GDP. Beyond accommodation, matching and trust based on refining personal data now also extend to other areas from ridesharing to renting out your car.
A hundred years ago, data got recorded with a fountain pen. The data deteriorated over time, whereas the fountain pen got better with consistent use. In the information age, the central question for companies is: Will their product or service get better over time, or worse? Data refineries such as Amazon, Google and Facebook get better.
Like the story of the genie in the bottle, the personal data servant can wield its power for good or evil. What it cannot do, however, is go back into the bottle. The new opportunities in this abundant data ecosystem will come from new ideas about how to refine this data.
The sign has flipped, like that of the shopkeeper in the morning, from CLOSED to OPEN.
Here are the Eight Rules discussed in the Master Classes on “Big Data, Smart Metrics and Customer Centricity” in Shanghai, Singapore, London, and at Stanford in January 2012:
By The Social Data Lab (socialdatalab.stanford.edu)
In today’s increasingly digitized world, we are creating data in unprecedented ways. Ubiquitous and conspicuous social and mobile connections have empowered social consumers to broadcast their locations, opinions, thoughts and emotions to the world in real-time with nothing more than a few clicks, many times a day, on multiple platforms.
by Jason Lee, Evelyn Larrubia, Sameh El Amawy and Michael Marcotte
In 1993, The New Yorker published a cartoon by Peter Steiner of two dogs at a computer that became an instant classic, tacked up on bulletin boards everywhere. The caption: “On the internet, no one knows you’re a dog.”
A recent survey of a group of 98 tech-savvy Stanford students shows that the world Steiner depicted may be behind us. Asked about how they behave—or would behave—in a variety of situations online, students painted a picture that looked surprisingly like real life.
Hi there, here is the 20-minute audio of the keynote on “The State of the Social Data Revolution” at the 2011 Predictive Analytics World in San Francisco.
Would love to get your comments. Thanks!
The reception in Beijing will be on Sunday 12 Sep from 4pm to 7pm at 798 Art District. We already have some great participants to celebrate the launch decision. Please email me if you are interested in attending.
The Office of the Mayor of San Francisco and the City’s Chief Information Officer are hosting an event that will change the way you think about data:
How the Social Data Revolution Changes (almost) Everything
Why do people share, what do people share?
And how does this influence their behavior?
Speaker: Andreas Weigend (@aweigend)
Location: One South Van Ness, 2nd Floor Atrium
Date: Tuesday, August 10, 2010
Time: The speech begins at 4pm, and is followed by a reception at 5pm, sponsored by Open-First.
Andreas Weigend studies people and the data they create and share. He works with companies that are eager to develop strategies to realize the untapped power of data, including Alibaba, Best Buy, Lufthansa, Nokia, and Thomson Reuters, and fun startups including San Francisco-based MrTweet.com and Skout.com (Boy Ahoy). Previously, as the Chief Scientist of Amazon.com, he helped build the customer-centric, measurement-focused culture central to Amazon’s success. As a partner with San Francisco-based Open-First, he helps organizations absorb a set of insights based on data, mobile and social technologies.
Andreas teaches at Stanford and shares his insights at top conferences, such as the World Innovation Forum. He received his PhD from Stanford in physics, and lives in San Francisco, Shanghai, on weigend.com, and on Facebook.
Here are a few related press mentions (Summer 2010):
Shanghai, China. Quite early, 3:30am local time. (Or maybe very late? Actually just only barely back to my condo from a relaxing foot massage.) My US mobile rings. Austin Carr calling from New York. Austin Carr? Sounds like a superposition of two friends, Austin Ku who took me to see CHINGLISH by David Henry Hwang in New York last month, and Jeremy Carr, my Stanford TA who kept the class in shape last quarter. But we right away started having a fascinating conversation… which made it into Fast Company very fast (and served as starting point for a great article Are these Nobodies the New Somebodies? with the London-based Evening Standard)! Here you go:
by AUSTIN CARR Wed Jul 14, 2010
Andreas Weigend knows how to influence people. As the former chief scientist at Amazon, Weigend helped implement a series of ingenious tools to help customers “make better decisions,” from recommended purchases and one-click checkouts, to wish lists and book-interest sharing. With our recent launch of the Influence Project, we spoke with Weigend about what “influence” means on the Web. Weigend, a professor at Stanford, approached the subject philosophically, picking apart the complicated concept of influence by each attribute and nuance. Read the rest of this entry »
Hello, you can do four things with the speech I gave at the 2010 World Innovation Forum in New York:
It was exciting to be part of the World Innovation Forum, an event packed with insights and a turnout of more than 800 thought leaders and a fantastic line-up of speakers. I had great company on stage, speaking between Chip Heath (who I went to grad school with) and Biz Stone (who co-founded Twitter).
I have put up the audio of my talk [mp3, 35 min, 32MB], the transcript [pdf | docx], and the slides [pdf | pptx]. And in terms of press commentary, check out what The Huffington Post, FastCompany, HSM, OnInnovation, and Steve Todd write about it, and please add your own thoughts via the comment box at the bottom of this post.
I am fortunate to present the insights on WIF2010 and the Social Data Revolution by two guest writers: Noah Burbank, a student in Stanford’s Social Data Revolution class this Spring, and Ted Shelton, the CEO of Open-First. And, as always, please do tell us what you think by leaving a comment below. Thanks!