By The Social Data Lab (socialdatalab.stanford.edu)

In today’s increasingly digitized world, we are creating data in unprecedented ways. Ubiquitous and conspicuous social and mobile connections have empowered social consumers to broadcast their locations, opinions, thoughts and emotions to the world in real-time with nothing more than a few clicks, many times a day, on multiple platforms.

Read the rest of this entry »

Hi there, here is the 20-minute audio of the keynote on “The State of the Social Data Revolution” at the 2011 Predictive Analytics World in San Francisco.
Would love to get your comments. Thanks!

Shanghai, China. Quite early, 3:30am local time. (Or maybe very late? Actually just only barely back to my condo from a relaxing foot massage.) My US mobile rings. Austin Carr calling  from New York. Austin Carr? Sounds like a superposition of two friends, Austin Ku who took me to see CHINGLISH by David Henry Hwang in New York last month, and Jeremy Carr, my Stanford TA who kept the class in shape last quarter. But we right away started having a fascinating conversation… which made it into Fast Company very fast (and served as starting point for a great article Are these Nobodies the New Somebodies? with the London-based Evening Standard)! Here you go:

Amazon’s Former Chief Scientist on Influence, Twitter’s Fake Audience, and iPad Sex Appeal

by AUSTIN CARR Wed Jul 14, 2010

Andreas Weigend knows how to influence people. As the former chief scientist at Amazon, Weigend helped implement a series of ingenious tools to help customers “make better decisions,” from recommended purchases and one-click checkouts, to wish lists and book-interest sharing. With our recent launch of the Influence Project, we spoke with Weigend about what “influence” means on the Web. Weigend, a professor at Stanford, approached the subject philosophically, picking apart the complicated concept of influence by each attribute and nuance. Read the rest of this entry »

Hello, you can do four things with the speech I gave at the 2010 World Innovation Forum in New York:

1. Play or download the mp3 of the speech,

2. Leave your comments on the slides and see the annotations of others,

3. Leave your comments on the transcript and see the annotations of others, and

It was exciting to be part of the World Innovation Forum, an event packed with insights and a turnout of more than 800 thought leaders and a fantastic line-up of speakers. I had great company on stage, speaking between Chip Heath (who I went to grad school with) and Biz Stone (who co-founded Twitter).

I have put up the audio of my talk [mp3, 35 min, 32MB], the transcript [pdf | docx], and the slides [pdf | pptx]. And in terms of press commentary, check out what The Huffington Post, FastCompany, HSM, OnInnovation, and Steve Todd write about it, and please add your own thoughts via the comment box at the bottom of this post.

I am fortunate to present the insights on WIF2010 and the Social Data Revolution by two guest writers: Noah Burbank, a student in Stanford’s Social Data Revolution class this Spring, and Ted Shelton, the CEO of Open-First. And, as always, please do tell us what you think by leaving a comment below. Thanks!

WTF is WIF??

Read the rest of this entry »

by Adrian Chan and Andreas Weigend

This post has been translated into German (GDI Impuls 2/2010),
Spanish, and Chinese (simplified).

The social data revolution

We live in an age in which social data has become the air we live and breathe. As individuals, our actions, preferences, habits, and even friendships, leave behind a wake of data. Not only data about us, but data that captures our communication and connections. Even our conversations are now data. Conversations that can be captured, stored, and re-distributed as data. Data that connects to us, and is shared with companies and brands with whom we have relationships. Like it or not, the social data revolution is the new business environment. Smart analysis of this social data demands a new mindset.

Business in this new environment has already been profoundly affected by the new datascape. Adaptation is an imperative. But for those who will do more than survive and actually thrive in this environment, the question is not one of adaptation. It is a matter of how best to respond to the world of social data,  how to metabolize it, and incorporate it as if it belonged to the very company DNA. Read the rest of this entry »

Listen to the Conversation with Kai Ryssdal (Marketplace) on the Social Data Revolution: Companies get smart on Digital Data. Produced by American Public Media. Broadcast by NPR and Public Radio International on November 18, 200.

• Play (publicradio.org)
• Download (weigend.com, mp3, 2MB)

And please share what you think… Comment (via Facebook Connect) below!

Transcript (from http://bit.ly/dataNPR)

KAI RYSSDAL: The data trail that we create every day is only growing. Every time we go online, every time we use our cell phones, companies log our preferences. They make suggestions, and they remember what we do. Even though a lot of consumers have gotten used to that, a lot of businesses are still trying to figure out how to use our data to the best effect. One of the first companies to realize the social potential of consumer data was Amazon.com. And Andreas Weigend used to be the chief scientist there. Welcome to the program. Read the rest of this entry »

In 2009, more data will be generated by individuals than in the entire history of mankind through 2008. Information overload is more serious than ever. What are the implications for marketing?

Check out this article at http://blogs.harvardbusiness.org/now-new-next/2009/05/the-social-data-revolution.html

by Ray Bradford and Andreas Weigend. Ray Bradford, currently a student at Stanford University’s Graduate School of Business, is taking Data Mining and E-Business (Stats 252)

You’re working on that big project when momentum stalls at 9:06 PM and you find yourself on Facebook staring at the news feed.  You are confronted by a stream of updates from that melodramatic train wreck of a former high school classmate, whose friend request you accepted out of guilt last week.  You couldn’t care less Read the rest of this entry »

This video, part of the Nokia IDEAS PROJECT, explains how to increase relevance on platforms including Twitter and Facebook

Transcript of video: Data Gathering Allows Platforms to Serve Customers Better

I think the key element of technology will be that we manage to convince people to give us data, which allows the relevance function of the system to serve them better. I always say that the 1990’s was the decade of given a set of data, what insights can I get, whereas the 2000’s is the decade of given a problem, what data can I get in order to solve my problem. Let’s say the problem of relevance is not solved by yet smarter algorithms. We have reached the ceiling there, but it is solved by people coming up with smart incentives where people see that, “Hey, if I show you something about myself, then I will be served much better. My attention will be rewarded more richly, and I’ll be disappointed less”.

Enroll Consumers in Helping Themselves

My view is that successful companies will see that the consumer is not their enemy. It’s not about snooping up behind the consumer, getting their digital exhaust, and then selling them stuff they don’t want. That’s not how it works. The successful companies will be those who manage to enroll the consumer in helping them and helping themselves. That’s how they will get people to share stuff, not because they want to primarily help the company, but primarily themselves. We see, in China, product development for cars being driven by consumers. As an example, there is a Japanese car maker who actually had a piece made for the car, by consumers, without having any say in that. People said, “We need this,” and someone stepped up and said, “Yes, I have a company in Guangzhou, which can produce it,” and there you are, not only user-created content, but also user created products. That’s part of what I see the future of communication to be.

Costs Have Shifted to the Consumer

The shift that has occurred is that the relevant costs to the recipient are now the dominant ones. If you think about sending out mail ten or twenty years ago, the cost was twenty five cents, which the sender had to pay. The intelligence used to sit on the side of the sender, for instance, Capital One carefully figuring out whom to target. But, with electronic communication, the costs have shifted to the recipient, our time, our attention, our cost to deal with the interruptions. My belief is that it’s not primarily a technology play, but it’s primarily a people play where people provide metadata, data where they predict how important their communication is for you, and then a model negotiates, over time. Given their reputation, how much you should be interrupted and whether given the situation you are in, which of course you devise measures much more finely than ever before, you should be interrupted or not.

Building Platform ‘Relevance’

I believe that progress in relevance to you, as an individual, is one of the key things to expect in the near future. For instance, take Twitter, Twitter organizes things by time. What a poor way; it’s one way, but it’s a very poor way. It’s understandable, but it doesn’t really help me manage my attention well. Potentially, Twitter knows a lot about my past actions; whom did I reply to, whom do I follow. It may also know how others reply. What do others think about my messages and what do I do in other networks? The people I follow on Twitter, are they also on my Facebook? Are they in my MySpace? Do I send emails to them? Taking all that information together, I believe, we can build a much better system, which shows the stuff which seems to be relevant to me, as opposed to just showing stuff in chronological order.

Discovery is the New Paradigm

We have seen the cost of communications, across the board, come down. That means a lot more gets produced by a lot more people. The necessary requirement, now, is that the sender, having an easy ability to reach everybody, needs to be matched by the receiver, which has some way of actually cutting through all the junk tofind those nuggets. So, in some way, it used to be that search was the paradigm where I go and I find something. Now, it’s more discovery. “Hey, delight me. Show me some stuff you’re interested in.” Why do people share things about themselves? I did a panel with Shoshana Zuboff and Esther Dyson, at the Fortune Brainstorm Conference. The upshot was that people like to spread their genes and people like to spread their memes. Both of those make people immortal, in some way, your children, in one case, and your ideas, in the other case. To be honest, I think the Web or let’s say Facebook, does provide for both of them. You find people who you can spread your genes with and you definitely find ways to spread your memes.

IdeasProject keywords: Nokia,Ideas Project,Thought leaders,Big ideas,New website,Big thinkers,Information space, Andreas Weigend,Chief Scientist,Amazon.com,Amazon,Berkeley,Stanford,Tsinghua,University

Yesterday alone, Facebook users issued 21 million friend requests. 17 million requests were accepted. So many new connections, and yet they’re all treated the same—what an oversimplification!

All Facebook links are created equal. But links can differ in strength—for example, a close friend versus a casual acquaintance. Links can be in different categories, like your boss versus a random hookup. And links can be asymmetric—Amy may think that Bob is a good friend, yet Bob may not trust Amy at all! The world is not a binary place.

Discovering Discovery: Don’t ask, Do tell

How can we use data to investigate these different properties of links? Today’s social networks do a lousy job of leveraging our existing data. Why do you need to manually confirm my friend request if we’re already calling, IM-ing, and emailing each other all the time? These data sources should be able to make a good guess about the strength and type of our relationship. Why not use existing data sources to propose better default responses?

If we give our networks a richer structure for our links and relationships, we will also be able to discover interesting facts about ourselves. Why is this important? By investigating implicit relations, we can gain insight into our relationships and how they work. For example, I might be surprised to find out that whenever I email my friend John, he always writes me back promptly whereas I always take 10 times longer to respond to him! Armed with this knowledge, I would ask my system to tell me to get my act together and crank out that response if I’m getting too delinquent.

Facebook 1.0 has helped us create an intimate network of our 17,000 friends. Will Facebook 2.0 help us manage them?

Mind the Explicit, Mine the Implicit

What else can data tell us about the quality of our relationships? One way to use data is to figure out differential interest in budding relationships. It’s easy to do this by looking at communications patterns in email, for example—does one person spend hours crafting that perfect email, only to get a reply that took only a few minutes to write? Or has he suddenly acquired a brand new set of favorite books, movies, and music that just happens to match his new love interest? People leave rich traces on the web—we can discover much more about them than the data they explicitly give.

This is only possible if we can look at the user’s history. After all, we can only make inferences about our behavior if we have a past to compare it against. But this introduces new questions: how much would you pay to know how long Monty spent writing you that email? How much would you pay to keep your data private?

Trust Networks

Social networks are also great for learning about trust. Let’s say that I’m thinking of entering in a business deal with you, but I don’t know you too well. Should I trust you?

There’s an easy way to use the power of networks to answer this question. Let’s just look at all of your other connections: do they trust you? We can give people reputation scores by allowing users to rate their interactions with friends. To make the system even more powerful, we could allow users to link their reputations. To illustrate: let’s say I trust my friend Mike so much that I am willing to attach a trust coefficient of 0.9. This implies that if Mike’s rating goes up by 1, I should get a rating boost of 0.9. Conversely, if someone has a bad experience with Mike and downgrades his rating by 1, my rating will also go down by 0.9. Through the power of the community, reputation ratings would spread quickly. (What trust coefficient would you attach to the author of this post?)

Reward Content Generation

One of the best ways to engage users is to get them to understand how every bit of data they contribute will end up benefiting them. In the example of trust networks, people can improve their own reputations by linking themselves with others. In my previous post on communication, I talked about a system where providing feedback on an email’s relevance would directly benefit you in the future. Online social networks need to reward people to provide explicit data, too.

The Facebook Feed was a brilliant idea for surfacing relevant content created by friends. Ideally, the Feed would create a positive feedback loop: good content provided by friends would get high ratings, which would motivate them to post even more good content. However, an early system of allowing users to rate the submissions of their friends was poorly designed—only 21% of users used the feature. On a rainy day, April 15, 2008, Facebook turned off the feedback system. What a step backward! I wish Facebook instead had created a better machine learning system to reward its users to generate and surface good content.

Social networks based on mutually confirmed binary relations was Day One in evolution of social networks. Introducing, richer semantics, more expressive structures including trust coefficients are the beginning of Day Two. What will the second week bring?