A Presenta*on from the NewMR “Social Media Research” Event 9 October, 2012
Enriching Social Data for Market Research Jasper Snyder, Converseon
All copyright owned by The Future Place and the presenters of the material For more informa:on about NewMR events visit newmr.org
#NewMR
Enriching Social Data for Market Research Jasper Snyder 9 October 2012
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
Questions for Today’s Session n ions ca t s e u q f sorts o research and t a h W • edia social m answer? is analys millions m o r f e v you mo n- and semio d w • Ho billions of u ents to and docum research d e r u t c stru arket m e l b a action ? s insight d ia-base to d e m l ia an soc ch be tailored c w o H • sear e r t cases? e k e r s u ma t n t clie differen g learnin r for e n i h c matte e s ma • How d, oand why does it work research? market
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
3
Social-media research can support both traditional market research goals and PR functions. Traditional Market Research through Social Media Listening
Communications Functions through Social Media Monitoring
• Thoughts and opinions about products and brands
• Consumer complaints and product malfunctions
• Market awareness of products or brands
• Crisis monitoring and response
• Purchase triggers
• Reputation management
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
4
These two use cases – market research and communications – closely align with two services.
Social Listening
Social Media Monitoring
When what matters most is understanding a consumer segment or market.
When what matters most is delivering customer service, navigating a crisis situation or detecting reputation threats.
Goal is to understand a population “out there” in the world.
Goal is comprehensive, real time coverage.
Higher tolerance for missing content.
Higher tolerance for irrelevant content.
Lower tolerance for irrelevant content.
Lower tolerance for missing content.
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
5
The Social Media Research Process: From Raw Data to Insights
1. Data Collection
2. Data Enrichment
3. Analysis & Insight Generation
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
6
Stage 1: Social Data Collection Primary Goal: 1. Data Collection
2. Data Enrichment
Identify and acquire the data that can answer your business questions.
Primary Challenges:
3. Analysis & Insight Generation
1. Pull in relevant data and metadata 2. Coverage of appropriate social media channels 3. Eliminate spam and irrelevant content.
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
Stage 2: Data Enrichment Primary Goal: 2. Data Enrichment
1. Data Collection
Implement document- and subdocument-level enrichments like topic, consumer segment, emotion and sentiment. Primary Challenges:
3. Analysis & Insight Generation
1. Data normalization 2. Classification 3. Scalability
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
8
Stage 3: Analysis & Insight Generation Primary Goal: 2. Data Enrichment
1. Data Collection
Connect the dots between a suite of metrics and data points in order to reach sound strategic conclusions. Primary Challenges:
3. Analysis & Insight Generation
1. Reliability 2. Strategic Value
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
9
Social media represent a massive compendium of documents…
Approx. Monthly Volume
Furthermore…
Blogs
30 million new posts
On-site comments and social cues and sharing
Facebook
1.8 billion status updates
Social cues (e.g., “likes”) and comments
Twitter
4 billion tweets
Social cues like favoriting and flagging other users
400 million social actions
240 years of video content uploaded each month
Social Media Channel
YouTube
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
10
…and these documents are all of different types.
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
11
Harvesting Data and Metadata from Social Media Documents: A Tweet Dissected
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
12
Harvesting Data and Metadata from Social Media Documents: A Tweet Dissected Datapoints: • Author Name • Text • Publication Date • Some hashtags
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
13
Harvesting Data and Metadata from Social Media Documents: A Tweet Dissected Metadata: • Person or tweet that a tweet is in reply to • Follower count of author • Times retweeted • Times favorited • Author description
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
14
Sorting Social Metadata
A B
Tweets that contain #Ford in the text.
C
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
15
Relevancy as a Sorting Task… Irrelevant Documents
All Social Media Documents
• Spam • Documents not in target language (e.g., not English)
All Documents Containing Your Boolean Query
• Contain keyword but not relevant to client question Relevant Documents
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
16
Data Enrichment: What Should We Measure?
Metric
Explanation
Sentiment
Does the author make a negative or positive point about a product or brand?
Topics
What topic is the author talking about the product or brand in relation to?
Purchase Stage
Has the author of a document already purchased the product when writing about it online?
Consumer Segmentation
What segment is the document’s author from?
Emotions
What emotions do authors express toward the target brand or product?
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
17
Data Enrichment: What Should We Measure?
Metric
Sorting Categories
Sentiment
Positive, negative, neutral
Topics
Pre-selected topic and unexpected topics
Purchase Stage
Before making a purchase or after.
Consumer Segmentation
Young male, middle-aged woman, etc.
Emotions
Joy, anticipation, surprise, fear, etc.
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
18
How can we implement the sorting tasks we’ve discussed so far?
Machine Sorters
Human Sorters
Sorting Tasks Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
19
Q: How do you know when a computer is correct?
A: The same way you know that a human is correct:
“I know it when I see it…”
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
20
Establishing A Basis for How Well Humans Agree With One Another Example 1: Inter-Coder Agreement on Sentiment
Example 2: Inter-Coder Agreement on Emotion
Item
Coder 1
Coder 2
Tweet
Coder 1
Coder 2
1
Positive
Positive
I do not like the cats with thumbs “advert”
Disgust
Anger
2
Positive
Neutral
Trust
3
Neutral
Neutral
I say that video is real, definitely.
No Emotion Expressed
4
Negative
Positive
etc.
…
…
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
21
Using Human Parallel Coding to Establish Gold Standards
Confusion Matrix: Human as Gold Standard
POSITIVE
NEGATIVE
NEUTRAL
TOTAL
POSITIVE
365
24
159
548
NEGATIVE
57
81
65
203
NEUTRAL
274
60
415
749
TOTAL
696
165
639
1500
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
Raw Accuracy: 61.5%
22
Using A Credit Matrix to Create Improved Measurement Credit Matrix
POSITIVE
NEGATIVE
NEUTRAL
POSITIVE
100%
0%
50%
NEGATIVE
0%
100%
50%
NEUTRAL
50%
50%
100% Partial Credit Figure of Merit: 82.3%
Confusion Matrix: Human 1 as Gold Standard
POSITIVE
NEGATIVE
NEUTRAL
POSITIVE
365
24
159
NEGATIVE
57
81
65
NEUTRAL
274
60
415
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
23
But how does the machine learn?
1. Collection of Human Annotated Data
2. Machine ingests coded data and finds patterns in each category classification
3. Machine applies model from step two on raw data. Results are compared to human coding of same material.
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
24
In conclusion…. e case s u r u o about y ch or c i f i c e p • Get sit market resear – is ring? monito to your k a e p s at trics th s e m t c • Seleiness question bus e e.g., th ( a t a d f so ntage o e say), but al a v d a • Takeof what peopl e additional th text a (e.g., most social t a d a t t me tion tha nts contain). a m r o f in e docum a i d e m ll ines wi s, h c a m n at tand th rfect” decisio s r e d n • U ly make “pe ations and c rare classifi herently e s u a bec them. p are in l e s t h n n e judgm e, but we ca tiv subjec
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
25
Q&A
Ray Poynter VCU, Vision Critical
Jasper Snyder Converseon
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
Thank you Jasper Snyder, VP, Converseon
[email protected]
Speaker: Jasper Snyder, Converseon NewMR Social Media Research Event, 9 October 2012, Session 2
© 2011 Converseon Inc. Proprietary and Confidential
27
A Presenta*on from the NewMR “Social Media Research” Event 9 October, 2012
Enriching Social Data for Market Research Jasper Snyder, Converseon
All copyright owned by The Future Place and the presenters of the material For more informa:on about NewMR events visit newmr.org