Kevin Karty.pptx


[PDF]Kevin Karty.pptx - Rackcdn.comb219de03d4fa586d9adb-0e04f025bcde650ce6d504251ebbc914.r60.cf3.rackcdn.co...

1 downloads 169 Views 7MB Size

A  Presenta*on  from  The  NewMR  “Advances  in   Quan*ta*ve  Research”  Event   19  September,  2012  

Taking  Nothing  Seriously:  A  Review  of  Approaches  to   Modeling  the  ‘None’  Op=on     Kevin  D.  Karty  PhD,  Affinnova        

Event  sponsored  by  Affinnova  

All  copyright  owned  by  The  Future  Place  and  the  presenters  of  the  material   For  more  informa=on  about  Affinnova  visit  www.affinnova.com   For  more  informa=on  about  NewMR  events  visit  newmr.org  

Taking Nothing Seriously:

Much Ado About Nothing

Kevin Karty Vice President, Analytics Bin Yu Director, Analytics www.affinnova.com

Kevin Karty, Affinnova, USA

NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

The Problem Nothing Usually Doesn’t Mean Very Much

•  In a conventional choice base conjoint study, we typically show something like the example below •  In actual implementation, the expression of None may be conducted in different ways… –  A choice option with equal size and consistent location (as below) –  A bottom bar below the other options (in a different color)

•  We’ve tried lots of different visual treatments, always with the same result – people just don’t pick “None” enough

3

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

The Problem

•  Over 60% of all respondents use “None” on fewer than 20% of all choice tasks – and nearly 40% of respondents NEVER used “None” •  This study offered strong insights, but the variation in simulated store usage across many formats was small even when major changes were made to inputs

4

60% of respondents selected ‘None’ on fewer than 5 out of 25 choice sets 40% of respondents never selected ‘None’ on any of 25 choice sets

Number of Times NONE Selected

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Cumulative Percentage of Users Who Selected None N or Fewer Times

•  Here is an example of the usage of “None” from a standard interface for a complex (and lengthy) study on new retail store layouts

Number of Users Who Selected “None”

Nothing Usually Doesn’t Mean Very Much

The Problem Nothing Usually Doesn’t Mean Very Much

Frequency of “None” Selections Across Respondents

This study proved useful for a few reasons… •  There was no true answer, since the store layout was hypothetical – a directional simulator was sufficient •  “None” usage INCREASED throughout the duration of the respondent experience •  Respondents became more discriminating as they saw things they liked in previous choice tasks –  Actually had negative aggregate utility for some features due to this displacement effect

5

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Initial Exploration of Alternatives Using Multiple “None’s” •  Overstatement is a recognized problem in behavioral economics, where “None” is interpreted as the “external good” –  We know from experimental behavioral economics that when consumers compare a thing to an unspecified external good, they are more likely to buy the thing

•  This can be improved by having consumers compare a thing to specific goods – they are usually less likely to buy the thing

None

None

–  However, this can be twisted around by offering a comparison to an asymmetrically inferior good (e.g. asymmetric dominance)

•  So we tried to solve the “None” problem by offering multiple “None” options…

None

•  Our question: If we offer multiple “None” options, each describing a different class of external goods from a related category, would this help?

None

6

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Initial Exploration of Alternatives

–  Current line had low market penetration…

•  To increase the sensitivity to the “external good”, we created five “None” buttons –  One generic, and four specifying other types of items in adjacent categories –  Compared to similar studies we saw increased “None” usage, but not enough to solve the problem…

7

70% of respondents selected ‘None’ on fewer than 5 out of 20 choice sets

Number of Times NONE Selected

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Cumulative Percentage of Users Who Selected “None”N or Fewer Times

•  Goal in this study was to optimize a new line for a snacking product in a niche subcategory

Number of Users Who Selected “None”

Example: Using Multiple “None’s”

Initial Exploration of Alternatives •  Interestingly, “None” usages (for all “None” buttons as an aggregate) did NOT increase as respondents progressed through the exercise… •  However, “None” usage was still unrealistically low

8

Frequency of “None” Selections Across Respondents

Using Multiple “None’s”

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Initial Exploration of Alternatives Using Multiple “None’s” •  Actual product penetration among targeted grocery shoppers was <20% even with substantial price promotion and awareness, and decent product placement •  However, when we simulated the line of products (without any competitors present), our projected purchase rate is 84% even when we include all five “None’s” in the simulation –  When we only include the generic “None”, our projected purchase rate is 90%, which is close to our result from a similar study in an adjacent category for the same client where we did not include multiple “None” buttons

•  We can of course “calibrate” the data in any number of ways, but these calibrations always create problems… –  For example, if we increase the utility of None by a fixed amount for each respondent until our simulations match HH penetration, we end up arbitrarily assigning some number of people who NEVER said “None” to “None” (we need simulated Penetration <20%, but >40% of people never used any of the “None” buttons ever) –  Other calibration methods incur different problems

•  All of this is because we fundamentally have BAD DATA: When consumers tell us they would buy something (rather than nothing) they don’t really mean it!

9

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Initial Exploration of Alternatives Dual Response è Ordinal Scale Sequential Response •  One possible explanation for the under-usage of “None” was lack of visibility or focus •  We know that Dual Response (yes / no) can increase “None” usage…but generally not enough •  If we expand the standard Dual Response (yes / no) to a 5 point scale sequential response, can we increase “None” usage further? •  To answer this question, we fielded two separate legs in a fairly standard conjoint study –  Category: online payment processing product –  6 attributes: 8 x 7 x 6 x 6 x 4 x 3 •  (The attribute with 3 levels was sub-branding)

10

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Initial Exploration of Alternatives Example of Sequential Response •  The dual response interface picks up ~50% “None” usage… •  The “Probably” response in a 5 point scale purchase question yielded ~55% “None” usage •  The “Definitely” response yielded ~85% “None” usage •  In this case, the client expected the true conversion rate to be very low…(less than 100% of those who indicated they would “Definitely” use the product)

11

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

“None” Problem: Really Two Separate Problems Overstatement Consumers are far more likely to give us a thumbs up on a product than they are to actually buy it

12

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

These Two Problems Have Solutions… Overstatement Separate choice task from confirmation task Use a scale that can create separation in responses based on commitment

13

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

“None” Problem: Really Two Separate Problems Lack of a Valid Reference Point to Anchor the External Good

When consumers are considering the alternative to a purchase, they are rarely cognizant of the real alternatives

14

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

These Two Problems Have Solutions… Lack of a Valid Reference Point to Anchor the External Good Give consumers a valid anchor point to compare against when assessing purchase intent for a new product or service

15

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Our Experimental Setup •  Get a real apples-to-apples comparison… –  Initially we had hoped to conduct 5 parallel studies –  No [research] plan survives first contact with the enemy CFO

Three •  Five Parallel Studies –  Traditional “None” as extra option –  Dual Response “None” (yes / no) –  Sequential Response 5 Point Scale –  Multiple “None” Buttons –  Sequential Response 5 Point Scale With Dynamic Anchoring

16

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Choosing the Category… Selected a category that we were already studying for other internal research…

Our Category…

•  We already owned scanner data •  Growing category due to increased usage •  Familiar to everyone who is a potential user •  High rate of product innovation in last decade •  We had developed a set of new product concepts in-house with clear functional benefits in a growing segment of the market

17

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Our Cat Litter Study “Client” Objective: Assess potential of a set of new product concepts targeting the natural / green segment among cat litter purchasers –  We have already optimized these concept propositions for different market sub-segments

Challenge •  We would prefer NOT to represent the entire category, since the total number of products is quite large and this becomes costly •  Our goal is to predict which concept will perform best in market, and ideally by how much –  We want to represent real packaging, BUT we don’t have real pack-level creative for our new concept yet –  We do not want the complexity of doing a full brand / pricing study – need to keep it simple

•  This places heavy demands on our “None” option

18

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Our New Cat Litter Concepts Five New Concepts •  One starting point (manager’s best guess) •  Four concepts developed from an earlier optimization phase with dozens of names, benefits, reasons to believe, etc. that we “borrowed” from existing products Starting Point

19

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Six Included Competitors… •  Three main brands, three natural brands •  Less than 50% of the market by sales

20

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Interface I Standard “None” Button

Standard NONE represented as the fourth item in a choice set in the fixed location with equal size stimulus

21

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Interface II Dual Response Yes / No Follow-Up

After making a choice, indicate whether you would actually purchase the product (yes or no response)

22

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Interface III Sequential 5-Point Scale Follow-Up with Dynamic BMO Piping

•  After making a choice, indicate how likely you would buy the product instead of the brand you buy most often (5-point scale response) •  Brand-Most-Often is piped from a screener question

23

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Model Definitions Model

Interface

Definition of NONE

Standard NONE Model

NONE button

NOTA button

Dual Response Yes / No Model

Yes / No follow-up

No in the Yes / No follow-up question

Sequential Bottom 3 Model

Sequential 5-point scale follow-up with dynamic BMO piping

Bottom 3 choices in the 5-point scale follow-up question

Sequential Bottom 4 Model

Sequential 5-point scale follow-up with dynamic BMO piping

Bottom 4 choices in the 5-point scale follow-up question

n = 300 for all models

24

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Initial Summary Data Proportion of “None” selections on each choice set (across all respondents)

25

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

But Does It Matter? •  We can begin by comparing share projections for the products that are already in market to actual market sales •  We adjust for distribution by multiplying projected share by ACV, and renormalizing –  “None” is given the benefit of the doubt, and assumed to have 100% distribution Current Market Share (by Volume)

Distribution

Arm & Hammer Scoopable

15.2%

74.9%

Fresh Step Scoopable

11.8%

78.1%

Tidy Cats Scoopable

17.6%

77.9%

Arm & Hammer Essentials

0.6%

43.4%

Feline Pine

1.1%

47.0%

World’s Best Cat Litter

0.5%

44.8%

53.3%

100.0%

Brands

NONE (including Other Brands)

26

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Projections vs. Actual •  Unadjusted for distribution, Sequential Bottom 4 Model comes closest to matching reality •  All of the models over-predict the niche products substantially, with Standard NONE Model being the worst

27

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Projections vs. Actual •  Adjusted for distribution, Sequential Bottom 4 Model still comes closest to matching reality, but now over-predicts “None” slightly –  Was giving “None” 100% distribution too generous?

•  All of the models over-predict the niche products, but Sequential Bottom 4 Model does so only modestly –  We would expect some over-prediction due to inferior shelf placement and many fewer facings even when the product is in distribution

•  Note that distribution adjustments were not done at individual respondent level, so “lost share” was re-allocated proportionately

28

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

What If We Use Specific Effect Adjustments? •  Applied specific effect adjustments to match the predicted “None” share to the share of “Other Brands” in the market •  Identified respondents with highest individual utility for “None” è Assigned them to None

Current Brand Used Most Often

Assigned to “None” by Standard Model (with Specific Effects Adjustments)?

29

NOT Assigned to “None”

Assigned to “None”

8%

14%

Fresh Step Scoopable

20%

20%

Tidy Cats Scoopable

25%

11%

Arm & Hammer Essentials

3%

2%

Feline Pine

3%

1%

World’s Best Cat Litter

0%

3%

42%

49%

154

146

Most Often Brand Arm & Hammer Scoopable

Other Brands (“None”) Sample Size

Among respondents assigned to an included brand instead of “None”, 42% used an “Other Brand” most often… Among respondents assigned to “None”, only 49% used an “Other Brand” most often…

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

What If We Use Specific Effect Adjustments? •  Repeating this analysis for the other models, we see very weak linkage between allocation to “None” in the first two models and stated brand most often usage •  Slight improvement in Model 3 •  Substantial improvement in Model 4

Current Brand Used Most Often

Standard NONE Model

30

Most Often Brand

A

Arm & Hammer Scoopable

Dual Response Y / N Model

Sequential Bottom 3 Model

B

Sequential Bottom 4 Model

B

A

B

A

A

8%

14%

13%

10%

12%

11%

14%

9%

Fresh Step Scoopable

20%

20%

11%

14%

26%

17%

28%

15%

Tidy Cats Scoopable

25%

11%

21%

20%

21%

22%

28%

15%

Arm & Hammer Essentials

3%

2%

9%

3%

4%

5%

7%

1%

Feline Pine

3%

1%

0%

1%

1%

0%

0%

1%

World’s Best Cat Litter

0%

3%

1%

4%

1%

0%

1%

1%

Other Brands (“None”)

42%

49%

45%

49%

35%

46%

23%

58%

Sample Size

154

146

159

141

144

156

149

151

Segment A: Respondents for whom the highest utility is on one of the products included in the study Segment B: Respondents for whom the highest utility is on “None”

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

B

Should We Believe “None” Allocations? Possible Criteria: How often do the models allocate individuals to “None” even when they never selected “None”? Standard NONE Model

Dual Response Y / N Model

Sequential Bottom 3 Model

Sequential Bottom 4 Model

Number of Respondents Who Never Selected “None”

202

141

105

22

Percentage Allocated to “None

30%

14%

4%

0%

Model

31

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Should We Believe “None” Allocations? Possible Criteria: What impact does it have on our new product projections? •  Here we simulate adding the new concepts (without adjusting for distribution) to the mix of existing products and “None” •  The first three models yield similar answers •  All models yield the same ordering •  The Sequential Bottom 4 Model yields lower overall projections, and a substantially lower gap between Concept 1 and the Starting Point Standard NONE Model

Dual Response Y / N Model

Sequential Bottom 3 Model

Concept 1 vs. Comp.

17%

17%

18%

8%

Concept 2 vs. Comp.

17%

16%

17%

8%

Concept 3 vs. Comp.

17%

14%

14%

7%

Concept 4 vs. Comp.

15%

13%

16%

8%

Starting Point vs. Comp.

11%

13%

12%

6%

Model

32

Sequential Bottom 4 Model

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Some Theoretical Concerns •  While Sequential Bottom 4 Model predicts well, the “None” share is a bit too high •  Also, if respondents select “None” 4 of every 5 choice tasks, we lose a lot of relative preference information on second-best products… •  On the other hand, who cares? Why should the relative preference between Product A and Product B matter if the consumer would never purchase either of them? ALTERNATIVE: Formal Sequential Response Model

33

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Formal Sequential Response Model •  Assume that both data types (choice data and sequential response scale data) are consistent manifestations of the same underlying dynamic… –  That is, if we had an infinite amount of data of either type, we would arrive at the same parameter estimates for part worth utilities

•  Net result –  Uses more information –  Allows us to apply “normative weights” to Top Box and Second Box Purchase Interest Likert Scale Scores

Discrete Choice Data

Unified Latent Preference Model: Choice Likelihood x Scale Response Likelihood

34

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

One Advantage •  Using a formal model allows us to predict, for each person, the specific scale response we would observe •  This allows us to apply normative weights to each response •  For example: –  Definitely Would Purchase = 100% –  Probably Would Purchase = 20%

35

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Remaining Questions •  Does sequential response work as well if we do NOT pipe in individual-specific information? –  In our parallel testing we included conventional approaches and the most customized sequential response we could imagine

•  What normative weights are appropriate to apply to the top box and second box in a formal sequential response model? –  Do weights differ by category? –  What about the level of refinement of the stimulus?

•  How well does this approach apply to main effects models? –  For example, new product projections for bundled telecommunications or finance products?

36

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

Thank You

37

Q&A

Q&A

Ray Poynter VCU, Vision Critical

38

Kevin Karty Affinnova

Kevin Karty, Affinnova, USA NewMR Advances in Quantitative Research Event, 19 September 2012, Session 2

A  Presenta*on  from  The  NewMR  “Advances  in   Quan*ta*ve  Research”  Event   19  September,  2012  

Taking  Nothing  Seriously:  A  Review  of  Approaches  to   Modeling  the  ‘None’  Op=on     Kevin  D.  Karty  PhD,  Affinnova        

Event  sponsored  by  Affinnova  

All  copyright  owned  by  The  Future  Place  and  the  presenters  of  the  material   For  more  informa=on  about  Affinnova  visit  www.affinnova.com   For  more  informa=on  about  NewMR  events  visit  newmr.org