What is effect size?

---

Calculating and Interpreting Effect Size for your CME activities Jason J. Olivieri, MPH Director, Outcome Services, Imedex, LLC* Ben Collins Manager, Outcome Services, Imedex, LLC*

*Imedex LLC is a CME-accredited MEC and outcome services provider

Target Audience • Those with little or no experience with effect size • Those who’ve been working with effect size and have questions about certain conditions

Outline • What can we do with effect size measures? • How do we calculate effect size? • Q & A (20 minutes)

What Can You Do with Effect Size? • Summarize the educational effectiveness of a CME activity • Conduct aggregate analysis of CME effectiveness by format, topic, etc • Determine benchmarks for CME effectiveness

EIS for live CME activities by type: June 2010 – Sept 2013

EIS for live CME activities by type: June 2010 – Sept 2013 large*

medium*

small*

*Cohen. J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates.

EIS for live CME activities by type: June 2010 – Sept 2013 competence1

knowledge2

1. Drexel et al. Int J Chron Obstruct Pulmon Dis 2011; 6: 297–307. 2. Mansouri & Lockyer. J Contin Educ Health Prof 2007;27:6-15. .

EIS for live CME activities by type: June 2010 – Sept 2013

Expected magnitude of effect on knowledge outcome (.43 .65)

Using effect size allows CME providers to be more empirical with their CME • What’s a good CME effect size? • Why is my effect size lower than benchmarks? • How do my outcome methods / questions affect my effect size? • What affect does adding / subtracting certain CME elements have on my effect size? • Why is one format more / less effective than another?

How does it work?

CME activity

EOM plan

Data

Didactic presentation followed by case-based discussion

Pre- vs. post-activity assessment via ARS or paper survey

8 case vignette or clinical practice strategy questions

Example ARS question (case vignette) Frontline therapy for a former smoker with symptomatic advanced stage adenocarcinoma of the lung (EGFR+)

Example paper survey question (clinical practice strategy) Using bevacizumab-based combo therapy for non-squamous NSCLC

CME activity

Didactic presentation followed by case-based discussion

How do we summarize this data?

EOM plan

Data

Pre- vs. post-activity assessment via ARS or paper survey

8 case vignette questions or clinical practice strategies

And how do we then compare this result to results of other activities?

What is effect size? •Quantifies the magnitude of effect (maximum expected range: -3 to +3) •Difference in means (e.g., pre-test and post-test) divided by the square root of the pooled-group variances (Cohen’s d) •Enables the comparison of CME effects across activities on a common dimensionless scale •Calculated from comparison data (e.g., pre/post, post/control) linked directly to CME content −Knowledge questions −Case vignettes −Self-reported frequency of use of key clinical practice strategies

How is it calculated?

Calculating effect size • Can be done using only MS Excel® and free, online resources • Approach dependent upon variable type: – ordinal (e.g., clinical practice strategy) – categorical (e.g., case vignette)

Example paper survey question (clinical practice strategy) Using bevacizumab-based combo therapy for non-squamous NSCLC

Calculating effect size for an ordinal variable (e.g., clinical practice strategy) • Calculate average and standard deviation for each group (e.g., pre- and post-activity) – Pre-activity: mean (SD) = 2.5 (1.3) – Post-activity: mean (SD) = 3.1 (1.2) • Plug these values into an online calculator (http://www.uccs.edu/~lbecker/)

How is effect size interpreted? Ordinal variable example Clinical practice strategy (CPS) use rating (1= never, 5 = always) Pre-test

Post-test

Effect size

CPS #1

Standard deviation

CPS #1

Standard deviation

Cohen’s d

2.5

œ1.3

3.1

œ1.2

.48

*d = .2 (small effect), d = .5 (medium effect), d = .8 (large effect)

How big was the educational effect? How does the effect compare to other activities? Cohen (1988): .2 = small, .5 = medium, .8 = large Wolf (1986): .25 = educationally significant, .50 = clinically significant

Expressed in standard deviation units: The average score of a post-test respondent was .48 standard deviations above the average score of a pre-test respondent Effect sizes are proportional (.48 is twice as much effect as .24)

Example ARS question (case vignette) Frontline therapy for a former smoker with symptomatic advanced stage adenocarcinoma of the lung (EGFR+)

Calculating effect size for a categorical variable (e.g., case vignette) • Determine % correct for each group (e.g., pre- and postactivity) – Pre-activity: .51*65 = 33 correct / 32 incorrect – Post-activity: .75*65 = 49 correct / 26 incorrect • Plug these values into online calculator (www.vassarstats.net) – Click “frequency data” – Click “Chi-Square, Cramer’s V and Lambda”

Calculating effect size for a categorical variable, continued • Visit www.lyonsmorris.com/ma1/index.cfm • Select “Correlation coefficient (r) to Effect Size” • Enter Cramer’s V (.1474)

Determining overall effect size for a CME activity • Take average of all effect sizes (Cohen’s d) at each outcome level – Report overall effect size if you have at least four questions – Only combine effect sizes for the same question type • Keep a database of effect sizes to develop benchmarks

How effective was your live CME in 2013? 10 live CME activities

8 ARS questions per activity

80 slides

10 effect sizes

One summary effect size for live CME

Overall EIS (by format) June 2010 – Sept 2013 1Cohen

J 1988. Small effect = .2, Medium effect = .5, Large effect = .8

2Mansouri

& Lockyer 2007. Knowledge effect size = .6 4Casebeer 3Drexel

et al 2011. Competence effect size = .85

et al 2010. Knowledge effect size (eLearning) = .82

n=9 1. Cohen. J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates. 2. Mansouri & Lockyer. J Contin Educ Health Prof 2007;27:6-15. 3. Drexel et al. Int J Chron Obstruct Pulmon Dis 2011; 6: 297–307. 4. Casebeer et al. BMC Med Educ 2010;10: 42.

Effect Size Summary •Summarizes effectiveness of individual CME activities and allows for aggregate comparisons •Easy to calculate •Uses:

• Evaluate CME Mission • Make improvement plans and track progress • Establish effectiveness benchmarks • Communicate effectiveness to sponsors & ACCME

Questions? Resource: www.assessCME.wordpress.com