Chapter 11
How Do Chemistry Educators View Items That Test Conceptual Understanding? Downloaded by PURDUE UNIV on November 29, 2016 | http://pubs.acs.org Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011
Cynthia Luxford1 and Thomas Holme*,2 1Department
of Chemistry and Biochemistry, Texas State University, San Marcos, Texas 78666, United States 2Department of Chemistry, Iowa State University, Ames, Iowa 50011, United States *E-mail:
[email protected]
The ability to test student understanding of chemistry faces a number of challenges that result from the multi-faceted nature of the science. Students are expected to be capable of dealing with quantitative aspects and much of the introductory content of college chemistry course is geared towards this component. Over the past 3 decades, interest in the achievement of conceptual understanding by students has increased. Attempts have been made to allow demonstration of such knowledge through traditional written exams. However, the nature of what constitutes effective test questions for this task has received less attention. A survey on testing for conceptual understanding was given to roughly 13,000 general chemistry instructors across the nation. Responses from 1800 faculty were recorded and the responses of 1519 faculty members were analyzed after cleaning the dataset. Faculty were asked to determine whether a series of 6 questions similar to ACS exam questions were measuring conceptual understanding. Immediately after their rankings, they were asked to generate their own definition for the term ‘conceptual understanding’. Results indicate that there are some differences among chemistry instructors about the nature of testing conceptual understanding. In particular, depending on what components of conceptual understanding a teacher includes in their definition, items are perceived differently in terms of what they test.
© 2016 American Chemical Society Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.
Downloaded by PURDUE UNIV on November 29, 2016 | http://pubs.acs.org Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011
Introduction With a seminal publication in 1987, Nurrenburn and Pickering (1) ushered in an era of intense research interest in the apparent disparity between conceptual and quantitative learning among general chemistry students. In the nearly 30 years since this paper was published, dozens of studies have considered similar disparities and often found a number of students who were more capable of answering test items designed to measure quantitative reasoning than conceptual understanding (2–18). A key outcome of this work has been efforts to devise test questions that measure conceptual understanding. While a significant amount of the effort associated with devising conceptual test items has centered on the particulate nature of matter (PNOM) and representations of that level of chemistry (2, 3, 6), there are a number of ways that conceptual understanding has been tested. For example, for topics such as thermodynamic enthalpy change, the use of PNOM diagrams is less central to conceptual understanding (19). An enhanced classification scheme referred to the Expanded Framework for Analyzing General Chemistry Exams (EFAGCE) has been proposed (20) and used in subsequent work (21, 22) to further break down test items in terms of algorithmic, conceptual and recall/definition questions. In the US, one important venue for the development of testing materials is the ACS Exams Institute. There have been several versions of the Conceptual General Chemistry Exam (23–26) produced starting in 1996 which was developed to test conceptual content knowledge as defined by each individual exam committee. In addition, a unique form of exam, called the paired-question exam (27–30), was produced starting in 1997 that specifically addresses topics through both algorithmic and conceptual items. Analysis of the more recent version of the paired-question exam have established psychometric attributes of both the algorithmic and conceptual items (31, 32). In particular, these analyses found that the psychometrics for the conceptual understanding question is not generally lower than the paired algorithmic question. Regardless of progress made in testing related to conceptual learning, there are persistent calls to better identify what defines important conceptual ideas in chemistry, particularly as they relate to general chemistry. Some definitions focus on the cognitive processes required of students. Cracolice (33) has argued that a conceptual problem requires students to use knowledge of a concept behind a problem rather than relying on a memorized recipe for accomplishing an exercise. By contrast, when the first action of a student upon encountering a problem is to decide what recipe (set of mathematical steps) to use, the problem is best defined as algorithmic (34). Recently (35), analysis of over 1400 answers to an open response item in a national survey has given rise to a consensus level definition of conceptual understanding in the minds of chemistry educators. This multipart definition includes earlier published ideas but adds additional features to the articulation of the meaning of conceptual understanding, such that students may demonstrate different facets of the construct. Thus a current “community” definition of conceptual understanding is:
196 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.
“A student who demonstrates conceptual understanding can: -
Downloaded by PURDUE UNIV on November 29, 2016 | http://pubs.acs.org Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011
-
-
Apply core chemistry ideas to chemical situations that are novel to the student (Transfer). Reason about core chemistry ideas using skills that go beyond mere rote memorization or algorithmic problem solving (Depth). Translate across scales and representations (Translate). Demonstrate the critical thinking and reasoning involved in solving problems including laboratory measurement (Problem Solve). Expand situational knowledge to predict and/or explain behavior of chemical systems (Predict) (35).”
While this definition is capable of capturing the definitions as articulated by a large majority of the sample in the survey, there are very few participants who included all five components in their open response. Thus, an additional important aspect of understanding the nature of chemistry instructor understanding of student learning about chemical concepts lies in determining how those instructors apply their definitions to the categorization of chemistry test items. This component of the earlier survey work is reported here.
Survey Information The American Chemical Society Examinations Institute (ACS-EI) has conducted survey research work related to assessment for several years. Tests produced by ACS-EI are examples of “grass roots” test development. Each new exam committee determines the most important variables associated with the test to be developed, including the content coverage that the exam will have (36). When the committee that developed the most recent version of the General Chemistry Conceptual Exam (26) first met, in addition to determining chemistry content coverage, they were interested in determining a sense from the chemistry education community about their needs and expectations for the conceptual exam. As a result, in August of 2013, a national survey was conducted that sought to answer several questions related to testing student conceptual understanding. The committee itself created several prototype test items (referred to here as “mock” ACS Exam items) that they expected would provide a range of reactions from educators in terms of whether or not those items test conceptual understanding. While there were additional traits contained in the survey this component is the subject of the current report. Committee members also served as participants for pilot testing of the survey before the full scale data collection was undertaken. Ultimately, the survey contained six sections that participants completed sequentially. They were:
197 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.
• • •
Downloaded by PURDUE UNIV on November 29, 2016 | http://pubs.acs.org Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011
• • •
Part 1: Experience teaching general chemistry Part 2: Importance of topics for developing conceptual understanding Part 3: Evaluating whether mock ACS Exam questions are conceptual questions Part 4: Defining ‘conceptual understanding’ Part 5: Student familiarity with visual diagrams and representations Part 6: Course objectives and goals
The material in Part 4 provided the information for prior reports (35) from this survey research, and this paper will focus largely on information from Part 3, where instructors indicated whether or not they believed specific mock exam items test student conceptual understanding.
Survey Data Collection and Participant Demographics Participants were recruited from a regularly maintained database of contact information, including email addresses, for chemistry instructors in the US. ACS-EI does a full-scale revision of this database at least every 2 years. It contains instructors from 2-year colleges, primarily undergraduate institutions and research universities. Roughly 13,000 participants were sent emails requesting their participation in this study. As part of the recruiting process, participants could elect to sign up for random drawings of iPad tablets. Approximately 1800 chemistry instructors logged into the survey system, which was housed within a Survey Monkey account. For the purposes of this study, all who logged into the survey were asked if they had taught general chemistry in the past 5 years and provided the opportunity to give informed consent for their participation. This protocoal was approved as exempt by the Iowa State Institutional Review Board (13-361). A small number of participants did not complete the entire survey. The full sample for analysis included 1519 participants. The data was cleaned to include only the 1296 participants who (a) indicated they had taught general chemistry in the past 5 years; (b) signed the informed consent document and (c) completed most of the items on the survey. The distribution of participants in terms of years teaching is fairly broad as seen in Table 1.
Table 1. Distribution of participants by years teaching Years 0-4 Teaching
5-9
10-14
15-19
20-24
25-29
30-34
35-39
>40
Percent
21.5
18.9
13.7
13.5
7.8
4.5
2.6
3.2
13.0
198 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.
Downloaded by PURDUE UNIV on November 29, 2016 | http://pubs.acs.org Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011
The distribution of participants based on the highest degree offered at their school is summarized in Figure 1 which shows the sample includes instructors from a wide range of environments in general chemistry is taught.
Figure 1. Distribution of school type among participants. Given the variety of institutions from which participants were drawn, it is not surprising that the reported class sizes for general chemistry also varied. Class sizes that were less than 100 students were reported by 71% of the participants. Many of these participants had class sizes less than 50 students, with 45% of the total participants being in this category.
Mock ACS Exam Items This paper reports on the judgements made by participants as to whether or not items designed to mimic those found on ACS Exams test conceptual understanding. A total of six items were devised with the intention of presenting a range of possibilities for participants to consider. Thus, the General Chemistry Conceptual Exam committee suggested some items they collectively felt were likely to be rated as testing conceptual understanding and others that would be less likely to be so rated. This section of the survey followed immediately after queries about the relative importance of various topics in the teaching of general chemistry, and prior to the item asking participants to articulate their definition of conceptual understanding. Results of this latter section have been reported previously (35). The six mock ACS Exam items are provided in Tables 2a and 2b. For each of the 6 mock ACS Exam items, participants were asked to respond on a 5-point likert scale from “Strongly Agree” to “Strongly Disagree”. Specifically, participants were asked to: “Please rate your level of agreement with the statement: If used on an ACS final exam, this question assesses students’ conceptual understanding in General Chemistry.”
199 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.
Downloaded by PURDUE UNIV on November 29, 2016 | http://pubs.acs.org Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011
Table 2a. Mock ACS Exam items 1 through 4 from the survey
200 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.
Downloaded by PURDUE UNIV on November 29, 2016 | http://pubs.acs.org Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011
Table 2b. Mock ACS Exam items 5 and 6 from the survey
Item #1 is about connecting Lewis structures and VSEPR theory and is designated as VSEPR. Item #2 is a LeChâtelier’s Principle question, asking students to understand the direction of response to an external perturbation to an equilibrium system and is designated as “equil”. Item #3 is a form of molar mass calculation, but seeks to heighten the concept of estimation rather than calculator use, and it will be designated as “Molar Mass”. Item #4 is essentially a description of paper chromatography but in this context it is testing the role of intermolecular forces in whether substances dissolve in a particular solvent, and it designated as “IMF”. Item 5 represents what has become a standard means of representing chemistry at the particulate level or a test of student understanding of PNOM and will be designated as “PNOM”. Finally, the expectation of the committee was that item #6 the least likely to be viewed as testing conceptual knowledge. It is a relatively traditional stoichiometry problem and will be designated as “Stoich” here on.
Instructor Impressions of Mock ACS Items As noted earlier, after participants indicated content coverage that they deemed most valuable in general chemistry, they were asked to rate the extent to which the six mock ACS Exam items measure conceptual understanding. The overall responses to this set of items are provided in Table 3. The data at this level reveals that, in the broadest sense, Item 2 (Equil) and 5 (PNOM) are most strongly associated with conceptual understanding. Items 1 (VSEPR) and 4 (IMF) also tilt strongly towards being rated as conceptual. Items 3 (Molar Mass) and 6 (Stoich) have fairly broad views in terms of rating conceptual understanding and neither of these items share the type of consensus seen for items 1, 2, 4 and 5. 201 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.
Downloaded by PURDUE UNIV on November 29, 2016 | http://pubs.acs.org Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011
Table 3. Percentage of faculty rating each Mock ACS Items on a 5-point Likert Scale as measuring students conceptual understanding Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
Strongly Disagree
3.3
1.2
4.2
1.5
2.1
9.0
Disagree
11.2
1.8
20.7
5.1
3.5
23.8
Neutral
13.9
4.8
22.4
10.2
7.7
17.2
Agree
49.3
47.9
35.9
46.5
39.6
26.6
Strongly Agree
22.3
44.4
16.8
36.6
47.1
23.2
Additional insight into how chemistry instructors view the assessment of conceptual understanding is available through the use of the open-response definitions of conceptual understanding. For this analysis, responses have been grouped so that “Strongly Disagree” and “Disagree” are considered together, “Neutral” is considered alone, and “Strongly Agree” and “Agree” are grouped together. As noted in previous work (35), all open response definitions of conceptual understanding have been coded to determine if the participant included any of the five segments of the consensus definition in their response. Some participants include more than one segment, and they are counted in both groups. Thus, it is possible to compare instructors who include a particular idea related to conceptual understanding with all those who do not include that component. Once this grouping was done, statistical analysis using the Mann Whitney U test (37) was performed to determine if any differences are statistically significant between the groups of participants who used each segment and those who did not use the segment in their definition . Because this approach results in multiple comparisons, a post-hoc Bonferoni correction is used to adjust the criterion for significance. This correction means that a p=0.05 significance level requires a p=0.01 value to hold true due to the multiple tests. The results for each item are presented next in a series of stacked-histogram plots with percent of participants in each category: disagree, neutral, agree. There are five groups designated, corresponding to whether or not a definition segment was included in a participant’s definition of conceptual understanding. Finally, the rightmost stacked histogram provides the percentages for the full sample for comparison.
Item 1: VSEPR of One Atom with Two “Central” Atoms The stacked histogram plot for Item 1 is provided in Figure 2. In this case, there is one component of the conceptual definitions where the Mann-Whitney U test show significant difference. Those that express that students be able to translate across scales or representations are less likely to agree that this item is conceptual (p=0.0019).
202 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.
Downloaded by PURDUE UNIV on November 29, 2016 | http://pubs.acs.org Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011
Figure 2. Instructors’ classification of Mock ACS Item 1 (VSEPR) as testing conceptual understanding based on their expressed definitions of ‘conceptual understanding’.
Item 2: Equilibrium and LeChatelier’s Effect The stacked histogram plot for Item 2 is provided in Figure 3. In this case, the Mann-Whitney U test shows no significant differences for any of the different definition segments. This result is unsurprising because a large majority of all participants viewed this items as testing conceptual understanding.
Figure 3. Instructors’ classification of Mock ACS Item 2 (Equil.) as testing conceptual understanding based on their expressed definitions of ‘conceptual understanding’.
203 Schultz et al.; Technology and Assessment Strategies for Improving Student Learning in Chemistry ACS Symposium Series; American Chemical Society: Washington, DC, 2016.
Item 3: Molar Mass Calculation, with Ranges for Answers
Downloaded by PURDUE UNIV on November 29, 2016 | http://pubs.acs.org Publication Date (Web): November 22, 2016 | doi: 10.1021/bk-2016-1235.ch011
The stacked histogram plot for Item 3 is provided in Figure 4. The MannWhitney U test shows significant differences for all of the different definition segments.
Figure 4. Instructors’ classification of Mock ACS Item 3 (Molar Mass) as testing conceptual understanding based on their expressed definitions of ‘conceptual understanding’. Looking at this response pattern in detail, people who include the idea of transferring core chemistry ideas to novel situations are more likely to rank this item as conceptual (p=0.0004). This may imply that the instructors who include the ides of “transfer” in their definition are more likely to expect that having a range available as an answer will be different enough from a customary calculation item that students will perceive a need to do something different. Those who include the idea that conceptual understanding includes depth that goes beyond memorization are less likely to agree that this item tests conceptual understanding (p
