Making Sense of 6th Grade Virginia SOL Results Through Distractor Analysis
Dr. Virginia Lewis
As a former middle school mathematics teacher and a current assistant professor of mathematics education, I have a professional interest in improving the information available to teachers regarding student performance on the Virginia Standards of Learning (SOL) assessments. For teachers to make purposeful changes to instruction "the link between assessment and instruction needs to be made strong and explicit" (National Research Council, 1999, p. 5). For this reason, I chose to examine for my dissertation the results of the Virginia Mathematics SOL Assessments to better understand why students select incorrect choices. The intent of this study was to reveal these reasons so that teachers could better understand how they can use this information along with their own distractor analysis to improve students’ understanding of mathematics and performance on mathematics assessments. The following research questions guided this study.
In order to examine these questions, the Virginia Department of Education (VDOE) provided the frequency and percentage of students who selected each answer choice for each of the fifty items on the 2007 and 2008 sixth, seventh, and eighthgrade released tests. These data are not normally available to teachers and were obtained through a special data request. The VDOE also provided the frequencies by pass level which made it possible to examine the responses of those students who excelled on the assessment (Advanced), passed the assessment but were not considered to have excelled (Proficient), failed but were close to passing (Fail Basic), and Fail Below Basic. This article will focus on the sixthgrade data. The number of students included in this sample and the number of students in the data set in each pass level is included in Table 1.
Table 1. Frequency of students in the data set for pass level by year 

Response Categorization 
2007 
2008 
Advanced 
6,102 
14,866 
Proficient 
15,205 
20,067 
Fail Basic 
10,280 
11,559 
Fail Below Basic 
4,119 
4,064 
Total 
35,706 
50,556 
It is not possible to ask students about their thinking during the standardized assessments, in situations like this Patton (1990) advised the researcher to “do the best job he could in describing the patterns that appeared to him to be present in the data” (p.483). Kloosterman and Lester (2007) used distractor analysis to better understand student performance on individual items on the Main NAEP. Their work served as the inspiration for this project which used a similar framework to analyze the distractor data for the sixth grade SOL assessment. While we cannot know for sure how many students picked each distractor for a particular reason we are able to anticipate student misconceptions and are able to examine distractors on an assessment to identify the most likely reasons students would select those choices. In fact, it is standard practice to purposefully use anticipated student errors to create distractors when writing a multiple choice assessment.
In this study of the sixth grade SOL assessment results, the explanations for why a student would likely select a distractor were generated by a Document Analysis Team (DAT). The DAT was made up of three document analysts who independently examined the tests. These document analysts had “different biases, different strengths, so they can complement each other” (Miles and Huberman, 1994, p. 267). The DAT, which consisted of a sixthgrade mathematics teacher, a sixthgrade special education teacher, and myself, generated possible explanations for why a student would choose each distractor. Each member of the team independently analyzed twentyfive items before meeting as a team to discuss possible explanations for each distractor. Since there were one hundred items on the 2007 and 2008 tests combined, the DAT met four times over approximately two months to analyze all one hundred items. Members of the team generated the same explanation for choosing a distractor about twothirds of the time. When more than one explanation was offered during the team meeting, multiple explanations were accepted as reasonable. In a small number of cases a team member withdrew her explanation from consideration when other explanations were more plausible.
The major categories used to classify each of the possible explanations for each distractor were based on the Mathematical Abilities described in the National Assessment for Educational Progress (NAEP) Framework for their Main NAEP mathematics assessment (National Center for Educational Statistics, 2011). Once the DAT produced a listing of possible explanations for each distractor I categorized each of the explanations as Conceptual Understanding (Conceptual), Procedural Knowledge (Procedural), or Problem Solving error. After all the distractor explanations were categorized, these categorizations were coded alongside the frequencies provided by the VDOE to look for patterns in the data.
In both 2007 and 2008, Conceptual and Problem Solving explanations for possible error were much more common than Procedural errors. Table 2 shows that Conceptual and Problem Solving errors most likely accounted for nearly onefourth of answer choices selected on both assessments.
An examination of the percentage of incorrect responses, displayed in Table 3, revealed that more than 80% of incorrect responses, 84% for 2007 and 86% for 2008, were likely the result of Conceptual or Problem Solving errors. This demonstrates the much smaller impact of Procedural Errors on student performance on this assessment, 14% for 2007 and 13% for 2008. Distractor explanations classified as Conceptual errors included a lack of understanding of mathematical terms, an inability to interpret or use symbols correctly, or a lack of relational understanding among the various forms of rational numbers such as fractions, decimals, and percents. The Problem Solving classification mostly resulted from a failure to effectively use selfmonitoring and control strategies, focus on the conditions of the problem, or failure to answer the question that was asked. Distractor explanations were classified as Procedural errors mostly if a student would select that distractor due to a failure to apply a series of steps correctly.
Table 3. Percentage of Incorrect Sixth Grade Student Response by Category 

Response Categorization 
2007 
2008 
Conceptual Errors 
43 
44 
Problem Solving Errors 
41 
42 
Procedural Errors 
14 
13 
Unknown Errors 
2 
1 
Total 
100 
100 
Example Analysis of Two Items from the 2008 SOL Assessment
The following examples were chosen from the sixthgrade SOL assessment to illustrate the impact of Conceptual and Problem Solving errors. Before reading further, solve the problems in Figures 1 and 2. Then brainstorm a list of anticipated misconceptions, procedural errors, or problem solving errors you think could cause a student to select each distractor. Then compare your thinking to the following analysis.
According to the classification at the end of the released test, the VDOE classified the question in Figure 1 as belonging to the Computation and Estimation reporting category on the 2008 released SOL test. Fifty percent of all students (25,311 of 50,556) answered this question correctly by selecting Choice J.
One way to solve this problem correctly is to first determine the whole number miles ran in one week by adding all the whole numbers, 0 + 2 + 3+ 2 + 3+2+10 = 22. Students could then combine fractional parts, ½ + ½ + ½ = 1 1/2. Next, combine the whole number and fractional parts, 22 + 1 ½, to find the number of miles ran in one week, 23 ½. Finally, double this mixed number to find the miles ran in two weeks, 23 ½ x 2 = 47 and then double again to find the number of miles run in 4 weeks, 47 x 2 = 94 miles.
Examination of the distractors reveals the other fifty percent of students who chose an incorrect answer most likely made a Problem Solving error by failing to answer the question that was asked. Notice this type of error seems to have affected students at all pass levels. This information is useful for teachers as it provides evidence that students at all pass levels would benefit from instruction focused on the problemsolving process. Students need to be sure to check their work at the end of this process to be sure the solution found answers the original question posed in the problem.
Table 4. Item 8 Percentage of Students Selecting Each Answer Choice by Pass Level 

Answer Choice 
Below Basic 
Basic 
Proficient 
Advanced 
Overall 
F 
42.1

42.1 
32.0 
17.7 
30.9 
G 
24.9 
15.7 
7.1 
1.7 
8.9 
H 
19.2 
16.6

10.0 
2.5 
10.0 
J (correct) 
13.4 
25.6

50.9 
78.0 
50.1 
T he analysis of this item provides evidence that students at all pass levels are likely selecting incorrect choices due to Problem Solving errors and not just due to conceptual or procedural misunderstandings. In the current reporting of the SOL assessment results, it is important to note the teacher/student are aware a question in the Computation and Estimation strand was missed. However, they are unaware the mostly likely reason for missing this item is due to Problem Solving and not a computation or estimation error. This is important to know as instruction needed to improve performance on this item should focus on the problemsolving process and not a particular Computation and Estimation topic.
Analyzing an Item from the Patterns, Functions, and Algebra Strand
Figure 2. Item 40 from the 2008 6^{th} grade Standards of Learning Released Test
“Conceptual understanding refers to an integrated and functional grasp of mathematical ideas” (Kilpatrick, Swafford & Findell, 2001, p. 118). As a constructivist, I believe that in order to learn new mathematical ideas students incorporate them into their existing ideas by building connections. “The greater the number of connections to a network of ideas, the better the understanding” (Van de Walle, Karp & BayWilliams, 2010, p. 24). For example, students who have a misconception that confuses squaring and doubling patterns could use color tiles to explore these patterns to make visual connections between these concepts. A discussion of these visual patterns and a reexamination of a question similar to the item in Figure 2 will help students assess their own answer choices and explicitly recognize they were making this error in confusing these patterns. In this way students will adapt their connected web of ideas to include a better understanding of differences in squaring and doubling patterns. The instructional methods chosen to improve conceptual understanding will be different as the concepts vary. Teachers need to be familiar with and anticipate the Conceptual errors that are mostly likely to occur for the various mathematical strands.
To reduce the impact of Problem Solving errors, teachers must explicitly focus on the problem solving process during mathematics instruction. Polya described problem solving as a four phase process: understand the problem, devise a plan, carry out the plan, and looking back (Polya, 1945). Effective problem solvers use this same general process regardless of the mathematical strand for a specific problem. In order for students to devise a proper plan, they need a tool bag filled with various strategies they can rely on when solving problems. To foster the development of these strategies, teachers need to provide students "frequent opportunities to explain their problemsolving strategies and solutions” (National Council of Teachers of Mathematics [NCTM], 2000, p. 258). Sharing strategies helps students build their strategic competence (Kilpatrick, Swafford & Findell, 2001) so they can more effectively analyze the conditions of other problems and select an appropriate strategy to find a solution.
Students can take turns sharing strategies for solving problems while teachers facilitate class discussion that illuminates similarities and differences in these strategies. This sharing process helps students make sense of how their own strategies connect to the strategies of their peers. Teachers can then ask students to solve a similar problem using another student’s strategy to encourage students to develop multiple ways to tackle the same problem. During these activities students may also discover they prefer the strategy of another student because it is more efficient or is more generally applicable for a wider variety of problems.
The results of this study showed students also need to be taught how to monitor their own problemsolving process. Teachers can use thinkaloud activities to foster students’ development of selfmonitoring by sharing their own thinking with students, or providing opportunities for students to share their thinking out loud. As part of the looking back phase of the problemsolving process, students “should be honestly convinced of the correctness of each step” (Polya, 1945, p. 13). Students also need to understand the importance of reflecting over their process when they think they have solved the problem. "Good problem solvers realize what they know and don't know" and "understand that the problemsolving process is not finished until they have looked back at their solution and reviewed their process" (NCTM, 2000, p. 261). Selfmonitoring also helps students realize when an approach is not working and change to a more appropriate strategy.
This study revealed Conceptual errors were likely responsible for approximately 40% of incorrect choices on the 2007 and 2008 on the Virginia SOL sixthgrade assessment. Conceptual errors stretched across all strands and pass levels; therefore, appropriate instruction needed to impact these errors will vary widely depending on the mathematical strand. It is important for teachers to understand which concepts are causing their students difficulty and also the impact poor conceptual understanding has on their students’ achievement. Once a teacher is aware of possible misconceptions he/she can design instruction to "intentionally diagnose and discriminate the known misconceptions" (Nesher, 1987, p. 36).
Throughout the school year students are given multiplechoice assessments to gauge their understanding of concepts. Teachers can work together as a team to brainstorm possible explanations for why their students would select particular incorrect choices. This analysis will help them identify possible student misconceptions and errors in order to target instruction to the specific needs of the students beyond what is possible if only the strand of the question is considered in the analysis. Teachers can then choose or create appropriate tasks and activities that focus on these misconceptions in order for students to "realize for themselves where they were wrong" (Nesher, 1987, p. 35).
The results of this study also revealed Problem Solving errors are likely having a major impact on students’ performance on the sixthgrade SOL assessments. Teachers may not be aware the extent to which Problem Solving errors are affecting their students’ assessment results. According to the Mathematics Standards of Learning for Virginia Public Schools, "The development of problemsolving skills should be a major goal of the mathematics program at every grade level" (VDOE, 2009, p.17). It is important to note similar Problem Solving errors cause students to incorrectly answer questions in multiple strands. This means a sustained focus on problem solving will positively impact performance in multiple strands on the SOL assessments for students at all pass levels.
It is my hope this article helps teachers realize the extent to which Problem Solving errors are impacting the performance of their students when answering questions in all of the strands. Focusing on problem solving all year long throughout all the strands is likely to positively impact student performance across all strands. This focus on process is generally applicable while instruction needed to address Conceptual errors requires very specific information in order to respond accordingly. In other words, if teachers want to improve student performance and do not know where to begin, start with problem solving.
I hope this discussion will also help teachers think about how they can make more effective use of the results of multiplechoice items they use with students throughout the year. These items are constantly providing information about student misconceptions/errors and can be very useful in designing future instruction that focuses on specific concepts or general aspects of the problemsolving process if we look deep enough.
Kilpatrick, J., Swafford, J., & Findell, B. (Eds.). (2001). Adding it up: Helping children learn mathematics. Washington, DC: National Academy Press.
Kloosterman, P., & Lester, F.K., Jr. (Eds.). (2007). Results and Interpretations of the 2003 Mathematics Assessment of the National Assessment of Educational Progress. Reston, VA: National Council of Teachers of Mathematics.
Miles, M.B., & Huberman, A.M. (1994). Qualitative Data Analysis (2^{nd} ed.). Thousand Oaks, CA: SAGE Publications.
National Center for Education Statistics. (2011, June 19). Mathematical Abilities. Retrieved from http://nces.ed.gov/nationsreportcard/mathematics/abilities.asp
National Council of Teachers of Mathematics. (2000). Principles and Standards for School Mathematics. Reston, VA: Author.
National Research Council. (1999). Testing, teaching, and learning. Washington, DC: Author.
Nesher, P. (1987). Towards an Instructional Theory: The Role of Student’s Misconceptions. For the Learning of Mathematics, 7(3), 3340.
Patton, M.Q. (1990). Qualitative evaluation and research methods (2^{nd} ed.). Newbury Park, CA: SAGE Publications.
Polya, G. (1945; 2^{nd} edition, 1985). How to Solve It. Princeton, NJ: Princeton University Press.
Virginia Department of Education. (2009). Mathematics Standards of Learning for Virginia Public Schools. Retrieved from http://www.doe.virginia.gov/testing/sol/standards_docs/mathematics/review.shtml
Dr. Virginia Lewis
Assistant Professor
Longwood University
lewisvv@longwood.edu