Item difficulty analysis of a high-stakes mathematics examination using Rasch analysis

Abstract

The National Senior Certificate examination is the most important school examination in South Africa. Analysis of learners’ performance in Mathematics in this examination is normally carried out and presented in terms of the percentage of learners who succeeded in the different bands of achievement. In some cases item difficulties are presented – item refers to the subsection of each examination question. Very little attention is paid to other diagnostic statistics, such as the discrimination indices and item difficulties taking into consideration partial scores examinees achieve on items. In this article we report on a study that, in addition to the usual item difficulties, includes a discrimination index of item difficulties taking into account partial scores examinees achieved. The items, considered individually, are analysed in relation to the other items on the test. The focus is on the topic sequences and series and the data were obtained from a stratified sample of the marked scripts of the candidates who wrote the National Senior Certificate examination in Mathematics in November 2010. Rasch procedures were used for the analysis. The findings indicate that learners perform differently on subsections of topics, herein referred to as items, and that focusing on scores for full topics potentially mask these differences. Mathematical explanations are attempted to account for difficulties learners exhibit in these subsections, using a hierarchy of scale. The findings and our analysis indicate that a form of measurement-driven testing could have beneficial results for teaching. Also, for some items the difficulty obtained from the work of examinees runs counter to the commonly perceived wisdom that an examination ought to be structured in such a way that the less difficult items are at the start of a topic. An explanatory device anchored around the construct of ‘familiarity with problem types through repeated productive practice’ is used to account for the manifested hierarchy of difficulty of the items.