Early Career Researcher Award Winner: On the diagnostic power of the items in a pool
Miguel A. Sorrel – Autonomous University of Madrid, Spain
Abstract: In recent years, computerized adaptive testing for diagnostic purposes (CD-CAT) has gained relevance. Under this framework, the item response function is operationalized with a cognitive diagnosis model (CDM). Due to the very narrow definition of attributes in education (e.g., addition, subtraction, simplification), items usually have a complex structure. Despite this, previous research has shown that the available item selection rules may show a preference for administering simpler items. The question arises as to whether simple items are indeed the best way to obtain diagnostic information. As a possible influencing factor in this situation, the item pool calibration protocols and model selection indices available are explored in a Monte Carlo simulation study where several item selection rules are compared in terms of accuracy and item pool usage. The item pool structure (% of items with simple and complex structure), calibration sample size, adaptive test length, starting rule, number and distribution of attributes, and the model estimation and uncertainty consideration methods are manipulated as factors. By relating the calibration sample size to the complexity of the item pool and the calibration protocols available, this research results in a fundamental practical guide on how to approach diagnostic evaluation in low sample size contexts in the most optimal way possible.