The Radicals of Abstract Reasoning Assessments: An Explanatory Item Response Modelling Approach

Abstract reasoning assessments are widely considered the gold standard of intelligence testing. Although Raven Progressive Matrices (RPM) are probably the most prominent example, there are many more less studied and less developed abstract reasoning assessments used in practice. To improve these assessments and generalize RPM study results, we need a better understanding of what factors of the item design drive item response behavior. In modern test design such factors are called radicals, item properties that when modified lead to for instance a change in item difficulty; this in contrast to incidentals, item properties that are mere cosmetic hanges not affecting psychometric item characteristics. Here, we se an explanatory item response modelling approach to examine he effects of potential radical item properties derived from a cognitive lab and from an artificial intelligent item solver to provide validity evidence for an non-Raven-like abstract reasoning assessment. We discuss how this validity evidence can be a first step in a more systematic redesign of the assessment opening up opportunities for automatic item generation and computerized adaptive testing.