Fredrik Helland-Riise

• Helland-Riise, Fredrik & Martinussen, Monica (2017). Psychometric properties of the Norwegian version of Ravens matriser [Standard Progressive Matrices (SPM)/Coloured Progressive Matrices (CPM)]. PsykTestBARN. ISSN 1893-9910. 7(2), p. 1–21. Full text in Research Archive Show summary

  Helland-Riise, F. & Martinussen, M. (2017). Psychometric properties of the Norwegian version of Ravens matriser [Standard Progressive Matrices (SPM)/Coloured Progressive Matrices (CPM)]. PsykTestBarn, 2:2. There are different versions of Ravens Matrices including Coloured Progressive Matrices (CPM) (5–11 years) and Standard Progressive Matrices (SPM) (8–65 years) which may be used for children and adolescents. The test was first developed by John C. Raven, and the SPM was published in 1938, and the CPM in 1998. The test is assumed to measure non-verbal intelligence or abstract reasoning ability with items consisting of geometrical figures where the task is to figure out the system underlying the figures. Both versions of Raven have items with increasing difficulty organized in different sets. The test is scored in terms of the number of correct answers, which is summed to a total score. The total score can be converted to percentiles based on norm tables included in the manuals. The test is usually administered without time constraints for children and adolescents, and the test user is required to be a psychologist or a certified user of ability tests. Pearson Assessment holds the copyright to Ravens Matrices, and is responsible for sale and distribution of the test in Scandinavia and internationally (pearsonassessment.no). A total of 15 Norwegian and 24 Swedish/Danish publications were included in the review. None of these were psychometric studies, but mainly studies where Raven had been used to measure intelligence either as an outcome variable, control variable or to describe the group. Approximately half of the included studies were based on clinical groups (e.g., autism, epilepsy or deaf), while the remaining studies included school children of 5–16 (SPM) and 5–7 (CPM) years. None of the included studies had performed adequate studies of test reliability and there were no norm studies based on Norwegian or Swedish/Danish samples. The findings support the construct validity of the test as a good measure of abstract reasoning ability by in general high correlations with other cognitive tests. There is solid evidence supporting the construct validity of the test, but studies examining test reliability using Scandinavian samples are lacking. There are no norm studies from Scandinavia, which is problematic for clinical and applied test use.

View all works in Cristin

• Helland-Riise, Fredrik & Braeken, Johan (2017). The Radicals of Abstract Reasoning Assessments: An Explanatory Item Response Modelling Approach. Show summary

  Abstract reasoning assessments are widely considered the gold standard of intelligence testing. Although Raven Progressive Matrices (RPM) are probably the most prominent example, there are many more less studied and less developed abstract reasoning assessments used in practice. To improve these assessments and generalize RPM study results, we need a better understanding of what factors of the item design drive item response behavior. In modern test design such factors are called radicals, item properties that when modified lead to for instance a change in item difficulty; this in contrast to incidentals, item properties that are mere cosmetic hanges not affecting psychometric item characteristics. Here, we se an explanatory item response modelling approach to examine he effects of potential radical item properties derived from a cognitive lab and from an artificial intelligent item solver to provide validity evidence for an non-Raven-like abstract reasoning assessment. We discuss how this validity evidence can be a first step in a more systematic redesign of the assessment opening up opportunities for automatic item generation and computerized adaptive testing.

• Helland-Riise, Fredrik & Braeken, Johan (2016). Explanatory Item Response Modelling Of An Abstract Reasoning Assessment: A Case For Modern Test Design. Show summary

  General description Reasoning tests are popular components of the assessment toolbox for selection and admission into higher education and job employment (Leighton, 2004; Stanovich, Sá, & West, 2004). Abstract reasoning tests tap into a core reasoning ability (Carpenter, Just, & Shell, 1990) with tasks that require the examinee to generate and apply rules (Wüstenberg, Greiff, & Funke, 2012), but require neither explicit prior contents-specific knowledge of the examinee nor specific language skills (Raven, 2000). Traditionally, test construction and assembly have been the product of creative item writing processes and post-hoc psychometric evaluations, without explicit consideration of cognitive theory (Hunt, Frost, & Lunneborg, 1973). Yet, abstract reasoning provides a case that in principle is ideally suitable for modern test design (e.g. Embretson, 1998; Mislevy, Almond, & Lukas, 2003), combining cognitive theory with a more systematic approach to construction and assembly of test items. Objective and Research Questions. This study is part of a larger project aimed at reverse engineering an existing abstract reasoning test from a modern test design perspective to setup a virtual item bank that does not store individual items, but instead uses automatic item generation rules based on cognitive complexity (see e.g., Gierl & Haladyna, 2013). The objective of the current study represents one step towards such a virtual item bank with research questions focusing on (i) identifying the cognitive relevant item features (i.e. “radicals”) that impact the behaviour of the test and of the participants and (ii) identifying the merely “cosmetic” irrelevant item features (i.e., incidentals). The test. The abstract reasoning test is composed of testlets consisting of items related to the same problem situation from which a set of rules need to be derived that are necessary to solve the individual items. Each testlet is structured around a problem set consisting of a varying number of rows each consisting of a specified input stimulus configuration, an activated set of action buttons and a resulting output stimulus configuration. This problem set allows the examinee to derive the transformations that will happen to the input when a specific action button is activated. This rule knowledge is necessary to solve the connected items. An item consists of a single row with a specified input stimulus configuration, the activated set of action buttons for that item, and four alternative output stimulus configuration possibilities of which the examinee has to decide on the correct one. Theoretical framework. A rational task analysis of the abstract reasoning test proposes an artificial intelligent algorithm (see Newell & Simon, 1972) that consists of 4 core steps. (1) Inventorisation: all the characteristics of input stimulus configurations and output stimulus configurations of the problem set are registered; (2) Matching: an input/output dissimilarity matrix is computed; (3) Rule finding: computationally this would be similar to solving a system of equations or a more greedy version using elimination; (4) Rule application. The test has some characteristics built in by design that can be directly connected to the artificial intelligent algorithm and the related (i) cognitive load of the stimulus material and (ii) cognitive complexity of the rules that need to be derived. Examples of the former characteristics can be as simple as the number of symbols in the input stimulus configuration, examples of the latter characteristics can be whether or not the transformation caused by a specific action button can be derived on its own (i.e., independent of the other action buttons in the problem set). Some theoretically irrelevant item features can also be defined such as the type of symbols used in a stimulus configuration (e.g., triangle or circle).

View all works in Cristin