Fredrik Helland-Riise

• Helland-Riise, Fredrik; Norrøne, Tore Nøttestad & Andersson, Björn (2024). Large-Scale Item-Level Analysis of the Figural Matrices Test in the Norwegian Armed Forces: Examining Measurement Precision and Sex Bias. Journal of Intelligence. 12(9). doi: 10.3390/jintelligence12090082. Full text in Research Archive Show summary

  Figural matrices tests are common in intelligence research and have been used to draw conclusions regarding secular changes in intelligence. However, their measurement properties have seldom been evaluated with large samples that include both sexes. Using data from the Norwegian Armed Forces, we study the measurement properties of a test used for selection in military recruitment. Item-level data were available from 113,671 Norwegian adolescents (32% female) tested between the years 2011 and 2017. Utilizing item response theory (IRT), we characterize the measurement properties of the test in terms of difficulty, discrimination, precision, and measurement invariance between males and females. We estimate sex differences in the mean and variance of the latent variable and evaluate the impact of violations to measurement invariance on the estimated distribution parameters. The results show that unidimensional IRT models fit well in all groups and years. There is little difference in precision and test difficulty between males and females, with precision that is generally poor on the upper part of the scale. In the sample, male latent proficiency is estimated to be slightly higher on average, with higher variance. Adjusting for measurement invariance generally reduces the sex differences but does not eliminate them. We conclude that previous studies using the Norwegian GMA data must be interpreted with more caution but that the test should measure males and females equally fairly.

• Helland-Riise, Fredrik & Martinussen, Monica (2017). Måleegenskaper ved de norske versjonene av Ravens matriser [Standard Progressive Matrices (SPM)/Coloured Progressive Matrices (CPM)]. PsykTestBarn. 7(2), p. 1–21. Full text in Research Archive Show summary

  Ravens matriser består av flere varianter der Coloured Progressive Matrices (CPM) (5–11 år) og Standard Progressive Matrices (SPM) (8–65 år) kan anvendes for barn og unge. Testen ble utviklet av John C. Raven, og SPM ble først publisert i 1938, mens CPM-versjonen ble publisert i 1998. Testen er ment å måle non-verbal intelligens eller evne til abstrakt resonnering ved at barnet skal finne systemet bak hvordan ulike geometriske figurer er satt opp. Begge versjonene av Raven har oppgaver med økende vanskelighetsgrad organisert i ulike sett. Testen skåres ved at antallet riktige svar summeres opp til en totalskår. Dette kan konverteres til prosentiler basert på normtabellene i manualene. Testen administreres vanligvis uten tidsbegrensning for barn og unge, og det stilles krav om at de som skal anvende testen er sertifisert for å anvende evnetester eller er psykologer. Pearson Assessment har rettighetene til salg og distribusjon av testen i Skandinavia og internasjonalt (pearsonassessment.no). Til sammen ble 15 norske og 24 svenske/danske artikler inkludert i oppsummeringen. Ingen av disse var rene psykometriske studier, men stort sett studier der Raven ble brukt for å måle intelligens enten som utkommevariabel, kontrollvariabel eller for å beskrive gruppen. Litt under halvparten av studiene var basert på kliniske grupper (for eksempel autisme, epilepsi eller døve), men de øvrige studiene var basert på skolebarn fra 5–16 (SPM) og 5–7 (CPM) år. Ingen av studiene hadde gjennomført adekvate studier av testens reliabilitet og det er heller ikke gjennomført normstudier basert på norske eller svensk/danske utvalg. Resultatene fra studiene støtter begrepsvaliditeten til testen som et godt mål på abstrakt resonneringsevne ved generelt høye korrelasjoner med andre kognitive tester. Konklusjon. Det er god dokumentasjon på testens begrepsvaliditet, men det mangler studier av testens reliabilitet basert på norske eller svenske/danske utvalg. Det er heller ikke normstudier fra Skandinavia, noe som er problematisk ved klinisk bruk av testen.

View all works in NVA

• Helland-Riise, Fredrik & Braeken, Johan (2017). The Radicals of Abstract Reasoning Assessments: An Explanatory Item Response Modelling Approach. Full text in Research Archive Show summary

  Abstract reasoning assessments are widely considered the gold standard of intelligence testing. Although Raven Progressive Matrices (RPM) are probably the most prominent example, there are many more less studied and less developed abstract reasoning assessments used in practice. To improve these assessments and generalize RPM study results, we need a better understanding of what factors of the item design drive item response behavior. In modern test design such factors are called radicals, item properties that when modified lead to for instance a change in item difficulty; this in contrast to incidentals, item properties that are mere cosmetic hanges not affecting psychometric item characteristics. Here, we se an explanatory item response modelling approach to examine he effects of potential radical item properties derived from a cognitive lab and from an artificial intelligent item solver to provide validity evidence for an non-Raven-like abstract reasoning assessment. We discuss how this validity evidence can be a first step in a more systematic redesign of the assessment opening up opportunities for automatic item generation and computerized adaptive testing.

• Helland-Riise, Fredrik & Braeken, Johan (2016). Explanatory Item Response Modelling Of An Abstract Reasoning Assessment: A Case For Modern Test Design. Full text in Research Archive Show summary

  General description Reasoning tests are popular components of the assessment toolbox for selection and admission into higher education and job employment (Leighton, 2004; Stanovich, Sá, & West, 2004). Abstract reasoning tests tap into a core reasoning ability (Carpenter, Just, & Shell, 1990) with tasks that require the examinee to generate and apply rules (Wüstenberg, Greiff, & Funke, 2012), but require neither explicit prior contents-specific knowledge of the examinee nor specific language skills (Raven, 2000). Traditionally, test construction and assembly have been the product of creative item writing processes and post-hoc psychometric evaluations, without explicit consideration of cognitive theory (Hunt, Frost, & Lunneborg, 1973). Yet, abstract reasoning provides a case that in principle is ideally suitable for modern test design (e.g. Embretson, 1998; Mislevy, Almond, & Lukas, 2003), combining cognitive theory with a more systematic approach to construction and assembly of test items. Objective and Research Questions. This study is part of a larger project aimed at reverse engineering an existing abstract reasoning test from a modern test design perspective to setup a virtual item bank that does not store individual items, but instead uses automatic item generation rules based on cognitive complexity (see e.g., Gierl & Haladyna, 2013). The objective of the current study represents one step towards such a virtual item bank with research questions focusing on (i) identifying the cognitive relevant item features (i.e. “radicals”) that impact the behaviour of the test and of the participants and (ii) identifying the merely “cosmetic” irrelevant item features (i.e., incidentals). The test. The abstract reasoning test is composed of testlets consisting of items related to the same problem situation from which a set of rules need to be derived that are necessary to solve the individual items. Each testlet is structured around a problem set consisting of a varying number of rows each consisting of a specified input stimulus configuration, an activated set of action buttons and a resulting output stimulus configuration. This problem set allows the examinee to derive the transformations that will happen to the input when a specific action button is activated. This rule knowledge is necessary to solve the connected items. An item consists of a single row with a specified input stimulus configuration, the activated set of action buttons for that item, and four alternative output stimulus configuration possibilities of which the examinee has to decide on the correct one. Theoretical framework. A rational task analysis of the abstract reasoning test proposes an artificial intelligent algorithm (see Newell & Simon, 1972) that consists of 4 core steps. (1) Inventorisation: all the characteristics of input stimulus configurations and output stimulus configurations of the problem set are registered; (2) Matching: an input/output dissimilarity matrix is computed; (3) Rule finding: computationally this would be similar to solving a system of equations or a more greedy version using elimination; (4) Rule application. The test has some characteristics built in by design that can be directly connected to the artificial intelligent algorithm and the related (i) cognitive load of the stimulus material and (ii) cognitive complexity of the rules that need to be derived. Examples of the former characteristics can be as simple as the number of symbols in the input stimulus configuration, examples of the latter characteristics can be whether or not the transformation caused by a specific action button can be derived on its own (i.e., independent of the other action buttons in the problem set). Some theoretically irrelevant item features can also be defined such as the type of symbols used in a stimulus configuration (e.g., triangle or circle).

View all works in NVA