Large-scale alternate assessments based on fine-grained learning maps: Opportunities and challenges
Session 5A, 13:00 - 14:30, HAGEN 2
In the United States of America, students with the most significant cognitive disabilities participate in statewide academic assessment systems through alternate assessments based on alternate achievement standards (AA-AAS). The population of students who take AA-AAS is very small (approximately 1% of students) and extremely heterogeneous (Kearns et al., 2011). AA-AAS were first conceived nearly 20 years ago. Since then the educational assessment field has dealt with tensions between the standardization typical of large-scale assessment and the flexibility needed to ensure accessibility for the population. Due to design constraints and the population, AA-AAS also have unique challenges with regard to evidence of validity and technical quality.
In 2010 a consortium of states began developing a next generation AA-AAS. First used operationally in 2015, the Dynamic Learning Maps (DLM) alternate assessment system now serves 90,000 students across 18 states. DLM assessments are based on large, fine-grained learning maps with thousands of nodes (skills) and multiple pathways by which students develop understanding of academic domains (Kingston, Karvonen, Bechard, & Erickson, 2016). Assessments are designed using a combination of evidence-centered design and universal design principles. Assessments are delivered in short testlets with varying degrees of complexity relative to the content standard. Unlike most large-scale academic assessments, the DLM system goes beyond summative uses. Testlets are designed to be instructionally relevant. Teachers select and use instructionally embedded assessments throughout the year so results guide instruction. Consistent with the highly multidimensional nature of the learning maps, DLM assessments are scored using Cognitive Diagnostic Modeling. Summative results are based on aggregated mastery of discrete skills. Score reports feature fine-grained, diagnostic information to guide instruction as well as summative results used for program evaluation and accountability.
The proposed session for the innovative assessment strand will begin with a brief description of the philosophical underpinnings and design of the DLM alternate assessment system. Several opportunities and challenges will be described in more depth, using evidence from early development and four years of operational test administration. Depending on the length of the session, topics would likely include: (1) evaluation of the test development approach that integrates evidence-centered design and universal design for learning; (2) implementation evidence interpreted in light of the program’s theory of action; (3) an overview of the modeling research needed to support the use of CDM for scoring; and (4) the standard setting approach designed for use with CDM-based results. The session concludes with a summary of future directions for the DLM system and potential implications for other assessment systems.