Jos Keuning

Various procedures for establishing performance standards have been proposed. Among the best-known examples are the Angoff procedure, the Bookmark procedure and the Direct Consensus procedure. The procedures have their strengths and weaknesses. Some procedures make it possible to establish performance standards relatively efficiently and quickly, but lack empirical rigor. Other procedures do include empirical data, but are time consuming and not very intuitive. In the present study, it was attempted to bring together the strengths of existing standard setting procedures. The newly developed procedure can best be considered an extension of Direct Consensus. It adds the use of empirical data to this procedure. The Data-Driven Direct Consensus (3DC) procedure assumes a test to consist of multiple items that can be divided into a number of clusters. Panelists are asked to indicate the score that students would be expected to achieve in each cluster if they were exactly on the borderline of the selected mastery level. The cut-off scores are marked on a specially designed assessment form which relates the clusters to the full test by means of an empirical prediction model. In this way, the panelists’ assessment is easily allowed to be based on both content information and student data.

Published Feb. 25, 2015 2:05 PM