Validating Reading Comprehension Assessment Under the GDINA Model
Abstract
Cognitive diagnosis models (CDMs) are latent variable models mainly developed to assess students’ specific strengths and weaknesses in a set of skills or attributes within a particular domain. In this study, the reading comprehension assessment is diagnostically designed, constructed, and developed from the very first step. The predetermined attributes or sub-skills are explicitly defined in the construction phase as they should align with the instructional goals. Using R package CDM, the Generalized-DINA model (GDINA) was applied to the reading comprehension assessment. A total of 900 Year 4 primary students from the Eastern District of Pulau Pinang national and vernacular schools sat for this assessment. Through the cognitive analysis, the study is expected to provide detailed diagnostic feedback on students’ strengths and weaknesses in the underlying skills identified in the reading comprehension assessment. Such detailed information can help teachers in classroom teaching, designing remedial courses, and developing material according to the student's needs.
Downloads
References
Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716-723. doi:10.1109/TAC.1974.1100705
Alderson, J. C. (2005). Diagnosing foreign language proficiency: The interface between learning and assessment. London: Continuum.
Aryadoust, V. (2011). Application of the fusion model to while-listening performance tests. Shiken: JALT Testing & Evaluation SIG Newsletter, 15(2), 2-9.
de la Torre, J. (2009). DINA model and parameter estimation : A didactic. Journal of Educational and Behavioral Statistics, 34, 115-130.
de la Torre, J. (2011). The generalized DINA framework. Psychometrika, 76, 179-199. doi:10.1007/s11336-011-9207-7
DiBello, L. V., Roussos, L. A., & Stout, W. F. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In C. R. (Eds.), Handbook of statistics (Vol. 26, pp. 979-1030). Amsterdam, Netherlands: Elsevier.
George, A. C., Robitzsch, A., Kiefer, T., & GroB, J., & Unlu, A. (2016). The R package CDM for cognitive diagnosis models. Journal of Statistical Software, 74(2), 1-24.
Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities : Blending theory with practicality. Doctoral dissertation: University of Illinois at Urbana-Champaign.
Henson, R. A., Templin, J. L., & Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191-210.
Huebner, A. (2010). An overview of recent developments in cognitive diagnostic computer adaptive assessments. Practical Assessment, Research, and Evaluation, 15(1), 3. doi:10.7275/7fdd-6897
Jang, E. E. (2008). A Review of cognitive diagnostic assessment for education: Theory and application. International Journal of Testing, 8(3), 290-295.
Javidanmehr, Z., & Anani Sarab, M. R. (2017). Diagnostic Assessment: Issues and Considerations. International Journal of Language Testing, 7(2), 73-98.
Lee, Y. S., de la Torre, J., & Park, Y. S. (2012). Relationships between cognitive diagnosis, CTT, and IRT induces : An empirical investigation. Asia Pacific Education Review, 13, 333-345.
Lee, Y. W., & Sawaki, Y. (2009a). Application of three cognitive diagnosis models to ESL reading and listening assessments. Language Assessment Quarterly, 6(3), 239- 263. doi:10.1080/15434300903079562
Lee, Y. W., & Sawaki, Y. (2009b). Cognitive diagnosis approaches to language assessment :An overview. Language Assessment Quaterly, 6(3), 172-189. doi:10.1080/15434300902985108
Leighton, J. P., Gierl, M. J., & Hunka, S. M. (2004). The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka's Rule-Space Approach. Journal of Educational Measurement, 41, 205-237.
Li, H., & Hunter, C. V. (2016). The selection of cognitive diagnostic models for a reading comprehension test. Language Testing, 33(3), 391-409.
Li, H., Hunter, C. V., & Lei,P. W. (2015). The selection of cognitive diagnostic models for a reading comprehension test. Language Testing, 33, 391–409.
Ma, W., Minchen, N., & de la Torre, J. (2020). Choosing between CDM and Unidimensional IRT: The Proportion Reasoning Test Case. Measurement: Interdisciplinary Research and Perspectives, 80(2), 87-96. doi:10.1109/15366367.2019.1697122
Ravand, H. & Robitzsch, A. (2015). Cognitive diagnostic modeling using R. Practical Assessment, Research & Evaluation, 20, 1-12.
Ravand, H. (2016). Application of a cognitive diagnostic model to a high stakes reading comprehension test. Journal of Psychoeducational Assessment, 34, 782-799. doi:10.1177/0734282915623053
Ravand, H., Barati, H., & Widhiarso, W. (2013). Exploring diagnostic capacity of a high stakes reading comprehension test: A pedagogical demonstration. International Journal of Language Testing, 3(1), 11-37.
Rupp, A. A., Templin, J., & Henson, R. A. (2010). Diagnostic assessment : Theory, methods, and applications. New York, NY: Guilford.
Rupp. A. A., & Templin, J. L. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement: Interdisciplinary Research and Perspectives, 6(4), 219-262.
Schwarz. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461-464. doi:10.1214/aos/1176344136
Tatsuoka, K. (1983). Rule Space; An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354.
Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorder using cognitive diagnosis models. Psychological Methods, 11(3), 287-305.
von Davier, M. (2005). A general diagnostic model applied to language testing data (RR-05-16). Princeton, NJ: Educational Testing Service.
Wang, C., Shu, Z., & Xu, G. (2015). Assessing item-level fit for the DINA model. Applied Psychological measurement, 39(7), 525-538. doi:10.1177/0146621615583050
