5.1.2 The Effectiveness of the Q-matrix ............................... 70
CHAPTER FIVE DATA ANALYSIS AND DISCUSSION
5.1 The Effectiveness of the CDA in Diagnosing Students’ Writing Ability
To explore whether the CDA is effective in diagnosing high school students’ English writing ability, the MFRM (Many-facet Rasch model) analysis was conducted to validate the effectiveness of the CD checklist, and the parameters of the Q-matrix were then estimated to examine the discriminative capacity of the checklist.
5.1.1 The Validity of the Cognitive Diagnostic Checklist
The MFRM analysis was performed in MINIFAC. MINIFAC is a reduced version of Facets with free access. It has complete FACETS functionality but is limited to 2,000 responses. As there were 21 descriptors, only 46 out of the 127 essays were selected by equally spaced sampling (21 descriptors*46 essays*2 scores from two raters=1932).
With the help of MINIFAC, a vertical ruler of all facets was established, including students, raters, and descriptors (Figure 5-1). The first column shows the logit values ranging from -1 to 3. Students' ability is displayed in the second column, with more capable students plotted higher. The third column shows the rater severity, and raters who are more severe are plotted higher. The rater severity ranges between -1 and 1 logits, indicating that the difference in rater severity was small. In the last column, 21 descriptors are spread across two logits, with more difficult descriptors plotted higher than easier ones (He & Zhang, 2008).
CHAPTER SIX CONCLUSION
6.1 Major Findings of the Study
The central principle of diagnostic assessment is that it assesses fine-grained knowledge processes and structures in a test domain, thus providing detailed information about students’ understanding of the test materials (Kim, 2011). To investigate the strengths and weaknesses of high school students’ argumentative writing ability, this study constructed a cognitive diagnostic checklist with reference to Kim’s EDD checklist and the CSE. Based on MFRM analysis and CDM analysis, the major findings are summarized as follows, with Finding 1 corresponding to Research Question 1, Findings 2 and 3 to Question 2, and Finding 4 to Question 3.
Finding 1: Cognitive diagnostic assessment is a reliable, discriminant, and accurate way to diagnose high school students’ writing ability.
Firstly, the MFRM analysis provided a rich set of statistical results, ranging from separation ratio and reliability to fit statistics. At the student level, the separation (2.03) and the Infit and Outfit MnSq values (0.68-1.41) indicated that the writing test could provide a reasonable and differentiated measure of students’ ability. At the rater level, although the rater severity differed significantly, the Infit and Outfit MnSq values (1.00-1.02) indicated that raters were generally consistent in rating these essays with the CD checklist. The bias analysis also revealed no significant bias in raters-students and rater-descriptor interaction.
reference(omitted)
