Abstract

This study aims to identify how feasible it is to use comparative judgement (CJ) and student judges to assess the writing performance of English language learners. For this purpose, 35 paragraphs written by the students who were enrolled in a freshman Academic Writing course at a semi-private university located in the Turkish Republic of Northern Cyprus were selected and uploaded to http://www.nomoremarking.com website. Ten instructors of the Academic Writing course and 112 students taking the course volunteered to participate in the study. Then, the students were taken into 5 groups according to their writing performance level. In total, around 350 comparisons were done by each group. The results suggested that it could be feasible to use CJ to assess short writing performance like paragraphs if the instructors were experienced and trained. Moreover, instructors liked CJ and described it as a more practical, easier, fairer, faster, more enjoyable way of marking student papers. The students also liked CJ and it was also found that students who were high achievers in paragraph writing might be used to mark student papers through comparative judgement as long as they were trained.

Keywords

References

• Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43(4), 561-573. https://doi.org/10.1007/BF02293814
• Barmby, P. (2019). The three Rs: Reliability, writing and rich tea biscuits. No More Marking Blog. https://blog.nomoremarking.com/the-three-rs-reliability-writing-and-rich-tea-biscuits-2de57f33e754
• Bloxham, S. (2009). Marking and moderation in the UK: False assumption and wasted resources. Assessment & Evaluation in Higher Education, 34(2), 209-220. https://doi.org/10.1080/02602930801955978
• Bloxham, S., Boyd, P., & Orr, S. (2011). Mark my words: The role of assessment criteria in UK higher education grading practices. Studies in Higher Education, 36(6), 655–670. https://doi.org/10.1080/03075071003777716
• Bramley, T., Bell, J. F., & Pollitt, A. (1998). Assessing changes in standards over time using Thurstone Paired Comparisons. Education Research and Perspectives, 25(2), 1-24.
• Bramley, T., & Vitelio, S. (2018). The effect of adaptivity on the reliability coefficient in adaptive comparative judgement. Assessment in Education: Principles, policy & Practice, 26(1), 43-58. https://doi.org/10.1080/0969594X.2017.1418734
• Brogden, H.E. (1977). The Rasch model, the law of comparative judgment and additive conjoint measurement. Psychometrika 42, 631–634. https://doi.org/10.1007/BF02295985
• Charney, D. (1984). The validity of using holistic scoring to evaluate writing: A critical overview. Research in the Teaching of English, 18(1), 65-81.
• Crisp, V. (2013). Criteria, comparison, and past experiences: How do teachers make judgements when marking coursework? Assessment in Education: Principles, Policy & Practice, 20(1), 127-144. https://doi.org/10.1080/0969594X.2012.741059
• He, Q., Anywyll, S., Glanville, M., & Deavall, A. (2013). An investigation of the reliability of marking of the key stage 2 national curriculum English writing tests in England. Educational Research, 55(4), 393-410. https://doi.org/10.1080/00131881.2013.844942
• Humphry, S. M., & Heldsinger, S. A. (2014). Common structural design features of rubrics may represent a threat to validity. Educational Researcher, 43(5), 253-263. https://doi.org/10.3102/0013189X14542154
• Humphry, S. M., & McGrane, J. A. (2015). Equating a large-scale writing assessment using pairwise comparisons of performances. The Australian Educational Researcher, 42(4), 443-460. https://doi.org/10.1007/s13384-014-0168-6
• IBM Corp. (2013). IBM SPSS Statistics for Windows, Version 22.0. Author.
• Jones, I., & Alcock, L. (2014). Peer assessment without assessment criteria. Studies in Higher Education, 39(10), 1774-1787. https://doi.org/10.1080/03075079.2013.821974
• Jones, I., & Wheadon, C. (2015). Peer assessment using comparative and absolute judgement. Studies in Educational Evaluation, 47, 93-101. https://doi.org/10.1016/j.stueduc.2015.09.004
• Kimbell, R., Wheeler, T., Stables, K., Sheppard, T., Martin, F., Davies, D., Pollitt, A., & Whitehouse, G. (2009). E-scape portfolio assessment: phase 3 report. Technology Education Research Unit. https://www.teachertoolkit.co.uk/wp-content/uploads/2014/08/e-scape_phase3_report.pdf
• Liu, P., & Li, Z. (2012). Task complexity: A review and conceptualization framework. International Journal of Industrial Ergonomics, 42(6), 553-568. https://doi.org/10.1016/j.ergon.2012.09.001
• Madsen, H. S. (1987). Techniques in Testing. Oxford University Press
• McGrane, J. A., Humphry, S. M., & Heldsinger, S. (2018). Applying a Thurstonian, two-stage method in the standardized assessment of writing. Applied Measurement in Education, 31(4), 297-311. https://doi.org/10.1080/08957347.2018.1495216
• Moss, P. A. (1994). Validity in high stakes writing assessment: Problems and possibilities. Assessing Writing, 1(1), 109-128. https://doi.org/10.1016/1075-2935(94)90007-8
• Oya, T., Manalo, E., & Greenwood, J. (2004). The influence of personality and anxiety on the oral performance of Japanese speakers of English. Applied Cognitive Psychology, 18(7), 841–855. https://doi.org/10.1002/(ISSN)1099-0720
• Pollitt, A. (2004). Let’s stop marking exams [Paper presentation]. IAEA Conference, Philadelphia, PA. https://www.cambridgeassessment.org.uk/Images/109719-let-s-stop-marking-exams.pdf
• Pollitt, A. (2012). Comparative judgment for assessment. International Journal of Technology and Design Education, 22(2), 157–170. http://dx.doi.org/10.1007/s10798-011-9189-x
• Popham, W. J. (2005). The instructional consequences of criterion-referenced clarity. Educational Measurement: Issues and Practice, 13(4), 15-18. https://doi.org/10.1111/j.1745-3992.1994.tb00565.x
• Sadler, D. R. (2009). Transforming holistic assessment and grading into a vehicle for complex learning. In G. Joughin (Ed.), Assessment, learning and judgement in higher education (pp.1-19). Springer. https://doi.org/10.1007/978-1-4020-8905-3_4
• Steckler, A., McLeroy, K. R., Goodman, R. M., Bird, S. T., & McCormick, L. (1992). Toward integrating qualitative and quantitative methods: An introduction. Health Education Quarterly, 19(1), 1–8. https://doi.org/10.1177/109019819201900101
• Thurstone, L., L. (1927). A law of comparative judgement. Psychological Review, 34, 273-286.
• Topping, K. (2010). Methodological quandaries in studying process and outcomes in peer assessment. Learning and Instruction, 20(4), 339-343. https://doi.org/10.1016/j.learninstruc.2009.08.003
• van Daal, T., Lesterhuis, M., Coertjens, L., Donche, V., & De Maeyer, S. (2019). Validity of comparative judgement to assess academic writing: examining implications of its holistic character and building on a shared consensus. Assessment in Education: Principles, Policy & Practice, 26(1), 59-74. https://doi.org/10.1080/0969594X.2016.1253542
• Verhavert, S., Bouwer, R., Donche, V., & De Maeyer, S. (2019). A meta-analysis on the reliability of comparative judgement. Assessment in Education: Principles, policy & practice, 26(5), 541-562. https://doi.org/10.1080/0969594X.2019.1602027
• Wheadon, C., Barmby, P., Christodoulou, D., & Henderson, B. (2020). A comparative judgement approach to the large-scale assessment of primary writing in England. Assessment in Education: Principles, Policy & Practice, 27(1), 46-64, https://doi.org/10.1080/0969594X.2019.1700212

License

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.