AsiaTEFL Logo
AsiaTEFL MENU
Indexed in SCOPUS
About AsiaTEFL
Notice
2023 Conference
Conference
Webinars
AsiaTEFL Book Series
AsiaTEFL Links
Gallery
The Journal of Asia TEFL
Information of the Journal
Submission Guidelines
Ethical Guidelines
Manuscript Submission
Editorial Board
Past Issues
Journal Order
Membership
Contact Us
My Page
Past Issues
Go List

Volume 14 Number 4, Winter 2017, Pages 587-836   


 http://dx.doi.org/10.18823/asiatefl.2017.14.4.7.687 PDF Download
   

Investigating the Effect of Training on Raters' Bias toward Test Takers in Oral Proficiency Assessment: A FACETS Analysis

    Houman Bijani & Mona Khabiri


Typically, variability among raters in scoring and their bias is mediated through rater training. However, questions still remain about whether training can affect raters' severity or leniency. Furthermore, few studies have looked at the differences between trained and untrained raters in oral assessment. Oral test scores of 200 test takers rated by 20 raters and were analyzed before and after a training program using the multifaceted Rasch measurement (MFRM). The results demonstrated the constructive impact of training programs in reducing raters' biases and increasing their consistency measures. This study indicated that inexperienced raters benefited more from a training program than experienced raters and thus achieved higher measures of consistency afterward. It also demonstrated a higher biased interaction for test takers on the extreme ends of the oral ability continuum. The findings demonstrated that it is almost impossible to completely eradicate rater variability even through rater training. Therefore, rater training should be viewed as a procedure to establish within-rater consistency rather than between-rater consistency. Since this study showed that inexperienced raters can rate even more reliably than experienced ones after training, there is no evidence whereby decision makers can exclude inexperienced raters solely because of their lack of adequate experience. Consequently, decision makers need to use their budgets for establishing rater training programs for inexperienced raters instead.

Keywords: bias, feedback, interrater reliability, intra-rater reliability, multifaceted Rasch measurement (MFRM), rater severity/leniency, rater training



JACET
KAFLE
KOTESOL
MELTA
ThaiTESOL
HAAL
TEFLIN
단체명 : AsiaTEFL (아시아영어교육학회)
고유번호 : 605-82-77130
소재지 : 광주광역시 광산구 호남대길 59, 1층 4101호
대표자 : 박주경
고객센터 : canteras@naver.com 043-230-3543 (09:00~18:00)