From PhD Research to International Audience: My Learning Journey on Educational Assessment and Rasch Measurement Theory
By Muhammad Affan Ramadhana
I was driving when the email notification from PROMS 2025 committee arrived. I peeked through my smartwatch but could only see the subject headline: Congratulations – Distinguished Student Scholarship PROMS2025.
Having arrived home, I tried to read the email word-by-word, in case I misread it. Alhamdulillah. My plan to present my research at the conference is saved. I was awarded travel grant to attend and present at the conference. This award is given by Pacific Rim Objective Measurement Society (PROMS) and Singapore University of Social Sciences (SUSS) as the committee of PROMS conference this year. I did not expect much when I submitted the application. I just thought that it is better to fail while trying than to regret never trying.
“The award recognizes outstanding students across Southeast Asia who have demonstrated academic excellence, research potential, and a commitment to advancing the field of measurement and assessment.” (PROMS 2025 website).
My Research Background
I was trained academically in English language education field. I obtained my Bachelor of Education degree in 2011 and my Master of Education in 2014, both from Universitas Negeri Makassar (UNM), South Sulawesi province, Indonesia. Since 2022, I was admitted to the PhD Program at the Faculty of Education, Universitas Islam Internasional Indonesia (UIII). I did my undergraduate thesis on English test analysis using classical test theory. Now, my doctoral research focuses on educational assessment and evaluation, with specific interest in language testing and assessment, rater-mediated assessment, and Rasch measurement theory.
It was in the Educational Assessment class, when I was first introduced to Rasch measurement model by Dr. Bambang Sumintono. As one of prominent figures in Rasch measurement in Indonesia, he clearly explained several traditions in measurement model and present a solid argument on why using raw score in measurement is problematic. Later in the following weeks, I was introduced to multi-rater assessment – a concept that I later used in my dissertation research.
My dissertation specifically explores assessment behavior of Indonesian university lecturers when grading English essays. This research focus emerged from my experience listening to stories of how my colleagues admit inconsistencies when assessing students’ writing. In rater-mediated assessment, inconsistencies and biases can be a threat to validity, reliability, and fairness of the assessment, which can impact the students’ future opportunities.
My dissertation research utilizes Many-Facet Rasch Measurement (MFRM) analysis, which is a modern psychometric model built on the idea that observed ratings are influenced by multiple facets, where each contributes to the final measurement. These facets include examinee ability, rater severity, and assessment rubric criteria. MFRM allows estimation of the logit scale for each facet and adjustment for inconsistencies. This model is important for identifying systematic biases or interactions that may be introduced by indirect factors.
From PROMS 2024 to PROMS 2025: My First Steps in the International Arena
Participating in PROMS 2024 provided a very valuable first-time experience for me. PROMS, as a scholarly society that promotes objective measurement and Rasch measurement, provides a forum for researchers from various backgrounds to share their works. In the 2024 conference, I directly witness the presence of leading experts in the field of Rasch measurement, such as Prof. Trevor G Bond (Founder of PROMS), Prof. George Engelhard (President of PROMS), and Prof. Yan Zi (Vice-President of PROMS), among others. These names I previously knew from books and journal articles only.
I was just started to deeply learn about this field when I participated in PROMS 2024 in Kuala Lumpur, Malaysia last year. I actually did not understand most of the theory when I presented my preliminary study. It was just my strategic action to gain insights and measure the worthiness of my plan and preparation for dissertation research. But I did my best. The feedback that I received, both from Rasch experts and from language assessment experts, resulted in substantial refinements to the analytical framework and methodology of my dissertation research. This experience also strengthened my determination to continue exploring this field.
Upon browsing the PROMS 2025 website, I noticed that the committee of this year’s conference opened applications for Distinguished Student Scholarship. I have just finished analyzing data from the first phase of my dissertation research when I saw the opportunity. I immediately prepared my extended abstract, curriculum vitae, and most importantly, recommendation letter from dissertation supervisor. He strongly supported me and believed that participating in PROMS 2025 will provide valuable feedback at the crucial stage of my dissertation research.
In PROMS 2025, I will present the result of first phase of my dissertation research. In the study, I analyzed assessment behavior of 36 university lecturers in Indonesia. They were tasked to assess English essays written by undergraduate students from various Asian countries. The assessment was conducted using standard analytical rubric with five categories (content, organization, vocabulary, language use, and mechanics).
The analysis indicates the presence of at least three groups of raters based on their level of severity/leniency with significant differences across raters. The statistical fit analysis also identified raters who exhibited inconsistent scoring behavior (overfit and misfit). The most interesting finding of this study was the identification of biases between rater background and scoring criteria. The results show different assessment behaviors of lecturers based on their academic qualifications, field of study, professional rank, and gender. This study is expected to offer practical insights for assessment committees to minimize potential biases, especially when designing multi-rater English writing assessment programs
My Future Plans
Receiving this scholarship will provide me with the opportunity to present some of my dissertation findings at the PROMS 2025 conference. The main event will be held on July 22-23, 2025, located in Singapore University of Social Sciences (SUSS) campus. The Distinguished Student Scholarship presentation session is scheduled for the afternoon of Day One. I believe the scholarship will provide me with a platform to gain insights and interact with leading experts and colleagues from across the Pacific region and beyond. I believed the forum will help me expand my professional network into the field of objective measurement.
I am very grateful for this opportunity and the recognition given through the PROMS 2025 Distinguished Student Scholarship. I believe research focused on improving educational practices, no matter how small, can have international relevance. I hope to make the most of this opportunity to learn from experts, share my research findings, and continue to contribute to the development of the field of educational measurement and language assessment, particularly within Rasch measurement theory.