From PhD Research to International Audience: My Learning Journey on Educational Assessment and Rasch Measurement Theory
From PhD Research to International Audience: My Learning Journey on Educational Assessment and Rasch Measurement Theory
By Muhammad Affan Ramadhana
I was driving when the email notification from PROMS 2025 committee arrived. I peeked through my smartwatch but could only see the subject headline: Congratulations – Distinguished Student Scholarship PROMS2025.
Having arrived home, I tried to read the email word-by-word, in case I misread it. Alhamdulillah. My plan to present my research at the conference is saved. I was awarded travel grant to attend and present at the conference. This award is given by Pacific Rim Objective Measurement Society (PROMS) and Singapore University of Social Sciences (SUSS) as the committee of PROMS conference this year. I did not expect much when I submitted the application. I just thought that it is better to fail while trying than to regret never trying.
“The award recognizes outstanding students across Southeast Asia who have demonstrated academic excellence, research potential, and a commitment to advancing the field of measurement and assessment.” (PROMS 2025 website).
My Research Background
I was trained academically in English language education field. I obtained my Bachelor of Education degree in 2011 and my Master of Education in 2014, both from Universitas Negeri Makassar (UNM), South Sulawesi province, Indonesia. Since 2022, I was admitted to the PhD Program at the Faculty of Education, Universitas Islam Internasional Indonesia (UIII). I did my undergraduate thesis on English test analysis using classical test theory. Now, my doctoral research focuses on educational assessment and evaluation, with specific interest in language testing and assessment, rater-mediated assessment, and Rasch measurement theory.
It was in the Educational Assessment class, when I was first introduced to Rasch measurement model by Dr. Bambang Sumintono. As one of prominent figures in Rasch measurement in Indonesia, he clearly explained several traditions in measurement model and present a solid argument on why using raw score in measurement is problematic. Later in the following weeks, I was introduced to multi-rater assessment – a concept that I later used in my dissertation research.
My dissertation specifically explores assessment behavior of Indonesian university lecturers when grading English essays. This research focus emerged from my experience listening to stories of how my colleagues admit inconsistencies when assessing students’ writing. In rater-mediated assessment, inconsistencies and biases can be a threat to validity, reliability, and fairness of the assessment, which can impact the students’ future opportunities.
My dissertation research utilizes Many-Facet Rasch Measurement (MFRM) analysis, which is a modern psychometric model built on the idea that observed ratings are influenced by multiple facets, where each contributes to the final measurement. These facets include examinee ability, rater severity, and assessment rubric criteria. MFRM allows estimation of the logit scale for each facet and adjustment for inconsistencies. This model is important for identifying systematic biases or interactions that may be introduced by indirect factors.
From PROMS 2024 to PROMS 2025: My First Steps in the International Arena
Participating in PROMS 2024 provided a very valuable first-time experience for me. PROMS, as a scholarly society that promotes objective measurement and Rasch measurement, provides a forum for researchers from various backgrounds to share their works. In the 2024 conference, I directly witness the presence of leading experts in the field of Rasch measurement, such as Prof. Trevor G Bond (Founder of PROMS), Prof. George Engelhard (President of PROMS), and Prof. Yan Zi (Vice-President of PROMS), among others. These names I previously knew from books and journal articles only.
I was just started to deeply learn about this field when I participated in PROMS 2024 in Kuala Lumpur, Malaysia last year. I actually did not understand most of the theory when I presented my preliminary study. It was just my strategic action to gain insights and measure the worthiness of my plan and preparation for dissertation research. But I did my best. The feedback that I received, both from Rasch experts and from language assessment experts, resulted in substantial refinements to the analytical framework and methodology of my dissertation research. This experience also strengthened my determination to continue exploring this field.
Upon browsing the PROMS 2025 website, I noticed that the committee of this year’s conference opened applications for Distinguished Student Scholarship. I have just finished analyzing data from the first phase of my dissertation research when I saw the opportunity. I immediately prepared my extended abstract, curriculum vitae, and most importantly, recommendation letter from dissertation supervisor. He strongly supported me and believed that participating in PROMS 2025 will provide valuable feedback at the crucial stage of my dissertation research.
In PROMS 2025, I will present the result of first phase of my dissertation research. In the study, I analyzed assessment behavior of 36 university lecturers in Indonesia. They were tasked to assess English essays written by undergraduate students from various Asian countries. The assessment was conducted using standard analytical rubric with five categories (content, organization, vocabulary, language use, and mechanics).
The analysis indicates the presence of at least three groups of raters based on their level of severity/leniency with significant differences across raters. The statistical fit analysis also identified raters who exhibited inconsistent scoring behavior (overfit and misfit). The most interesting finding of this study was the identification of biases between rater background and scoring criteria. The results show different assessment behaviors of lecturers based on their academic qualifications, field of study, professional rank, and gender. This study is expected to offer practical insights for assessment committees to minimize potential biases, especially when designing multi-rater English writing assessment programs
My Future Plans
Receiving this scholarship will provide me with the opportunity to present some of my dissertation findings at the PROMS 2025 conference. The main event will be held on July 22-23, 2025, located in Singapore University of Social Sciences (SUSS) campus. The Distinguished Student Scholarship presentation session is scheduled for the afternoon of Day One. I believe the scholarship will provide me with a platform to gain insights and interact with leading experts and colleagues from across the Pacific region and beyond. I believed the forum will help me expand my professional network into the field of objective measurement.
I am very grateful for this opportunity and the recognition given through the PROMS 2025 Distinguished Student Scholarship. I believe research focused on improving educational practices, no matter how small, can have international relevance. I hope to make the most of this opportunity to learn from experts, share my research findings, and continue to contribute to the development of the field of educational measurement and language assessment, particularly within Rasch measurement theory.
Workshop: Multidimensional Rasch Analysis with Conquest
Dear Students
We're delighted to announce a Workshop on Multidimensional Rasch Model Analysis with Conquest.
Details:
* Date: Tuesday, June 3, 2025
* Time: 13:00-16:00 (Jakarta Time)
* Place: Teleconference, Faculty A UIII
Limited seats available!
To secure your seat, please fill out the Google Form linked below. Only the first 30 registrants will be accepted.
https://forms.gle/rdLokiqj3dyVnaCH7
Thank you
UIII Hosts Book Forum Highlighting Gender, Disability, and Religious Diversity in Classrooms
UIII Hosts Book Forum Highlighting Gender, Disability, and Religious Diversity in Classrooms
May 15, 2025
Contributor: Supriyono | Editor: Dadi Darmadi | Photo: Alief Fathan
In an effort to amplify classroom voices and enrich the national conversation on students’ literacy and inclusivity, the Jusuf Kalla Library at UIII held a Book Forum featuring the authors of ‘Putri, Angpau dan Topi Duka’. The event offered a space for reflection and dialogue on how children’s literature can be a powerful medium to discuss gender, disability, and religious diversity in educational settings.
The book, written in Bahasa Indonesia by Dr. Sofie Dewayani, Dr. Tati L. Durriyah, and Asep Ropiudin, was the product of a two-year collaborative research project involving educators from multiple universities and foundations across Indonesia. Moderated by Asep Ropiudin himself, the discussion centered around how classrooms can become spaces to introduce social issues for children utilizing meaningful stories and literature.
“Seeing this book physically—it feels like a dream,” said Dr. Tati L. Durriyah, recalling the project’s beginnings during the COVID-19 pandemic in 2022. “It took years of hard work, delays, and collaborative efforts. We designed this book specifically for practitioners, using accessible language and authentic classroom experiences.”
The book follows the experiences of three master teachers from various regions of Indonesia as they navigate discussions on identity and inclusion through literature in their classrooms. These stories, the authors argue, are often overlooked in national discourses around students’ literacy, which tend to focus on rankings and generalized assumptions.
Dr. Sofie Dewayani emphasized the importance of listening to students’ stories, noting, “When we zoom in to the classroom level, there are powerful narratives created by children. If we truly listen to them, we’ll realize they are literate—they just need something to connect with.”
She shared one example from the book in which a teacher introduced the story of Ratna Komala, a tale that reverses traditional gender roles with a brave sister rescuing her brother. “This flips the usual portrayal of women as passive figures awaiting rescue,” Dr. Dewayani explained. “And it prompted a classroom conversation about women’s roles in society, even touching on issues like why women might not receive holiday bonuses.”
The forum also highlighted another story in the book about the friendship between an Indonesian girl and an Afghan refugee—used in class to foster dialogue about borders, identity, and shared humanity. “We believe a book comes to life only when a teacher facilitates critical discussion,” said Dr. Dewayani.
In a time when there is growing concern over students’ literacy rates and reading habits in Indonesia, the forum underscored the importance of going beyond numbers to spotlight lived experiences and classroom conversations. By sharing these teacher-led stories, the book encourages educators to approach literacy not just as a skill, but as a gateway to critical thinking, empathy, and inclusivity.
Professor Yan Zi’s Call for Rethinking Assessment in the Age of AI
Professor Yan Zi’s Call for Rethinking Assessment in the Age of AI
May 14, 2025
Contributor: Supriyono | Editor: Dadi Darmadi | Photo: Achmad Jatnika
At the 4th Annual Conference hosted by UIII Faculty of Education, Professor Yan Zi of the Department of Curriculum and Instruction at The Education University of Hong Kong delivered a keynote that challenged the status quo of classroom assessment. His central argument: the future of effective learning in a digital age lies in synergising teacher-directed and student-initiated formative assessments, especially as education embraces the capabilities—and caveats—of generative AI.
In a presentation titled “Synergising Teacher-directed and Student-initiated Assessment in the Digital Learning Environment,” Professor Yan offered a research-grounded framework for what he called a “meaningful assessment ecology.” This ecology, he explained, must blend two major forces, namely teacher-driven assessment-for-learning and student-led assessment-as-learning.
Traditionally, formative assessment has relied heavily on teacher direction where teachers monitor student learning and adjust teaching strategies accordingly. This model has value, said Professor Yan, but it alone cannot unlock the full potential of student learning.
To complement this, Professor Yan spotlighted student-initiated assessment—or assessment-as-learning—where learners actively engage in self-assessment and metacognitive reflection. Drawing from his work developing the Self-assessment Practices Scale (SaPS), he demonstrated how this approach empowers students to take ownership of their learning, fosters self-regulated learning (SRL), and enhances academic outcomes.
Crucial for student-initiated assessment, Professor Yan highlighted the interplay between feedback literacy and self-assessment. He argued that these are not separate domains; feedback literacy fuels effective self-assessment, and vice versa. He elaborated that both processes centered on active student involvement and co-regulation, challenging conventional views of feedback as a one-way street from teacher to student.
In regards to feedback, Professor Yan then elaborated the role of generative AI—particularly tools like ChatGPT—in reshaping feedback and assessment. Generative AI, he explained, has the potential to make feedback more timely, personalized, and accessible. It democratizes assessment by shifting power to the learner.
However, he offered several cautionary notes, saying that students may struggle with using appropriate prompts, evaluating AI-generated responses, or may become overly dependent on AI outputs without critical reflection. “ChatGPT may be a game-changer,” he said, “but it still depends on the player.”
A recent study cited by Professor Yan revealed that while students showed cognitive engagement with ChatGPT, their behavioral engagement was often superficial. Emotional resistance was low, but trust levels varied. He warned that without proper cultivation of feedback literacy, students may not reap the full benefits of AI-enhanced learning environments.
Hence, Professor Yan called for a “bottom-up assessment reform,” advocating for an assessment culture that is not only methodologically diverse—encompassing theoretical, quantitative, and qualitative approaches—but also deeply grounded in stakeholder engagement, including students, teachers, and parents.
Professor Yan’s lecture marked a significant intellectual moment in the ongoing dialogue on education transformation at UIII. As digital tools become increasingly embedded in pedagogy, his message serves as both a roadmap and a reminder: technology should enhance, not replace, the human dimensions of teaching and learning.
source: https://uiii.ac.id/professor-yan-zis-call-for-rethinking-assessment-in-the-age-of-ai/
Dr. Bahrul Hayat: Reimagining Educational Assessment as a Tool for Equity
Dr. Bahrul Hayat: Reimagining Educational Assessment as a Tool for Equity
May 10, 2025
Contributor: Supriyono | Editor: Dadi Darmadi | Photo: Achmad Jatnika
In a compelling presentation at the 4th International Conference organized by UIII Faculty of Education, Dr. Bahrul Hayat, large-scale education assessment expert and former Secretary General of Indonesia’s Ministry of Religious Affairs, emphasized the transformative potential of educational assessment when it is designed and implemented with equity at its core.
Drawing from his rich background in education policy and research, the former UIII Vice- Rector for Academic Affairs argued that assessment is more than a tool for grading—it is a mechanism that shapes the trajectory of learning, access, and opportunity across the education system.
He began by outlining the functions of educational assessment, saying beyond individual student evaluations, assessments are used on a larger scale to track the quality of education systems. When used effectively, he said, assessment provides evidence of how well students are achieving learning objectives and guides decisions related to learning placement, diagnostics, grade promotion, and certification.
However, Dr. Hayat warned its potential drawback, saying that while assessment can help identify learning achievement gaps, support responsive teaching, and inform targeted interventions, they risk to marginalize students by misrepresenting their abilities or limit their opportunities if they are designed with bias, low validity, or without contextual sensitivity. “Assessment has a dual role in educational equity,” he explained. “It can be a tool for justice or a vehicle for exclusion.”
Reflecting on current assessment practices, Dr. Hayat highlighted the multilayered threats. These include rigid item formats, content that lacks curricular relevance, differential item functioning across student groups, and construct-irrelevant variance. On the administrative level, challenges include uniform testing modes, non-adaptive assessments, lack of accommodations, and digital inequality. Additionally, scoring and interpretation practices—such as the use of raw scores, norm-referenced grading, and standardized rubrics—can also introduce bias, particularly when student diversity is not taken into account.
Hence, central to his presentation, Dr. Hayat called for the equitable practice of assessment. He stressed that assessments must be fair, contextualized, and aligned with clearly defined learning outcomes. Students should be given varied and meaningful opportunities to demonstrate their learning, according to their physical and psychological conditions.
To achieve this, Dr. Hayat outlined six foundational principles for equitable assessment: validity and reliability; transparency and accountability; accommodations for special needs; cultural fairness; student-centered design; and opportunity to learn. These principles, he emphasized, are non-negotiable if assessments are to serve their intended purpose of supporting all learners.
Quoting education experts like W. James Popham and Linda Darling-Hammond, Dr. Hayat emphasized that assessments should be tools for learning, not instruments of gatekeeping. “We must ensure that our assessments are fair and instructionally informative, rather than tools that perpetuate privilege or punish disadvantage,” he said. His call to action was clear: the design and implementation of educational assessments must be inclusive, context-aware, and grounded in evidence to genuinely support equitable outcomes for all students.
Dr. Hayat concluded his presentation by urging education stakeholders to rethink the role of assessment not as a final judgment but as an ongoing support mechanism. If done right, he noted, assessments can help reduce learning disparities, inform meaningful interventions, and empower every learner to reach their full potential. His message resonated deeply with UIII’s mission to spark dialogue on how to build a more just and responsive education system—starting with the way we assess.
source: https://uiii.ac.id/dr-bahrul-hayat-reimagining-educational-assessment-as-a-tool-for-equity/
How Does Culture Fit In? Rethinking Inclusive Assessment with Assoc. Prof. Joanna Tai
How Does Culture Fit In? Rethinking Inclusive Assessment with Assoc. Prof. Joanna Tai
May 10, 2025
Contributor: Supriyono | Editor: Dadi Darmadi | Photo: Achmad Jatnika
At the 4th Annual Conference on Assessment and Evaluation hosted by the UIII Faculty of Education, Associate Professor Joanna Tai of Deakin University delivered a thought-provoking keynote titled “Designing Assessment for Inclusion: How Does Culture Fit In?” Her message was clear: inclusive assessment must go beyond fairness in form and structure; it must account for the cultural contexts that shape learning, expression, and success.
Drawing from her research at the Centre for Research in Assessment and Digital Learning (CRADLE), she reconsidered long-held assumptions about objectivity and equality in academic assessment. “Assessment should recognise diversity in student learning and endeavour to ensure that no student is discriminated against by virtue of features other than their ability to meet appropriate standards,” she quoted her 2023 co-authored publication. But she followed with a more pressing question: “What about culture?”
In her keynote, Assoc. Prof. Tai unpacked hidden cultural assumptions in common assessment practices. Assessment, she argued, is not culturally neutral. Academic traditions, societal values, and educational systems all shape how students are expected to demonstrate knowledge. Yet, despite its significance, her research revealed a critical gap: most literature on assessment and inclusion fails to explicitly define “culture.”
A scoping review she conducted for relevant publication between 2005 and 2025 found only nine studies that meaningfully addressed cultural dimensions of assessment, despite the global push for equity and diversity in higher education. She also highlighted that the definitions of culture in the relevant literature were just assumed and not explicit.
She then introduced a set of emerging design principles for cultural inclusion in assessment, saying that, first, assessment must be future-oriented, preparing students for their real-world, context-specific futures. Second, it should foster shared social understanding, encouraging dialogue and reflection among students and educators about what is valued in learning—and why.
Assoc. Prof. Tai further advocated for collaborative assessments that promote intercultural exchange, especially in multicultural classrooms. For this, she introduced three strategies for assessment design, encapsulated in authenticity, programmatic view, and distinctiveness.
Authenticity in assessment, she said, involves aligning tasks with students’ values and goals (authenticity to self), setting clear expectations (authenticity of task), recognizing differing perceptions (designed vs perceived authenticity), and including diverse students in design (authenticity in our values).
For the programmatic view of assessment, she encouraged that learning outcomes are scaffolded over time, and not everything has to be assessed at every occasion. Meanwhile, distinctiveness highlight that students should be prepared for their unique destinations, utilizing open-ended tasks to develop their distinct capabilities and offering a variety of ways to demonstrate learning aligns with Universal Design for Learning.
Assoc. Prof. Tai’s keynote concluded with a reminder that designing assessment for inclusion is not a one-off event, but an ongoing, systemic process. It involves multiple stakeholders, continuous dialogue, and a willingness to challenge entrenched academic norms. Her presentation sparked thoughtful discussion throughout the conference and set a transformative tone for UIII’s broader vision of equitable, culturally responsive education.
Joanna Tai finds a wonderful welcome for her Assessment for Inclusion Keynote
Recently Associate Professor Joanna Tai was invited to give a keynote at the Universitas Islam Internasional Indonesia Faculty of Education 4th annual conference, which was about “Assessment and Equity in Education”. Here, she shares some of her reflections on the conference.
Members of Deakin and Universitas Islam Internasional Indonesia (UIII) have been partners and collaborators for several years, and I was excited to contribute to the growing relationship by accepting the invitation to be a keynote speaker at the UIII Faculty of Education 4th annual conference. UIII is a relatively new university and the Faculty of Education has so far accepted four cohorts of Masters and PhD students, who were all in enthusiastic attendance alongside their Faculty members. The Faculty are small in number but mighty in productivity: in addition to offering four theme streams across the Masters and PhD programmes, they also edit a journal, Muslim Education Review, host visiting scholars, run an annual conference, and are soon to be admitting undergraduate students.
It was my first time visiting Indonesia, despite it being one of our closest neighbours, and I’m glad I had such welcoming hosts. UIII is situated in Depok, about one hour from Jakarta airport, and I was collected and returned to the hotel each day by a pair of volunteer students. The roads and traffic were definitely something I wouldn’t have wanted to navigate on my own! On arrival, the campus seemed vast, and the buildings were impressive – including an eight-floor library! Being an international university, they are building a collection across many languages – Bahasa Indonesia, English, Arabic, and several others.
So now, to the conference itself. After a lovely opening ceremony with a traditional Indonesian dance performance representing the different regions (put on by students from the Faculty!), I had the honour of being the opening keynote speaker. With a topic like designing assessment for inclusion, it’s always hard to tell exactly which aspects will be most pertinent for a broad audience.
I focused my talk on considering how culture might be acknowledged and incorporated in inclusive ways
My work with our own CRADLE Indonesian students has helped me to understand how Indonesia is a place of many cultures and languages. This seemed to resonate with many of the attendees, and we had an excellent discussion session afterwards which demonstrated how attentive they’d been – even more impressive, since English (as I learned from some students with a linguistics background) is frequently a third, fourth, or even further down the list language for many people.
Professor Bahrul Hayat followed with his keynote immediately after, discussing the importance of large scale educational assessment and its potential impact on equity. It was great to hear the same underlying messages about the purposes of education from a different perspective. Professor Yan Zi from the Education University of Hong Kong (and CRADLE Honorary Professor) also gave his keynote to open the second day, highlighting the importance of student agency even within teacher-constructed systems, and offering implications for what we might do about learning and assessment in a digital world.
I had the opportunity to also attend some of the panel sessions, with presentations from scholars representing several institutions. Again, I was impressed by the quality of discussion and interaction amongst participants – and chuffed to see work by CRADLE colleagues and collaborators referenced in presentations.
Beyond the intellectual climate, the food was also delicious, with snacks in the morning, and a full cooked lunch showcasing typical Indonesian food. The conference dinner took place on the first day, and what started as a singing performance turned into karaoke for staff and students. Of course, I had to participate (thanks Dr Destina for being my duet partner!) too. One notable difference between Indonesia and Australia is the importance of photos: I was happy to pose with many students and now also have a great collection of memories from the conference. With the ongoing connection between institutions, I hope we’ll meet again in the future!