Faculty of Education’s PhD Students Present Papers on Rasch Analysis Conference:  Reflection on PROMS 2024

Attending the Pacific Rim Objective Measurement Society (PROMS) 2024 conference on 20-21 August was a very enriching experience for us as a group. The theme of this year’s PROMS was Advancing Rasch Measurement Practically and Responsibly which was held at UCMI City Campus, Kuala Lumpur, Malaysia. This was the 19th PROMS event. God willing, next year’s PROMS will be held in Singapore. As four people who are deeply immersed in the field of measurement in education, we are very proud to have the opportunity to interact with global experts. Some of the keynote speakers were our role models in learning and whose books we refer to. They were Prof. Trevor G Bond (Founder of PROMS), Prof George Engelhard (University of Georgia), Dr Haniza Yon (Malaysian Psychometrics Association), Harris Shah Abd Hamid, Ph.D. (University College of MAIWP) and our role model lecturer in Educational Assessment from Universitas Islam Internasional Indonesia, Bambang Sumintono, Ph.D. These reflections aim to capture the essence of our collective experience, highlighting the key insights we gained, the challenges we faced, and the implications for our future work.

Diverse Perspectives and Global Collaboration

The opening ceremony on the morning of August 20th set a collaborative and inspiring tone for the conference. Welcoming speeches by Dr. Harris Shah Abd Hamid, Dean of FPPK, and Prof. Dr. George Engelhard, President of PROMS, emphasized the importance of cross-disciplinary and cross-cultural collaboration in advancing educational measurement practices. Throughout the two-day conference, we participated in numerous parallel sessions held in various rooms across the campus. These sessions provided platforms for researchers from different backgrounds to present their work. Each parallel room is chaired by a local committee and attended by at least one Rasch expert who are also PROMS board member, giving input and facilitating rich discussions and exchanges of ideas. We were particularly inspired by the presentations that showcased innovative applications of Rasch modeling in diverse educational contexts, highlighting how these techniques can be adapted to meet the specific needs of learners around the world.

Four delegates from UIII PhD students presented papers with their respective scientific fields but used the Rasch Model as a data analysis technique. All the presented papers are the result coming from the process of joining class in Educational Assessment which was taught by our lecturer at Faculty of Education at Universitas Islam Internasional Indonesia. We challenge ourselves to present in this global conference to learn further and get insight internationally. In the first parallel session, Faradillah Haryani presented a paper entitled: Spatial Orientation Assessment: Investigating Differences Between Residents Of Jogja and Non-Jogja Areas Using Rasch Measurement Model, then in a different parallel class Novinta Nurulsari presented with the title paper Misconception Detection Using Rasch Modeling in Thermodynamics, then still in the same class Eka Yusmaita presented with the title of the paper Chemical Literacy Assessment on Green Chemistry Topic Using Many-Facet Rasch (MFRM) and finally Muhammad Affan Ramadhana presented Rating Behaviour Difference Between Lecturers from ‘Language Education’ and ‘Linguistics/Literature’ Background in an Online EFL Writing Assessment Context.

This conference provided us for networking and building connections with fellow attendees. Informal interactions during breaks and the PROMS Dinner at Rumah Tangsi on the evening of August 20th allowed us to engage in meaningful conversations, discuss potential collaborations, and broaden our professional network. These interactions underscored the collaborative spirit of PROMS and reinforced our commitment to contributing to a global community dedicated to advancing objective measurement in education.

Key Insights and Personal Reflections

One of the most memorable aspects of the conference was the chance to meet some of the world’s leading experts in Rasch measurement theory. Interacting with prominent scholars like Prof. Dr. George Engelhard, Prof. Dr. Trevor Bond, and Prof. Dr. Yan Zi was both inspiring and humbling. Additionally, seeing our own lecturer, Dr. Bambang, among these figures further reinforced the importance of our work. Dr. Bambang himself delivered an insightful keynote on the second day of the conference (21st August 2024), which was followed by, yet another thought-provoking session led by Dr. Harris Shah Abd Hamid later that afternoon.

Novinta said, “I was particularly impressed by the quality of the keynote presentations. Every speaker brought forward innovative ideas and advanced research, but one of the most captivating sessions for me was on the ‘Next Generation of Attitude Measurement.’ The discussions opened my eyes to new possibilities in the field and underscored the importance of evolving our methods to stay relevant in an ever-changing educational landscape. In addition to the keynote speeches, the presentations by other speakers were outstanding. I gained a wealth of new knowledge, with one of the most intriguing topics being unfolding assessment. This concept was new to me and captured my attention due to its potential applications in my own work. Presenting my research at this prestigious event was a particularly valuable experience. Sharing my findings with an international audience and receiving feedback from experts in the field provided me with new perspectives on my research. This experience not only boosted my confidence but also inspired me to continue exploring and refining my work.”

Eka, on the other hand revealed that “This is a very high-quality conference that I have visited, I followed it solemnly. Every presenter who presented, the expert would provide supporting input and even explain the theory of the Rasch Model on the Whiteboard until we were amazed at their delivery. We hope that in the future we can remain part of the Rasch Model community and can continue to attend this conference for the following years.”

Faradillah observed that the Rasch experts in each session provided feedback in a manner that was not only insightful but also encouraging. Even in sessions where the focus was not primarily on Rasch, the experts skillfully demonstrated how the method could be applied to enhance the research, pointing out areas where existing approaches were lacking. This approach showed that the prominent figures in these discussions were open to different paradigms and encouraged deeper exploration rather than dismissing alternative views. Faradillah was particularly pleased with the positive reception of her presentation, even though it was recognized that further work, such as increasing the sample size, was necessary. The findings resonated with the international audience, as they see unique cultural aspects of Indonesian people, measured through Rasch modeling. Additionally, Faradillah gained valuable insights from Professor Trevor Bond, who emphasized that Reliability is number, and Validity is more into number. This point was especially important, as it challenged the common practice of describing reliability and validity solely with numerical descriptions.

Finally, Affan presented his work on how lecturers from language education and lecturers from linguistics/literature background behave differently in their rating severity in the context of writing assessment. His work is based on a preliminary study conducted as preparation for his dissertation research. “I was a bit nervous at the beginning, seeing many people including some Rasch experts in the room. I tried to explain as clearly as I could, but I think I was too fast.” Affan said.

Affan continued, “But it was insightful for me when one of the audience members asked for clarification if the rater participants in my study were prepared with the rubric or not. I said no. Then, he emphasized the possibility of raters who do not understand clearly the meaning of each category in the rubric might cause errors and became the most important factor that causes their rating behavior.” It was later revealed that he is Jeffrey Durant, one of Rasch expert from Toyo Gakuen University Japan, whose works focus on multirater assessment and rater network construction in large-scale multirater assessment. “He said that my study was interesting, and he gave me his contact for further discussion.”

Implications for Future Work

The insights gained from PROMS 2024 will undoubtedly shape our future work. The advanced understanding of Rasch modeling will be directly applicable to our ongoing research projects. For Novinta, this will enhance her efforts in detecting misconceptions in educational settings. Faradillah will be focusing on increasing the sample size to enable Differential Item Functioning (DIF) analysis. Eka plans to expand the study of the Multi-Facet Rasch Model and increase the number of persons to improve item reliability scores. Affan will also leverage these insights to further his research in multirater assessment, especially in preparing rater training and more detailed analysis on rater behavior, by applying Rasch modeling to achieve more robust results. We are now better equipped to refine our methodologies and improve the accuracy and fairness of our assessments.

Conclusion

Overall, PROMS 2024 was a transformative experience for us as a group. The opportunity to engage with leading experts, learn about cutting-edge research, and reflect on our own practices has had a profound impact on our professional development. We leave the conference with a renewed sense of purpose and a commitment to applying the knowledge and insights gained to our future work. We are excited about the potential for future collaborations and look forward to continuing our journey in the field of educational measurement with a deeper understanding and a broader perspective.