Researchers recognized at CHI for work on human-NLP system to create reading quiz questions

Xinyi Lu

Simin Fan

Jessica Houghton

Prof. Lu Wang

Prof. Xu Wang

A research team led by Prof. Xu Wang and including Prof. Lu Wang; Xinyi Lu, undergraduate student in computer science; Simin Fan (BSE CS 2022), PhD student at EPFL; and Jessica Houghton (BSE CS 2022), Program Manager at Microsoft has received an Honorable Mention at the ACM CHI ‘23 conference for their paper, “ReadingQuizMaker: A Human-NLP Collaborative System that Supports Instructors to Design High-Quality Reading Quiz Questions.”

The team studies ways to enable human-AI collaborative question generation to increase the adoption of question generation techniques in classrooms. The study focused on question generation based on readings, as they are so prevalent in college classes. 

The team’s prior work, “Towards Process-Oriented, Modular, and Versatile Question Generation that Meets Educational Needs,” found reasons why the adoption of automatic question generation systems in practice is low. Specifically, when hand-writing questions, instructors leverage a variety of information, including their teaching objectives, students’ misconceptions, and others, which are not considered by automatic approaches.

This led to the development of ReadingQuizMaker, which uses a human-AI collaborative approach. ReadingQuizMaker adapts to an instructor’s natural workflow of creating questions while providing NLP-based process-oriented support. The system enables instructors to decide when and which NLP models to use, to select the input to the models, and to edit the outcomes. 

The system’s interface allows an instructor to select text from a reading from one browser panel and send it to a question authoring panel. From there, the instructor has options for creating a question and using one of three levels of NLP support with immediate preview and editing options. In addition to describing the system in their paper, the team also demonstrates the system in this video.

In the evaluation study, instructors found the questions developed through the system to be comparable to their previously designed quizzes, while saving time to create. Instructors praised ReadingQuizMaker for its ease of use, and considered the NLP suggestions to be satisfying and helpful. 

The researchers compared ReadingQuizMaker with an automatic question generation technique, where instructors were given automatically generated questions to edit, and instructors showed a strong preference for the human-AI teaming approach provided by ReadingQuizMaker.  Instructors liked to have more control in the process of creating educational materials that align with their goals and felt reluctant to directly use AI-generated questions in their classes. Instructors found the auto-generated questions to be of lower quality. With generative models, many of the texts lose context and instructors had to spend more time reading the generated texts and matching them with the original text to make sure they are accurate.

With interest increasing in the use of using large language models in education, the overall findings suggest that for high-stakes tasks such as educational content creation, allowing users to provide input and giving them sufficient control is more preferable to fully automated approaches.