Answering Why Questions about Narrative Text

Abstract:

Being able to answer questions about why people perform particular actions is central to understanding and reasoning about narrative text. Despite recent progress in QA, it is unclear whether existing models have the ability to answer “why” questions, which generally require using commonsense knowledge external to the narrative, and inferring characters’ plans and goals. We have been developing novel data and methods for such why-question answering. TellMeWhy is a new crowd-sourced corpus consisting of more than 30k questions and free-form answers concerning why characters in short narratives perform the actions described. Given the limitations of automated evaluation for this task, we have also designed a systematized human evaluation interface for this dataset. Our evaluation of recent models shows that they are below human performance on answering such questions. We have also explored what aspects of the knowledge required to answer why questions are accessible in current large language models and what aspects can be made accessible via external commonsense-knowledge resources. Not surprisingly, larger models perform better, but all of the variable-sized models we explored benefited from the injection of question-specific knowledge extracted from the COMET knowledge base. We also observed that the best models produce a significant number of answers that humans rate as even better than human answers, but models also produce a significant number of terrible answers that are rated much lower than any human answer.

About the Speaker:

Raymond J. Mooney is a Professor in the Department of Computer Science at the University of Texas at Austin. He received his Ph.D. in 1988 from the University of Illinois at Urbana/Champaign. He is an author of over 200 published research papers, primarily in the areas of machine learning and natural language processing. He was the President of the International Machine Learning Society from 2008-2011, program co-chair for AAAI 2006, general chair for HLT-EMNLP 2005, and co-chair for ICML 1990. He is a Fellow of AAAI, ACM, and ACL and the recipient of the Classic Paper award from AAAI-19 and best paper awards from AAAI-96, KDD-04, ICML-05 and ACL-07.