Web Agents: A New Frontier for Embodied Agents

AI Seminar and Guest Lecture at CSE595 NLP Fall 2024

Location: George G. Brown Laboratories (GGBL) Building, Room 2505

Zoom: https://umich.zoom.us/j/9932414410

Meeting ID: 958 9853 8621

Passcode: aiseminar

Slides: https://docs.google.com/presentation/d/1YoFJhZuG1GdbAXtrR-o7Cf5uZrk_UgGVz9C0l_ZMqR4/edit#slide=id.g310e6f04209_2_70

Abstract:

The digital world, e.g., the World Wide Web, provides a powerful yet underexplored form of embodiment for AI agents, where perception involves understanding GUI renderings and the underlying markup languages, and effectors are mice and keyboards. Empowered by large language models (LLMs), web agents are rapidly rising as a new frontier for embodied agents that provide both the breadth and depth needed for driving agent development. On the other hand, web agents can potentially lead to many practical applications, thus also raising substantial commercial interest. In this talk, I will present an overview of web agents, covering the history, the promises, and the challenges. I will then give an in-depth discussion on multimodality and grounding. Finally, I will conclude with an outlook of promising future directions, including planning, synthetic data, and safety.

Bio:

Yu Su is a Distinguished Assistant Professor at the Ohio State University and a researcher at Microsoft. He founded and directs the OSU NLP group. He has broad interests in artificial and biological intelligence, with a primary interest in the role of language as a vehicle of thought and communication. His group is a leading force of LLM-based language agents, with many well-received works such as Mind2Web, SeeAct, HippoRAG, and MMMU. His work has received multiple paper awards, including Best Student Paper Award from CVPR and Outstanding Paper Award from ACL.