Report on the 1st Workshop on Reaching Efficiency in Neural Information Retrieval (ReNeuIR 2022) at SIGIR 2022

Bruch S., Lucchese C., Nardini F. M.

Settore INF/01 - Informatica  Management Information Systems  Efficiency-effectiveness tradeoffs  Hardware and Architecture  Settore ING-INF/05 - Sistemi di Elaborazione delle Informazioni 

As Information Retrieval (IR) researchers, we not only develop algorithmic solutions to hard problems, but we also insist on a proper, multifaceted evaluation of ideas. The IR literature on the fundamental topic of retrieval and ranking, for instance, has a rich history of studying the effectiveness of indexes, retrieval algorithms, and complex machine learning rankers and, at the same time, quantifying their computational costs, from creation and training to application and inference. This is evidenced, for example, by more than a decade of research on efficient training and inference of large decision forest models in Learning to Rank (LtR). As we move towards even more complex, deep learning models in a wide range of applications, questions on efficiency have once again become relevant with renewed urgency. Indeed, efficiency is no longer limited to time- and space-efficiency; instead it has found new, challenging dimensions that stretch to resource-, sample- and energy-efficiency with ramifications for researchers, users, and the environment. As a step towards bringing together experts from industry and academia and creating a forum for a critical discussion and the promotion of efficiency in the era of Neural Information Retrieval (NIR), we held the ReNeuIR workshop on July 15th, 2022 as a hybrid event---in person in Madrid, Spain along with online attendees---in conjunction with ACM SIGIR 2022. Recognizing the importance of this topic, approximately 80 participants answered our call and attended the workshop over three sessions. The event included a total of two keynotes and eight paper presentations, and concluded with a lively discussion where participants helped identify gaps in existing research and brainstormed future research directions. We had consensus in recognizing that efficiency is not simply latency, that a holistic, concrete definition of efficiency is needed to guide researchers and reviewers alike, and that more research is necessary in developing efficiency-centered evaluation metrics and standard benchmark datasets, platforms, and tools.

Source: SIGIR forum 56 (2022): 1–14. doi:10.1145/3582900.3582916

Publisher: Association for Computing Machinery,, New York, N.Y. , Stati Uniti d'America

Metrics