Retrieval Augmented Generation (RAG) has emerged as a key technology to mitigate the issues that Large Language Models (LLMs) face when they lack adequate knowledge. Given a user’s request, a RAG system searches auxiliary sources to augment the prompt associated with the request with relevant content. RAG is attracting a great deal of attention from the AI community, yet it is still hard to assess the quality of RAG systems in a systematic manner.
The goal of the LiveRAG Challenge is to allow research teams across academia and industry to advance their RAG research and compare the performance of their solutions with other teams, on a fixed corpus (derived from the publicly available FineWeb) and a fixed open-source LLM, Falcon3-10B-Instruct.
The SIGIR’2025 LiveRAG Challenge is organized by TII (Technology Innovation Institute) with support from AI71, AWS, and Pinecone. It requires an application process, after which selected teams will be
- Awarded up to 1500 USD in AWS compute credits to train their RAG solution, and up to 750 USD in Pinecone compute credits to use/generate their RAG indices.
- Given early access to TII’s DataMorgana tool to help them generate synthetic benchmarks for training and testing.
During the Live Challenge Day, the teams will be provided with a stream of unseen questions and will have to return their answers under strict response-time constraints. Finalists will be requested to present their results at the LiveRAG workshop day to be held at the SIGIR'2025 conference, during which winners will be announced and prizes will be awarded.