TREC RAGTIME
Last Updated: 6 December 2024
TREC RAGTIME is a TREC shared task to study and benchmark report generation from news (both English and multi-lingual).
Key features of the track are its focus on multi-faceted reports (going beyond factoid QA), and a citation-based evaluation (providing supporting evidence of claims made in the report).
It also benchmarks Cross-Language (CLIR) and Multi-lingual (MLIR) retrieval as supporting subtasks.
Languages: English, Arabic, Chinese, and Russian
Tasks
1. Multilingual Report Generation
Given a set of documents in multiple languages and a “report request” that describes the information need, generate a report in English that summarizes the information.
2. Monolingual English Report Generation
This task is similar to Multilingual Report Generation, but only English documents are used.
3. Retrieval (Cross-Language and Multi-lingual)
Given a set of documents in multiple languages and a “report request” that describes the information need, retrieve documents that help satisfy this information need.
Timeline
-
February 2025Document Collections Released
-
April 2025Track Guidelines Released
-
May 2025Dry run Topics Released
-
July 2025Dry run Submissions Due
-
July 2025Feedback on dry runs
-
August 2025Final Submissions Due
See Also: TREC NeuCLIR
TREC NeuCLIR (Neural CLIR) was the predecessor of TREC RAGTIME. In 2024, NeuCLIR included a pilot study of report generation. You can find more about NeuCLIR at the TREC NeuCLIR website.
Organizers
In alphabetical order:
- Dawn Lawrie, Johns Hopkins University, HLTCOE
- Sean MacAvaney, University of Glasgow
- James Mayfield, Johns Hopkins University, HLTCOE
- Douglas W. Oard, University of Maryland
- Luca Soldaini, Allen Institute for AI
- Eugene Yang, Johns Hopkins University, HLTCOE