Review Data Collection at *ACL

This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL. The documents have been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).

Updates

• March 2022: Progress report and amendments to the original proposal in terms of data collection, access and storage.

Summary

We propose two initiatives aiming at facilitating ethical, consent-driven research in NLP for peer reviewing, and peer reviewing for NLP. The first initiative concerns the collection of peer review and manuscript data from *ACL events. We cover the major challenges in this process and propose solutions, as well as report on a pilot study conducted at COLING-2020, where some of those solutions were implemented. The second initiative aims to provide a structured way to study and compare peer reviewing workflows in the *ACL community. For this we propose to create a centralized repository of anonymized numerical data from ACL conferences. This repository can serve as an anchor point for future conference organizers and provide a bridge to the meta-science and science-of-science communities that have been working on similar issues in other disciplines and research fields.

Documents

1. The slide deck briefly summarizes the two proposals;
2. The full document elaborates on the proposals in further detail. In addition, we provide
3. the example License agreements for blind submission data
4. and peer reviewing data.

Contact

If you have commentaries, questions or suggestions regarding the proposal, do not hesitate to contact:

Prof. Dr. Iryna Gurevych, gurevych@ukp.informatik.tu-darmstadt.de
Dr. Ilia Kuznetsov, PhD kuznetsov@ukp.informatik.tu-darmstadt.de