Review Data Collection at *ACL
This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL. The documents have been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).
Updates
- March 2022: Progress report and amendments to the original proposal in terms of data collection, access and storage.
Summary
We propose two initiatives aiming at facilitating ethical, consent-driven research in NLP for peer reviewing, and peer reviewing for NLP. The first initiative concerns the collection of peer review and manuscript data from *ACL events. We cover the major challenges in this process and propose solutions, as well as report on a pilot study conducted at COLING-2020, where some of those solutions were implemented. The second initiative aims to provide a structured way to study and compare peer reviewing workflows in the *ACL community. For this we propose to create a centralized repository of anonymized numerical data from ACL conferences. This repository can serve as an anchor point for future conference organizers and provide a bridge to the meta-science and science-of-science communities that have been working on similar issues in other disciplines and research fields.
Documents
- The slide deck briefly summarizes the two proposals;
- The full document elaborates on the proposals in further detail. In addition, we provide
- the example License agreements for blind submission data
- and peer reviewing data.
Contact
If you have commentaries, questions or suggestions regarding the proposal, do not hesitate to contact:
Prof. Dr. Iryna Gurevych, gurevych@ukp.informatik.tu-darmstadt.de
Dr. Ilia Kuznetsov, PhD kuznetsov@ukp.informatik.tu-darmstadt.de