Difference between revisions of "Review Data Collection at *ACL"
| Line 6: | Line 6: | ||
=== Documents === | === Documents === | ||
| − | # The [[Media:1._Review_collection_at_ACL_(slides).pdf| slide deck]] briefly summarizes the two proposals; | + | # The [[Media:1._Review_collection_at_ACL_(slides).pdf|slide deck]] briefly summarizes the two proposals; |
| − | # The [[Media:2._Review_collection_at_ACL.pdf| full document]] elaborates on the proposals in further detail. In addition, we provide | + | # The [[Media:2._Review_collection_at_ACL.pdf|full document]] elaborates on the proposals in further detail. In addition, we provide |
| − | # the example License agreements for [[Media:3._ACL_License_Blind_Submission_v1.pdf| | + | # the example License agreements for [[Media:3._ACL_License_Blind_Submission_v1.pdf|blind submission data]] |
| − | # and [[Media:4._ACL_License_Peer_Reviews_v2.pdf| | + | # and [[Media:4._ACL_License_Peer_Reviews_v2.pdf|peer reviewing data]]. |
=== Contact === | === Contact === | ||
Revision as of 03:11, 30 March 2021
This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL. The documents has been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).
Summary
We propose two initiatives aiming at facilitating ethical, consent-driven research in NLP for peer reviewing, and peer reviewing for NLP. The first initiative concerns the collection of peer review and manuscript data from *ACL events. We cover the major challenges in this process and propose solutions, as well as report on a pilot study conducted at COLING-2020, where some of those solutions were implemented. The second initiative aims to provide a structured way to study and compare peer reviewing workflows in the *ACL community. For this we propose to create a centralized repository of anonymized numerical data from ACL conferences. This repository can serve as an anchor point for future conference organizers and provide a bridge to the meta-science and science-of-science communities that have been working on similar issues in other disciplines and research fields.
Documents
- The slide deck briefly summarizes the two proposals;
- The full document elaborates on the proposals in further detail. In addition, we provide
- the example License agreements for blind submission data
- and peer reviewing data.
Contact
If you have commentaries, questions or suggestions regarding the proposal, do not hesitate to contact:
Prof. Dr. Iryna Gurevych, gurevych@ukp.informatik.tu-darmstadt.de
Ilia Kuznetsov, kuznetsov@ukp.informatik.tu-darmstadt.de