Difference between revisions of "Review Data Collection at *ACL"
(Created page with "''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.'' ''The documents has been prepared by Iryna Gurevych and Ilia...") |
|||
| Line 1: | Line 1: | ||
''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.'' | ''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.'' | ||
''The documents has been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).'' | ''The documents has been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).'' | ||
| − | |||
=== Summary === | === Summary === | ||
We propose two initiatives aiming at facilitating ethical, consent-driven research in NLP for peer reviewing, and peer reviewing for NLP. The first initiative '''concerns the collection of peer review and manuscript data from *ACL events'''. We cover the major challenges in this process and propose solutions, as well as report on a pilot study conducted at COLING-2020, where some of those solutions were implemented. The second initiative aims to '''provide a structured way to study and compare peer reviewing workflows''' in the *ACL community. For this we propose to create a centralized repository of anonymized numerical data from ACL conferences. This repository can serve as an anchor point for future conference organizers and provide a bridge to the meta-science and science-of-science communities that have been working on similar issues in other disciplines and research fields. | We propose two initiatives aiming at facilitating ethical, consent-driven research in NLP for peer reviewing, and peer reviewing for NLP. The first initiative '''concerns the collection of peer review and manuscript data from *ACL events'''. We cover the major challenges in this process and propose solutions, as well as report on a pilot study conducted at COLING-2020, where some of those solutions were implemented. The second initiative aims to '''provide a structured way to study and compare peer reviewing workflows''' in the *ACL community. For this we propose to create a centralized repository of anonymized numerical data from ACL conferences. This repository can serve as an anchor point for future conference organizers and provide a bridge to the meta-science and science-of-science communities that have been working on similar issues in other disciplines and research fields. | ||
| − | === | + | === Documents === |
| − | # The slide deck briefly summarizes the two proposals; | + | # The [[:File:1._Review_collection_at_ACL_(slides).pdf| slide deck]] briefly summarizes the two proposals; |
| − | # The full document elaborates on the proposals in further detail. In addition, we provide | + | # The [[:File: 2._Review_collection_at_ACL.pdf| full document]] elaborates on the proposals in further detail. In addition, we provide |
| − | # | + | # the example License agreements for [[:File: 3._ACL_License_Blind_Submission_v1.pdf| peer reviewing data]] |
| − | # and blind submission data | + | # and [[:File: 3._ACL_License_Blind_Submission_v1.pdf| blind submission data]]. |
=== Contact === | === Contact === | ||
Revision as of 02:47, 30 March 2021
This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL. The documents has been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).
Summary
We propose two initiatives aiming at facilitating ethical, consent-driven research in NLP for peer reviewing, and peer reviewing for NLP. The first initiative concerns the collection of peer review and manuscript data from *ACL events. We cover the major challenges in this process and propose solutions, as well as report on a pilot study conducted at COLING-2020, where some of those solutions were implemented. The second initiative aims to provide a structured way to study and compare peer reviewing workflows in the *ACL community. For this we propose to create a centralized repository of anonymized numerical data from ACL conferences. This repository can serve as an anchor point for future conference organizers and provide a bridge to the meta-science and science-of-science communities that have been working on similar issues in other disciplines and research fields.
Documents
- The slide deck briefly summarizes the two proposals;
- The full document elaborates on the proposals in further detail. In addition, we provide
- the example License agreements for peer reviewing data
- and blind submission data.
Contact
If you have commentaries, questions or suggestions regarding the proposal, do not hesitate to contact:
Prof. Dr. Iryna Gurevych, gurevych@ukp.informatik.tu-darmstadt.de
Ilia Kuznetsov, kuznetsov@ukp.informatik.tu-darmstadt.de