Difference between revisions of "Review Data Collection at *ACL"
(Created page with "''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.'' ''The documents has been prepared by Iryna Gurevych and Ilia...") |
|||
| (6 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.'' | ''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.'' | ||
| − | ''The documents | + | ''The documents have been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).'' |
| + | === Updates === | ||
| + | * March 2022: [[Media:Review_Data_Collection_ 2022_ Report_Proposal.pdf|Progress report and amendments]] to the original proposal in terms of data collection, access and storage. | ||
=== Summary === | === Summary === | ||
We propose two initiatives aiming at facilitating ethical, consent-driven research in NLP for peer reviewing, and peer reviewing for NLP. The first initiative '''concerns the collection of peer review and manuscript data from *ACL events'''. We cover the major challenges in this process and propose solutions, as well as report on a pilot study conducted at COLING-2020, where some of those solutions were implemented. The second initiative aims to '''provide a structured way to study and compare peer reviewing workflows''' in the *ACL community. For this we propose to create a centralized repository of anonymized numerical data from ACL conferences. This repository can serve as an anchor point for future conference organizers and provide a bridge to the meta-science and science-of-science communities that have been working on similar issues in other disciplines and research fields. | We propose two initiatives aiming at facilitating ethical, consent-driven research in NLP for peer reviewing, and peer reviewing for NLP. The first initiative '''concerns the collection of peer review and manuscript data from *ACL events'''. We cover the major challenges in this process and propose solutions, as well as report on a pilot study conducted at COLING-2020, where some of those solutions were implemented. The second initiative aims to '''provide a structured way to study and compare peer reviewing workflows''' in the *ACL community. For this we propose to create a centralized repository of anonymized numerical data from ACL conferences. This repository can serve as an anchor point for future conference organizers and provide a bridge to the meta-science and science-of-science communities that have been working on similar issues in other disciplines and research fields. | ||
| − | === | + | === Documents === |
| − | # The slide deck briefly summarizes the two proposals; | + | # The [[Media:1._Review_collection_at_ACL_(slides).pdf|slide deck]] briefly summarizes the two proposals; |
| − | # The full document elaborates on the proposals in further detail. In addition, we provide | + | # The [[Media:2._Review_collection_at_ACL.pdf|full document]] elaborates on the proposals in further detail. In addition, we provide |
| − | # | + | # the example License agreements for [[Media:3._ACL_License_Blind_Submission_v1.pdf|blind submission data]] |
| − | # and | + | # and [[Media:4._ACL_License_Peer_Reviews_v2.pdf|peer reviewing data]]. |
=== Contact === | === Contact === | ||
| Line 16: | Line 18: | ||
Prof. Dr. Iryna Gurevych, [mailto:gurevych@ukp.informatik.tu-darmstadt.de gurevych@ukp.informatik.tu-darmstadt.de] <br /> | Prof. Dr. Iryna Gurevych, [mailto:gurevych@ukp.informatik.tu-darmstadt.de gurevych@ukp.informatik.tu-darmstadt.de] <br /> | ||
| − | Ilia Kuznetsov, [mailto:kuznetsov@ukp.informatik.tu-darmstadt.de kuznetsov@ukp.informatik.tu-darmstadt.de] | + | Dr. Ilia Kuznetsov, PhD [mailto:kuznetsov@ukp.informatik.tu-darmstadt.de kuznetsov@ukp.informatik.tu-darmstadt.de] |
Latest revision as of 04:09, 25 August 2022
This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL. The documents have been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).
Updates
- March 2022: Progress report and amendments to the original proposal in terms of data collection, access and storage.
Summary
We propose two initiatives aiming at facilitating ethical, consent-driven research in NLP for peer reviewing, and peer reviewing for NLP. The first initiative concerns the collection of peer review and manuscript data from *ACL events. We cover the major challenges in this process and propose solutions, as well as report on a pilot study conducted at COLING-2020, where some of those solutions were implemented. The second initiative aims to provide a structured way to study and compare peer reviewing workflows in the *ACL community. For this we propose to create a centralized repository of anonymized numerical data from ACL conferences. This repository can serve as an anchor point for future conference organizers and provide a bridge to the meta-science and science-of-science communities that have been working on similar issues in other disciplines and research fields.
Documents
- The slide deck briefly summarizes the two proposals;
- The full document elaborates on the proposals in further detail. In addition, we provide
- the example License agreements for blind submission data
- and peer reviewing data.
Contact
If you have commentaries, questions or suggestions regarding the proposal, do not hesitate to contact:
Prof. Dr. Iryna Gurevych, gurevych@ukp.informatik.tu-darmstadt.de
Dr. Ilia Kuznetsov, PhD kuznetsov@ukp.informatik.tu-darmstadt.de