In the past years, peer review at *ACL venues has undergone numerous experiments with peer review forms, process, policies, tutorials, and incentives. Ideally, the results of all these experiments would be used for the benefit of the community, ensuring that any future policies are evidence-based and address real problems with peer review that we have. However, in practice the people serving in editorial roles are so overwhelmed, that even in-depth reports from one event are extremely rare. Furthermore, while ACL has the right to pull the internal data from different venues to improve its organization, there is currently no process or precedent for actually doing so.
We propose to establish the ACL Peer Review Committee: a working group dedicated to the development of peer review policies across ACL venues. This group would consider and help to develop proposals relevant to the peer review process, which originate either from ACL venues or the community. It would also monitor the implementation of any proposals, and ensure that the venues implementing them would consistently report on the results of any changes.
1. Purpose of the committee
This committee will have a broad mandate to analyze internal peer review data from ACL venues for the purposes of developing evidence-based policies and improving the peer review process (but not for independent research by the committee members). For the sake of transparency, any results of such analysis will be made available as public reports.
This committee will be composed to represent ACL venues (conferences and journals), as well as ACL members from its three geographical chapters. It will also consult other relevant ACL committees and institutional bodies of ACL on a case-by-case basis.
2. Proposed Initial composition of the committee
This committee will include two people representing *CL conferences and ARR (point (1) below), two people from the existing review venue types (point (2) below), at least three consulting members from the broader community to represent three geographical Chapters of ACL (point (3) below), two representatives of the ACL exec (points (4) and (5) below).
3. Committee composition (=9 people):
(1) Two former and current program co-chairs, or ARR EiC
(2) One person each from members of the existing reviewing venue types in the *CL community.
(3) One member from the broader *CL community from each of the three ACL chapters. These committee members should have dedicated expertise in peer-review, acquired either through service as PCs, EiC or through research activities.
(4) a representative of the ACL exec presidential chain
(5) the ACL conference officer, also in the ARR Advisory Board
The committee should work in an inclusive manner, flexibly accommodating additional advising members whose expertise is important in specific tasks.
The committee will elect two (co-)chairs who will serve regular terms on the committee and will be replaced by new (co-)chairs after their term is over.
The co-chairs may not represent the same region (as defined by the ACL chapters).
4. Term of service
The co-chairs representing ACL bodies will serve for 2 years, with staggered terms. (In the initial committee, 50% of the members would ideally commit to serve for 3 years to accommodate the knowledge transfer to the second batch in 2 years.) The consulting members would rotate similarly every two years, so as to work with the co-chairs on the current issues while representing the relevant stakeholder group(s). E.g. policies related to handling of non-anonymous reviews could benefit from input from the underrepresented minorities, who could be the most affected by them.
5. Current committee composition as of 1.08.2024
* Roy Schwartz (co-chair)
* Thamar Solorio (co-chair)
* Anna Rogers (designated co-chair)
* Steven Bethard
* Roi Reichart
* Yuki Arase
* Vivian Chen
* Barbara Plank
* Noah Smith