Short-Term Reform Proposals for ACL Reviewing

The rapid growth of submissions and the increasing popularity of preprints have caused four problems to the current ACL reviewing system:

Review is not blind to all papers. Any time a paper is posted on arXiv, it is not blind. The best reviewer for a paper would be familiar with all public work that is relevant to the paper being reviewed, which includes papers that were published on arXiv. When some papers are blind and others are not, our review process is biased and unfair.

Turnaround time is too long. A large factor in the incentive to go to arXiv is that turnaround time from when a paper is finished to when it can be made public can be many months, especially when you get several random rejects. Reducing this turnaround time is key to removing the incentive to go to arXiv before review. If turnaround time is fast enough, we can feel better about banning arXiv before review without worrying that it will drive people away to other venues.

Review quality is too low. Good, experienced reviewers do not have a strong external incentive to do a good job; we are relying on their internal incentives only. New reviewers do not have good mechanisms for getting trained.

Not enough space for all good papers. Program chairs have to reject good papers in order to meet acceptance rates, which are required by some in our community for various reasons. This is unfortunate by itself, and when combined with low review quality and randomness due to just three (perhaps poorly assigned) reviewers, it leads to a large number of resubmissions, dramatically increasing review load.

To address these problems, the ACL Committee on Reviewing has been working on two proposals for reforming the reviewing system of ACL-related conferences: short-term and long-term. This document presents the short term proposals. It consists of four complementary actions that can be realistically implemented to improve the ACL review process in the near future (while the committee continues to investigate changes that require a longer lead time). These actions jointly address the four problems identified below. They are likely to be accepted by the majority of NLP researchers and they involve keeping existing ACL policies in place, including those regarding submission, review and citation.