ACL Rolling Review Proposal

By the ACL Reviewing Committee (June 12, 2020)

The ACL reviewing committee has been working on improvements to the reviewing process, has recently introduced short term changes [add link here]. What follows below is a proposal for a more long-term transition to a new system for review in *ACL conferences, where reviewing and acceptance of papers to publication venues is done in a two-step process:

• Step 1 -- Centralized Rolling Review: Authors submit papers to a unified review pool with monthly deadlines (similarly to TACL). Review is handled by an action editor, and revision and resubmission of papers is allowed.

• Step 2 -- Submission to Publication Venue: When a conference/workshop/journal submission opportunity comes around, authors may submit already reviewed papers to the conference, workshop, or journal for publication. Program chairs (possibly with the help of senior area chairs) will then accept a subset of submitted papers for presentation.

These two steps already exist in current reviewing processes: reviewers and ACs write reviews and meta reviews, which then get sent to SACs and program chairs for final acceptance decisions. The main difference in this proposal is that the reviews and meta reviews are done in a centralized, rolling process, which provides numerous benefits, as described below.

Potential Advantages/Concerns

Potential positive points of this new system are highlighted in blue, concerns are highlighted in red (many of these are covered in detail in the following sections).

Details of Step 1 -- Centralized Review

The centralized review process will consist of submission to a centralized review system that has a large pool of potential “reviewers” (2000-4000). Among these reviewers, some percentage (20%?) who are particularly experienced or senior may also serve as “action editors”, who guide the review process and write meta-reviews for individual papers. There will also be a small number of “editors in chief” who oversee the entire process and handle any special cases (about 10).

Below is an outline of what the first-pass review process may look like on a step-by-step basis. The number of days is an idealized estimate for when the process goes smoothly, and will need to be adjusted on a case-by-case basis.

• Submission: Authors submit a paper to the unified review pool by a fixed deadline (e.g., the first of every month).
• Action Editor Assignment (0 days): A system automatically assesses the paper content, and automatically assigns an action editor from the pool of senior editors. This assignment is based on several criteria: (a) automatically determined content match between the paper content and the action editors’ past publications, (b) lack of COIs, and (c) any requests for a reduced/increased load entered into the system by the editor, and (d) review balance, where an action editor who has recently handled many papers will have a positive balance and thus not be assigned as often, and vice-versa for negative balance.
• Action Editor Confirmation (days 1-4): The action editor will confirm their ability to handle the paper and lack of COIs ASAP. In the case they have COIs or are not capable of handling the paper they will be in charge of selecting another serving action editor among several suggestions provided by the automatic system. If they do not feel that any of the action editors suggested by the system are adequately familiar with the topic, they can suggest an external action editor (who has not already reviewed for the reviewing pool), potentially also aided by a system. If there is no action editor in our community who can handle the paper at all, then the paper can be declined as out-of-scope (see “desk-rejects” below) after confirmation of this fact by an editor-in-chief.
• Desk Rejects and Reviewer Assignment (days 5-7): After the action editor has been confirmed, the action editor will quickly check the paper for any major violations in formatting or other factors and desk reject the paper if it is in violation. The action editor will then be presented with a list of reviewers automatically suggested from the pool by the system based on criteria (a-d) above. The action editor will choose reviewers, using this automatic suggestion as a base.
• Review (days 7-30): The reviewers conduct the normal review process.
• Editor Meta-review and Review Release (days 30-35): The action editor will read the reviews, discuss with the reviewers if necessary, and:
  • Summarize the review results in a meta-review.
  • Perform a compliance check, checking several boxes confirming “the paper satisfies the formatting requirements (e.g. abiding by the page limit),” “the paper is written in comprehensible English (or any language accepted by *ACL publications),” “the paper is compliant with the ACL ethical code,” “the paper is topically within the scope of at least one *ACL venue.” In the very rare case (e.g. 2-4% of submitted papers?) that a paper does not satisfy these minimal criteria, it will not be allowed to proceed to the next step and may face a moratorium on resubmission for a certain period of time.
• Next Steps: Assuming the paper passes the compliance check, the authors may then either:
  • Revise and request another round of review: If the authors are not yet satisfied with their review results, they may revise the paper to reflect reviewer comments, write an author response, and return to the beginning of Step 1. In this case, the paper will usually go to the same action editor and set of reviewers (with some exceptions, see detailed discussion below), and the previous reviews and the author response to them may be attached, as is currently done in TACL.
  • Submit to a publication venue: Proceed to Step 2 below, submitting the paper, with reviews and meta-reviews attached, to a conference/workshop/journal.