ACL Rolling Review Proposal

By the ACL Reviewing Committee (June 12, 2020)

The ACL reviewing committee has been working on improvements to the reviewing process, has recently introduced short term changes [add link here]. What follows below is a proposal for a more long-term transition to a new system for review in *ACL conferences, where reviewing and acceptance of papers to publication venues is done in a two-step process:

• Step 1 -- Centralized Rolling Review: Authors submit papers to a unified review pool with monthly deadlines (similarly to TACL). Review is handled by an action editor, and revision and resubmission of papers is allowed.

• Step 2 -- Submission to Publication Venue: When a conference/workshop/journal submission opportunity comes around, authors may submit already reviewed papers to the conference, workshop, or journal for publication. Program chairs (possibly with the help of senior area chairs) will then accept a subset of submitted papers for presentation.

These two steps already exist in current reviewing processes: reviewers and ACs write reviews and meta reviews, which then get sent to SACs and program chairs for final acceptance decisions. The main difference in this proposal is that the reviews and meta reviews are done in a centralized, rolling process, which provides numerous benefits, as described below.

Potential Advantages/Concerns

Potential positive points of this new system are highlighted in blue, concerns are highlighted in red (many of these are covered in detail in the following sections).

Details of Step 1 -- Centralized Review

The centralized review process will consist of submission to a centralized review system that has a large pool of potential “reviewers” (2000-4000). Among these reviewers, some percentage (20%?) who are particularly experienced or senior may also serve as “action editors”, who guide the review process and write meta-reviews for individual papers. There will also be a small number of “editors in chief” who oversee the entire process and handle any special cases (about 10).

Below is an outline of what the first-pass review process may look like on a step-by-step basis. The number of days is an idealized estimate for when the process goes smoothly, and will need to be adjusted on a case-by-case basis.

• Submission: Authors submit a paper to the unified review pool by a fixed deadline (e.g., the first of every month).
• Action Editor Assignment (0 days): A system automatically assesses the paper content, and automatically assigns an action editor from the pool of senior editors. This assignment is based on several criteria: (a) automatically determined content match between the paper content and the action editors’ past publications, (b) lack of COIs, and (c) any requests for a reduced/increased load entered into the system by the editor, and (d) review balance, where an action editor who has recently handled many papers will have a positive balance and thus not be assigned as often, and vice-versa for negative balance.
• Action Editor Confirmation (days 1-4): The action editor will confirm their ability to handle the paper and lack of COIs ASAP. In the case they have COIs or are not capable of handling the paper they will be in charge of selecting another serving action editor among several suggestions provided by the automatic system. If they do not feel that any of the action editors suggested by the system are adequately familiar with the topic, they can suggest an external action editor (who has not already reviewed for the reviewing pool), potentially also aided by a system. If there is no action editor in our community who can handle the paper at all, then the paper can be declined as out-of-scope (see “desk-rejects” below) after confirmation of this fact by an editor-in-chief.
• Desk Rejects and Reviewer Assignment (days 5-7): After the action editor has been confirmed, the action editor will quickly check the paper for any major violations in formatting or other factors and desk reject the paper if it is in violation. The action editor will then be presented with a list of reviewers automatically suggested from the pool by the system based on criteria (a-d) above. The action editor will choose reviewers, using this automatic suggestion as a base.
• Review (days 7-30): The reviewers conduct the normal review process.
• Editor Meta-review and Review Release (days 30-35): The action editor will read the reviews, discuss with the reviewers if necessary, and:
  • Summarize the review results in a meta-review.
  • Perform a compliance check, checking several boxes confirming “the paper satisfies the formatting requirements (e.g. abiding by the page limit),” “the paper is written in comprehensible English (or any language accepted by *ACL publications),” “the paper is compliant with the ACL ethical code,” “the paper is topically within the scope of at least one *ACL venue.” In the very rare case (e.g. 2-4% of submitted papers?) that a paper does not satisfy these minimal criteria, it will not be allowed to proceed to the next step and may face a moratorium on resubmission for a certain period of time.
• Next Steps: Assuming the paper passes the compliance check, the authors may then either:
  • Revise and request another round of review: If the authors are not yet satisfied with their review results, they may revise the paper to reflect reviewer comments, write an author response, and return to the beginning of Step 1. In this case, the paper will usually go to the same action editor and set of reviewers (with some exceptions, see detailed discussion below), and the previous reviews and the author response to them may be attached, as is currently done in TACL.
  • Submit to a publication venue: Proceed to Step 2 below, submitting the paper, with reviews and meta-reviews attached, to a conference/workshop/journal.

Notably, in the ideal case, this allows for a decision within a little bit over a month. Based on this, if the authors want to revise their paper, they have a little less than a month to do so before the next deadline for minor revisions, and can spend some more time if they so choose (up to some reasonable time limit, e.g. 9 months, to prevent the article from losing relevance and the old reviews from becoming out-of-date).

Details of Step 2 -- Submission to Publication Venue

The steps for Step 2, submission to a publication venue, are as follows:

• Call for Submissions (-2 months or so): The publication venue would notify the community that they will be accepting submissions, and that papers that have been reviewed (and satisfy the compliance criteria) can be submitted at this time.
• Submission Deadline (0 days): Authors will submit reviewed papers to the venue if they wish their paper to be considered for publication there.
• Selection Process (2-6 weeks): The program chairs, possibly aided by area chairs, will sort through the already-reviewed submissions and build the program for the conference. Similarly to how program chairs do now, they will select the papers to be presented based on a combination of numerical scores, reviews and meta-reviews by the action editor, optional discussion between the PCs and the action editor, and consideration for diversity/direction of the program.
• Post-Selection: Selected papers will be presented in the conference/workshop/journal. Authors of papers that are not selected can either submit them to the next publication opportunity, or can revise and resubmit the paper in an attempt to improve the reviews.

Frequently Asked Questions/Concerns

Handling of Blind Review

During step 1, the centralized review process, papers will be blind according to the ACL review policy (for simplicity, we are not considering any changes to the blind review policy in this proposal).

During step 2, consideration for presentation at a publication venue, the process could either be blind or not blind. Currently, in many conferences the authors are not blind to program chairs, and either blind or not blind to area chairs, but this policy could be adjusted appropriately. One advantage of not requiring papers to be blind in step 2 is that this would free the authors to make their papers public after they are satisfied with the results of step 1 (i.e. they have good reviews and are likely to be accepted to a venue of the authors’ choice).

In addition, the *ACL may optionally consider establishing a centralized public blind (or after review, de-blinded) repository of papers submitted to the review process. This repository would be opt-in, so authors would be able to decide whether they want their papers to be made public during review. This would have the advantage of allowing authors who want their paper to be made available to do so in a way that still preserves blind review.

Handling of “Areas” and Ensuring Diversity in Scientific Content

There are a few concerns regarding how a monolithic reviewing process would make changes to the current system:

Areas: In conferences we currently have “areas” and “area chairs”, but in journals such as TACL (as well as other conferences such as ICLR) we just have “action editors” who handle each paper individually. How would this be resolved? Also, what are the selection criteria that program chairs use for selecting papers?

Diversity: An *ACL conference still has autonomy to put together its own program. But this autonomy is greatly reduced because papers can only be drawn from the common pool, which is produced by a monolithic reviewing system.

Answer: For the review process, we may not need “areas”, rather just make sure we assign a competent action editor and set of reviewers, which would be area-independent. In the second step of deciding which papers are presented at a particular conference, then giving the program chairs guidance, but also freedom in their criteria, may be the best way. For example, program chairs can choose to use areas if they wish, but could also choose some other mechanism of ensuring quality and diversity of the scientific program.