2017 ACL 2017:AC 2nd update for discussion period

Sent to Track Chairs.

Subject: [ACL 2017] Discussion Period going into Author Response

Dear [reviewerFirstName] [reviewerLastName]:

Great work so far on getting most of the reviews in for your area! We are pleased with the progress, with only minor and small operational problems that have mostly been taken care of.

With the initial reviews in, we're now in the longer discussion window.  If you recall, this was a key changed and we decided to make in the reviewing schedule.  The idea is that the initial reviews would spark discussion among the assigned reviewers, and help to bring a constructive dialogue that would clarify reviews before going out to the authors for their response.

We would like to underscore the importance of making the review process more of a dialogue. We hope that you can try to spark discussion among your reviewers for the large bulk of borderline papers.  Whatever work we can ask reviewers to do up front before author response in cleaning up the reviews' direction and points of critique will make your job easier after author response.

You should remind reviewers to read each others reviews and to use Softconf's message board constructively to clarify any questions or doubts that could be answered among the reviewing parties.  In particular, you can refer to the advice that we compiled from various previously nominated best reviewers, which is on the PC Chair's blog (e.g., be polite, specific in your constructive criticism and how it can be addressed by the authors' response, and open-minded about what constitutes a worthy contribution).  To help you with this, we've included a draft email that you can circulate using Softconf's mail tool to email to reviewers as you did to remind reviewers of the review deadline in the last round (find the "ACL 2017 (Ready!) - Tracks - Discussion Period" template), feel free to modify it as you see appropriate.

To start off discussions, you may want to read the reviews themselves for any non- clear-cut cases (e.g., anything with averages 2--4, and even possibly other outliers, where some reviewer had low confidence scores).  For such papers, you can kickstart the dialogue with the reviewers, when feel it's appropriate, by summarizing what the feedback needs to be clarified.

In addition to this goal for the next weeks leading up to the author response period (starting March 13) response, there are a few issues that have cropped up that may also be applicable to your areas. Please take note.

1. Lost reviewers - if you have some reviewers that didn't bid, submit their TPMS profile, and have yet to turn in any reviews, you may have a problematic reviewer.  This can happened for all sorts of (valid) reasons, and your area needs to catch these problems before they get out of hand.  Fortunately, we have our longer discussion period, where you can try to find emergency reviewers to cover the shortfall of non-performing reviewers.

2. COIs - we have some reports of conflicts of interests from both area chairs as well as individual reviewers.  If you have a potential COI that needs to be resolved outside of your areas, please surface this to us immediately.

3. Preprints and double submissions - related to above are violations that impinge on the double-blind policy for reviewing.  Any pre-printing or parallel submission of a highly similar paper by any subset of the authors is considered a submission violation.  This includes submissions to preprint servers or other conferences after the ACL deadline.  A case in point: This week we worked with the ICML program co-chairs to identify parallel submissions and to disqualify them.  If you or your reviewers find similar cases, e.g., arXiv, IJCAI submissions, please alert us and we will judge the case accordingly.

4. Short paper reviews - For your information, the short review paper form erroneously had a field which asked whether "the paper had enough ideas for a *long* paper". This should have read "enough ideas for a *short* paper", and has been corrected in the form that reviewers now see.  

5. Review Dataset - the review form also has a final item to asked whether the review of would like their review included in an ACL review dataset.  Just to be sure about this, those reviews *will not* have the reviewer's name attached to it.  This experiment was conducted before by NAACL, where they had an option for reviewers to retain their names on the submitted reviews; however, we felt that this could be a decision that reviewers might later regret, so we have scrubbed the option of having attributed reviews.

6. Authors can respond directly to ACs, bypassing reviewers.  We need to let our reviewers know that the authors will have the opportunity to talk directly with you when they feel that the reviewers have to misjudged the paper. This is reflected in the draft email that you can send out to the reviewers during this discussion period.

Finally, we are collating your feedback on the process so far, for feedback to ourselves, Softconf, and the ACL Exec and conference officer (e.g., whether you'd recommend the usage of TPMS).  Please send your comments, brickbats our way.  

What happens next?
- Author response.  Authors will have approximately 3 days to respond to comments starting March 13. 
- To rank your selections for the area, following the author response period, by carefully examining the reviews (please do not react justify average review scores since the quality of reviews is often varied).  Each area will need to generate a ranking of both short and long papers.  Papers with AC COIs will need to be handled carefully. We will be communicating with you again soon about this.

Regina and Min