2017 ACL 2017:ACs before assignment, 2

Sent to Track Chairs.

Subject: Assignment and Reviewing stages, on your own (follow up)

Dear [reviewerFirstName] [reviewerLastName],

Thanks to all of you for your help thus far.  We received a few requests for clarification for certain areas, so we're sending you this follow up.

1 - All areas have been monitoring the bidding exercise, and that's great.  We caught a few problems thanks to your help!  If you haven't done already, send a reminder today, it typically has a very positive effect. Please, coordinate with your co-ACs. It's ready: go to your "Long" console, choose "Send Mail to Track Members", select the template "Tracks - Bidding Reminder", edit it appropriately and send it. 

2 - Papers authored by ACs. At reassignment time, we the PC chairs decided to place the papers authored by ACs to the Subject Area indicated by the authors in most cases. We did this because the match between reviewers and paper topic is likely to be better, and it is the way all papers are treated. However, it is critical to maintain confidentiality within your area. START will hide any information related to reviewing from the author AC. Please do not share listings of papers outside START, and be careful to share sensitive information among ACs.

3 - Conflicts-of-Interest (COI). Please identify COI between you and papers in your area, and also between reviewers and papers in your area. In essence, a program committee member has a COI with a paper if: (a) he is author of the paper; (b) he has had collaborations with an author of the paper in the last 5 years; (c) he works in the same institution as an author of the paper. For more context, please read the ACL policy on COI: http://aclweb.org/adminwiki/index.php?title=Conflict-of-interest_policy. If you identify COIs, enter them in the Manage Conflicts of Interest section of your track consoles. A reviewer who has COI with a paper will not be assigned the paper. An AC who has COI with a paper will not see the paper in START (and in particular will not see any sensitive information). If unsure about COIs, discuss with us or with your co-ACs.  If you find that you cannot enter the COI, please directly email this to us, as we will need to enter it in the system. Give us the paper ID and title.

How do you see the author affiliations or other detailed submission info in START? In the Track Management Console, go to  "Spreadsheet Maker", click on "Submissions" tab, and you can generate an Excel file containing detailed submission info for papers in your track. This may be useful in helping you identify COI between reviewers and authors.

4 - How we will deal with COIs? Eventually we will ask each area to produce a ranking of papers (for long and short ones). In each area, initially all ACs will meet to produce one ranking of all papers for which no AC has COI. The rest of papers will be inserted to the ranking by the ACs who are clean of COIs. When the time comes we will send instructions to keep everything confidential. For now, keep in mind that your view of your area is different from the views of your co-ACs, do not share sensitive information (for a paper, sensitive information includes bids, reviewer assignments, reviews, and post-review discussion).

5 - Reduced loads. Some reviewers are asking for reduced loads of long/short papers.  This info is in our shared Google Spreadsheet and also tracked in START. You may receive additional reduced load requests by email, and for those new requests, please update them yourself (best in both sources of info, to be consistent). In the Track Management Console, go to "Assign Reviewer Duties"->"Bid/Quota Adjustments". You can do this anytime between now and the start of reviewer-to-paper assignment. Please remember to do this separately for short and long paper tracks.

6 - Once bidding is over you will do the assignments. We will strongly encourage you to start with manual assignments of reviewers to papers (i.e., not to run the automatic assignment right away). In the Track Management Console > Assign Reviewer Duties you can see a global view of papers, reviewers and bids in the Check-box Spreadsheet. Please get familiar with review assignment tools. Do prepare for the reviewer-to-paper assignments! This is one of the most important parts of the review process. We really depend on your judgment here.  Ideally, each paper gets a well-rounded team of reviewers that balances different expertise, perspectives, and experience levels. Please get familiar with the submissions, reviewer backgrounds, as well as the START and TPMS tools for reviewer assignment (see separate email).

When you're ready to go (we see that some of the smaller areas are already ready), please do announce the assignments to your reviewers.  Use "Start the Review Process" to select the "ACL 2017 (Ready!) - Tracks - Start the Review Process Template email".  We're in the process of composing this email and it will have the "Ready" keyword when its completed -- we are modifying it to add more details about the review process schedule.  Again, feel free to customize it as you see fit before sending it.  N.B., the special variables for trackChairName and trackFullName may not work as nicely as you'd like, so feel free to sub out those variables for your actual text.

7 - Large areas. In large areas, you might want to partition the papers among ACs, such that each paper is chaired by one AC. You might use the bidding and assignment functions of START to manage this. Using external resources (e.g., Google docs) is fine as long as you do not share sensitive information with your co-ACs.

- Regina and Min

--
ACL 2017 Long - https://www.softconf.com/acl2017/papers
ACL 2017 Short - https://www.softconf.com/acl2017/shortpapers
Shared Spreadsheet - <shortlink/>