2014Q3 Reports: Reviewing Software

I have been developing a new system for managing the reviewer pool for ACL affiliated conferences. The primary purpose of this system is to 1) facilitate reviewer signup for area chairs and reviewers and 2) efficiently allocate the reviewer pool to areas based on area need and reviewer preference. In this system, area chairs email reviewers asking them to volunteer. A reviewer them completes an online form in which they can indicate which areas they would be willing to review for. After papers are submitted, we calculate how many reviewers are needed by area, and then assign the appropriate number of reviewers factoring in the requests of the area chairs and the reviewer preferences. Once reviewers are assigned to area, bidding proceeds as usual. The process can be repeated for short papers, though we usually reuse the area assignments from long papers.

Since the initial trial of the system in 2012, we have used it for 4-5 conferences, including ACL, NAACL and EMNLP. For some conferences where we received an unexpected surge of papers in a few areas, the system allowed us to respond without a last minute attempt to recruit more reviewers or increase the load on a particular area. The process has been refined each time based on feedback from the reviewers, area chairs and program chairs. We have reached a stable version and wrote up detailed instructions for running the process. For both ACL 2014 and EMNLP 2014, a graduate student has been in charge of running the process based on the provided code and instructions. We believe the process is now at the point where any conference can use the available software to run the process themselves.

There has been some discussion of the next steps for the process. One idea would be to integrate the system directly into the START review platform. This would make managing the process easier, but it would require cooperation of the START system, and reduce flexibility for further refinements in future years.

Going forward, each conference's program chairs should be made aware of this system and be encouraged to use it.

(by Mark Dredze)