Notes on the development of ACL Conference Awards Policy

Committee that developed this policy:

• Bonnie Webber
• Christy Doran
• Huang Xuanjing
• Joel Tetreault
• Jonathan Kummerfeld [chair]
• Yusuke Miyao [chair]

Edits were made by Jonathan Kummerfeld based on discussion with the ACL Executive. Feedback was solicited from the community and informed further improvements.

Anonymity

We discussed requiring authors to be anonymous to avoid potential a source of bias, but realised it was (1) not easy to do with camera-ready versions of papers, and (2) would be difficult to maintain while also handling conflict-of-interests on the best paper committee (COIs will be more likely than in the past because more papers are being considered). Given those concerns, we have made it optional.

Choice of terms

The committee had mixed opinions on the term 'best'. On one hand, it is common/standard and expected, on the other hand, it implies a metric with a clear ranking of papers, which we do not have. After discussing a range of alternatives we returned to 'best' because it is such a well established term. Several options for the second category were discussed, including 'noteworthy', 'honourable mention', and 'outstanding'. The final proposal uses 'outstanding' as it is a recognisable term.

Choice of numbers

• "Typically no more than 6" best papers. This value was chosen based on what we thought could be included in a single plenary session. However, the language is intentionally flexible.
• "1.5–2.5% of accepted papers" receive the outstanding paper award. This range was chosen to balance prestige (which favors a lower number) and the goal to recognize more work (which favors a higher number).
• For context, the graph below shows historical trends in awards at ACL going back to the very first year a best paper award was announced (note, these values include all award types together in the count for each year):

Several ideas came up that are worth considering in the future:

• Dissertation Awards. These exist in other communities (e.g. AAAI) and are intended to recognise a body of work. We did not include them as they are outside of the conference awards process. They would need a separate process that could, for example, be conducted by each organisation / society separately. We also discussed awards for thesis proposals, which would be suitable for the Student Research Workshop.
• Journal Paper Awards. Other communities have these (e.g. ISCA for papers in CSL). We discussed including them here by saying CL and TACL papers presented at a conference are eligible for awards, but then the awards committee needs to compare quite different papers (e.g. a 30+ page journal article and a 4 page short conference paper). The approach used elsewhere is that each journal chooses a 'best paper', but that is then outside the scope of this proposal regarding conferences.
• Requirements for papers to be well-presented. We discussed the idea that papers that receive awards should be examples of well-presented ideas that students can look to for understanding how to write a good paper. On the other hand, it would seem unfair to not reward an innovative idea just because the author is inexperienced and did not present it perfectly. We tried to come up with language to balance these factors, but decided there was no simple solution, so this requirement was dropped.
• Advice/guidance for reviewers on when to select a paper for an award. This is a good idea that is outside the scope of this policy.