ACL Conference Awards Policy

This document lays out a standard protocol for awards at ACL conferences (Note that for conferences colocated with a non-ACL event there may need to be some adjustments depending on the policies of the other event. Also, this process was not developed for workshops or journals.). The goal of introducing the policy is to have a consistent approach that suits the scale of our conferences today. The policy is designed to highlight work that is interesting along a variety of axes in ways that encourage discussion. Recognizing valuable work in a consistent way also means the value is clearer (e.g., for hiring and tenure cases).

Award Types and Criteria

Best Paper Award

We define "Best" as work that is particularly fascinating, controversial, surprising, impressive, and/or potentially field-changing.

• "Best Paper Award": Typically no more than 6 papers receive this award. These papers are presented in a plenary session at the conference. PCs should consider scheduling this plenary session early in the conference, to encourage discussion of the papers during the conference.
• "Outstanding Paper Award": 1.5–2.5% of accepted papers, selected as part of the same process. This allows for broader recognition of work that meets the criteria.
• "Area Chair's Award": Up to one paper per track. These are selected by the SACs for each track.
• Short and long papers are considered together for these awards.

Test of Time Award

See the policy at Test-of-Time Papers Award.

Special Categories

Two special awards will recognise work with a particular focus as described below. Papers that receive an award in a special category are also eligible to receive a general award.

• Social Impact Award - For papers that have the potential for significant positive societal impact.
• Resource Award - For papers that announce, describe, and share a fascinating, valuable, or potentially field-changing new resource (e.g., a dataset or knowledge graph).

Program Committee Chairs may also choose to include awards for other categories. Examples of additional special categories include:

• Software Award - For papers that describe and share (ideally via source code) a fascinating, valuable, or potentially field-changing new piece of software. Note that this is not the same as the Demonstration Paper Award since there could be papers outside of that track that meet this criterion.
• Demonstration Paper Award - For papers in the Demonstration Track.
• Theme Paper Award - For papers on the conference’s theme.
• Linguistic Insight Award - For papers that make a particularly significant contribution to our understanding of language.
• Low-Resource Paper Award - For papers that contribute to work on languages or domains with limited data.
• Interdisciplinary Research Award - For papers that contribute to NLP and another field in new and interesting ways.
• Reproduction Award - For papers that reproduce prior work in a particularly enlightening way, revealing additional features of the prior work.

Selection Process

Nominations

In the review process, reviewers (Note: PCs may choose whether or not to include reviewers in the nomination process.), AEs, ACs, and SACs will be asked to answer the following questions:

• 1. "Could the camera-ready version of this paper merit consideration for an "outstanding paper" award (up to 2.5% of accepted papers will be recognized in this way)? Outstanding papers should be either fascinating, controversial, surprising, impressive, or potentially field-changing. Awards will be decided based on the camera-ready version of the paper."
  • "Yes"
  • "Maybe"
  • "No"
• 2. "If yes/maybe, please briefly describe why:"
  • (short answer box with no word or character limit)

The best paper committee should consider any paper that was marked "Yes" or "Maybe" by any reviewer (Note: PCs may choose whether or not consider papers that were labeled 'maybe' by a reviewer and not nominated by the AE/AC.), AE, AC, or SAC (Note: This requirement is included to clarify how this policy interacts with ACL Rolling Review.).

SACs may also choose a paper in their area to receive the “Area Chair’s Award”. This will encourage diversity in the papers that are highlighted. The Best Paper Committee will not be told which papers have received this award, to avoid biasing their choices.

Papers are not eligible for awards if their authors include Program Chairs. The PCs may also choose to specify that authors in other senior organisational roles are not eligible for awards.

Special category awards can either be handled in the same way, by editing the question above to include them, or through a separate process defined by the PCs.

Selection

(Note, this process was developed based on the assumption that the nomination process leads to 80-100 papers being considered for awards. If the number of nominated papers turns out to be much higher or lower then the selection process should be adapted as necessary.)

A best paper committee will be selected by the program chairs. The committee size should be large enough to keep the load to around 10-15 papers per member. The committee should be diverse in composition in terms of research areas and demographics.

The committee will follow a multistage process to determine the final awards. The committee will receive the final camera-ready version of the paper (or its anonymized version if possible), anonymized reviews, and associated supplementary materials.

1. Papers are divided between the committee members based on research areas for a first pass in which each paper is read by at least 2 committee members. They independently place the papers into three groups: (1) consider for best paper, (2) consider for outstanding paper, (3) do not consider further.
2. All committee members read the papers that were identified as under consideration for 'best paper'.
3. The committee meets to make the final selections.

This proposal does not define a specific rubric. However, while reading and discussing papers, committee members should consider:

• Is each paper either fascinating, controversial, surprising, impressive, or potentially field-changing? Note that papers do not need to demonstrate all of these properties; any property is sufficient.
• Does the paper present as its motivating use case an application with significant negative social impact? Even if the motivating use case is an application with neutral or positive social impact, are there obvious applications with significant negative social impact which are left unaddressed or insufficiently addressed by the paper? (In such cases, the paper should *not* be an award candidate.)
• Is the work presented in the paper reproducible? For example, is there sufficient information in the paper to repeat the experiments? If not, is the lack of reproducibility justified in the paper?
• Do the awards highlight a broad range of research types and strengths?
• Do the awards include types of research that can be conducted at small labs?
• Do the awards include papers that show excellence in potential positive social impact?

Special awards may be selected by the best paper committee or by separate committees.

Announcement, Certificates, and Financial Awards

All awardees will be announced on the conference website or blog before the conference. All awardees will receive a certificate with the name of the conference, name of the award, title of the paper, and author names, signed by the Program Chair(s) and/or General Chair(s). Conferences may choose to have a financial component for any of the awards, which may be sponsored, and may be split between the authors as the authors choose.

Documentation

The ACL Anthology will have an indication of which papers received awards (as it does now). Eventually, a page will be created that lists all papers that have received awards.

The ACL will track the research area and author demographics (gender and country of affiliation) of papers that are (a) nominated, and (b) receive awards. This information will be used to inform efforts to avoid bias and inequality in the selection process. Care should be taken to track them appropriately to avoid harm (e.g., by incorrectly assuming a demographic attribute). The ACL Equity Director will be responsible for maintaining these records.

Notes on the development of this proposal

Committee that developed this policy:

• Bonnie Webber
• Christy Doran
• Huang Xuanjing
• Joel Tetreault
• Jonathan Kummerfeld [chair]
• Yusuke Miyao [chair]

Edits were made by Jonathan Kummerfeld based on discussion with the ACL Executive. Feedback was solicited from the community and informed further improvements.

Anonymity

We discussed requiring authors to be anonymous to avoid potential a source of bias, but realised it was (1) not easy to do with camera-ready versions of papers, and (2) would be difficult to maintain while also handling conflict-of-interests on the best paper committee (COIs will be more likely than in the past because more papers are being considered). Given those concerns, we have made it optional.

Choice of terms

The committee had mixed opinions on the term 'best'. On one hand, it is common/standard and expected, on the other hand, it implies a metric with a clear ranking of papers, which we do not have. After discussing a range of alternatives we returned to 'best' because it is such a well established term. Several options for the second category were discussed, including 'noteworthy', 'honourable mention', and 'outstanding'. The final proposal uses 'outstanding' as it is a recognisable term.

Choice of numbers

• "Typically no more than 6" best papers. This value was chosen based on what we thought could be included in a single plenary session. However, the language is intentionally flexible.
• "1.5–2.5% of accepted papers" receive the outstanding paper award. This range was chosen to balance prestige (which favors a lower number) and the goal to recognize more work (which favors a higher number).
• For context, the graph below shows historical trends in awards at ACL going back to the very first year a best paper award was announced (note, these values include all award types together in the count for each year):

Several ideas came up that are worth considering in the future:

• Dissertation Awards. These exist in other communities (e.g. AAAI) and are intended to recognise a body of work. We did not include them as they are outside of the conference awards process. They would need a separate process that could, for example, be conducted by each organisation / society separately. We also discussed awards for thesis proposals, which would be suitable for the Student Research Workshop.
• Journal Paper Awards. Other communities have these (e.g. ISCA for papers in CSL). We discussed including them here by saying CL and TACL papers presented at a conference are eligible for awards, but then the awards committee needs to compare quite different papers (e.g. a 30+ page journal article and a 4 page short conference paper). The approach used elsewhere is that each journal chooses a 'best paper', but that is then outside the scope of this proposal regarding conferences.
• Requirements for papers to be well-presented. We discussed the idea that papers that receive awards should be examples of well-presented ideas that students can look to for understanding how to write a good paper. On the other hand, it would seem unfair to not reward an innovative idea just because the author is inexperienced and did not present it perfectly. We tried to come up with language to balance these factors, but decided there was no simple solution, so this requirement was dropped.
• Advice/guidance for reviewers on when to select a paper for an award. This is a good idea that is outside the scope of this policy.