2024Q3 Reports: Ethic Committee Chairs

Overview of Ethics reviewing:

Overall, 391 papers were flagged for ethics review. 75 were identified as needing a full ethics review. About 10 papers received comments from the chairs based on minor issues identified at flagging time.

About 20 papers were desk rejected and/or withdrawn by the authors, waiving the need for ethics review. All other papers were reviewed by at least one ethics chair to determine the need for ethics reviewing.

Papers identified for ethics review were reviewed by one reviewer and one chair, to arrive at decisions: 1 no issue, 24 minor issues, 27 major issues, 4 reject.

• Among the four papers for which the ethics committee recommended rejection, one was committed to ACL and it was rejected.
• Among the 26 papers with serious issues, seven were committed to ACL. Four among these were rejected. one was accepted to findings and two to the main conference. The three accept decisions were based on author feedback and SAC recommendations, and we reiterated the ethics review in our feedback to the authors.
• Among the 24 papers with minor issues, 19 were committed to ACL. Seven were rejected, five were accepted to findings and seven to the main conference.

Accepted papers were reviewed to ensure ethics reviewing was taken into account in e.g., discussion.

Topics in Ethics reviewing:

Three themes were very salient in the papers reviewed by the ethics committee: corpus development (including corpus use agreements and annotation procedures), attacks on LLMs and adequacy between LLMs and applications.

Data use agreements should be carefully considered when building a corpus of using a corpus for experiments. Special consideration has to be given to the privacy implication of using Large Language Models or other NLP tools through APIs. For example, Reddit Terms and Conditions state "scraping the Services without Reddit’s prior written consent is prohibited"; the MIMIC data use agreement states "the PhysioNet Credentialed Data Use Agreement explicitly prohibits sharing access to the data with third parties, including sending it through APIs provided by companies like OpenAI, or using it in online platforms like ChatGPT." Furthermore, certain types of personal data are sensitive by nature (e.g., health or political information) and should not be used without a person's consent. Careful consideration is also needed when annotators are recruited to produced annotations for corpora. Such research should include reports of the annotators background (expertise and/or demographic when relevant) as well as compensation, which has been highlighted as a salient issue in crowd working set-ups (see, e.g., Fort et al, 2011).

Safety concerns regarding Large Language Models yield research into vulnerability of the models. We recommend that such research should be conducted according to norms established by the Ethical security research community to minimize harm (eg https://www.cs.columbia.edu/~angelos/Papers/2010/msp2010020067.pdf), such as disclosing intent of the research, seeking legal guidance before starting the research, being proactive about data protection, and notifying the companies in whose products they identify vulnerabilities *prior* to publication of the vulnerabilities, and giving the companies an opportunity to remediate the vulnerabilities (publication can proceed even if the vulnerabilities are remediated).

Large Language Models are used for a range of applications, including some that warrant caution and may require discussion within the community regarding their appropriateness. These include *predictive* applications where the outcome of system predictions, whether they are monitored by human experts or not, can have significant life impact on system users or the general public. Applications to sensitive fields such as health, law or education require special attention as do applications where predictions may affect protected demographic categories.

Recommendations for future conferences

Prior to review period:

• on-boarding of ethics chairs with information regarding tools and processes. Open Review is a complex system and having to manage ethics reviewing on the side is unexpected.
• defining the synergy/collaboration between conference ethics chars and ARR ethics chairs
• including ethics chairs in follow-up processes after the acceptance phase (e.g. notifications of paper acceptance status)

Major issue:

• anticipating review volume -> reviewer/chair recruitment; sharing statistics of the ethics reviewing and processes from several conference cycles would be helpful.

Technical issues:

• integration of ethics reviewing into ARR; reviewer contact and commitment to reviewing is done outside ARR, follow-up of the review process is done outside ARR
• it would be useful to have notifications for ethics review submission
• education of reviewers may be needed (confusion between messages related to ethics reviewing vs. regular reviewing)