2021Q3 Reports: Ethic Advisory Committee (EAC)

Executive Summary

For the ACL-IJCNLP ethics review process, the EAC successfully supported the processing of 247 papers, leading to a total of 28 Conditional Accept recommendations and no rejections from the EAC on ethical grounds. Our process included some effort-reducing modifications to previous practices, including additional EAC triage to reduce reviewing load for papers with only common, remediable ethical issues.

An imposed compressed timeline, confusion about committee scope, and complexities introduced through the use of Softconf caused several processing issues that we believe could be remedied with restructuring and further discussion at the organizational level about the structure, role, and support needed for the EAC process.

Statistics

Approximately 247 papers were referred to our committee for review. Of those, 50 were found to have been referred in error (either with no apparent justification or a justification suggesting a non-ethical reason for referral) and thus were not reviewed. 13 were withdrawn prior to the ethics review process being completed on the basis of the technical reviews.

Of the remaining, approximately 115 were deemed to have common issues that did not require detailed review in order to provide a recommendation. These papers with only common issues identified after further chair scrutiny received one reviewer instead of two; for papers deemed very likely to be rejected on the basis of technical review, that review was conducted by the committee chairs, while other reviews were conducted by the reviewer pool. Papers with more complex issues always had two ethics reviewers.

To reduce the workload of a fairly small committee, decisions were only made for papers that were on track to be accepted for ACL or Findings of ACL after technical review. After conducting our review, 28 papers were given Conditional Accept decisions by our committee, meaning they were asked to submit changes prior to final acceptance. No papers were rejected by our committee.

Importantly, this is not to say that all papers submitted to ACL would have been accepted by our review committee; instead, several papers that would have been rejected by our committee were already rejected in the technical review process due to methodological concerns (including those tied to ethics issues).

Ethics Reviewing Timeline

• Pre-review and recruiting (~1 month):
  • February–March: Recruit reviewers
  • March 20: Technical review deadline
• Reviewing (~1 month):
  • March 25: Received ethics review recommendations (start of author response period)
  • March 29: Send ethics review assignments to reviewers
  • March 31: Author response due
  • April 14: AC meta reviews due
  • April 15: Ethics reviews due
• Discussion (3 weeks):
  • April 16-22: Ethics review discussion
  • April 23: Received list of tentative paper decisions from technical review
  • April 26: Returned ethics decisions for papers slated by SAC to be accepted by technical review
  • April 29-May 4: Ethics metareview entry and resolution for papers whose decision changed from SAC recommendation/papers with late SAC recommendations
• Decisions:
  • May 5: Decisions sent out by PCs, along with instructions for conditional acceptance from EAC
• Conditional Accept Processing (~1 month)
  • May 5–June 1: Correspondence with conditional acceptance authors to help with revision
  • June 1: Camera ready deadline
  • June 7: Final conditional acceptance decision given

Pain Points

SoftConf. The current structure of SoftConf is poorly equipped to handle conducting ethics reviews on papers. In particular, our reviewers needed to be able to write reviews out of cycle with the other paper reviewers and with a different review form, which prohibited us from having our reviewer invitations overlap with the regular invitation period and complicated the review assignment and deadlines. Through the shared system, the only option to assemble a form for our reviewers was to "modify" the technical review form to repurpose some fields of the original review form to create the new form. As the progression for technical review continued, we would often also find information in our "area" incorrect, previously accessible information rendered no longer viewable after a technical review deadline passed, or attachments and other necessary information blocked from view while we conducted review. Our final system heavily leveraged additional manually-updated spreadsheets instead of direct SoftConf integration, which led to multiple cases of confusion when spreadsheets fell out of sync with SoftConf or when there was no SoftConf-amenable way to confirm a task was completed for all relevant papers.

Timeline. The timeline for the Ethics Advisory Committee had to be built around a contracted timeline for the original conference review process, and suffered for not having been included in the thought process for the original schedule. Almost every step of the process was rushed as a result, from the early publication of the guidelines and structure for ethics review through to the final decisions. This included the modification of the original schedule, in which ACs would help identify some papers for ethics review before reviews were turned in, to only have a single round of review starting after technical reviews were due. Timing issues were further exacerbated by issues with Softconf’s incompatibility, which often required manual intervention and multiple day delays in order to access information needed for tight decision turnaround times. While we are confident in the committee’s work, we regret that so much had to be rushed.

Ethics review expectations. While our committee attempted to provide detailed information for our reviewers providing the expectations in terms of length and contents of ethics review and what questions to consider, the effort to heed these instructions varied considerably. It is possible that the repurposed form led to some confusion about which instructions to honor, those attached to the instruction email or those embedded in the form. Regardless, it would be good to establish more consistent norms.

Committee scope. Several times, there was confusion between the PCs and EAC about which issues did or did not fall into the purview of the EAC. This included uncertainty about whether our committee would handle issues with multiple submission or other violations of the call for participation guidelines, whether to verify redistribution rights for anonymous datasets as a possible violation of the TOS of a website, as well as whether an AC rejection decision that includes ethics evidence should be recharacterized or ported to be an EAC rejection instead due to the ethics nature of the discussion. It seems that there may be conference-wide confusion about our committee’s role, which will need to be clarified to help the committee run more smoothly in the future.

In particular, we stand by the principle that our committee primarily exists to address author oversights in addressing and discussing issues that might cause ethical problems in their work and, in rare cases, to ensure that papers with unfixable ethical problems in their current protocol or presentation not be presented at the conference. We believe our role as a committee is one of education and reformation of both authors and the technical review committee. We want to help all involved in review to understand ethical issues as methodological flaws in a paper that are as relevant to the validity of a paper as whether a mathematical proof has an error. Just as we would want to make sure that a paper that includes a proof has at least one reviewer with enough expertise to judge the proof’s validity, we want papers with possible ethical concerns to have at least one reviewer able to verify that the authors have not compromised their work.

In light of this, we wish to ensure that the existence of the EAC does not inhibit, prevent, or discourage reviewers, ACs, or SACs from making decisions to reject a paper on the basis of ethical concerns. While our committee will presumably provide reviews and discussion for those papers, we believe it should be the ultimate goal of the ACL community that these considerations be considered as a relevant part of the standard review process.

Recommendations for future conferences / committees

Setting a timeline. The EAC chairs should be recruited early in the conference planning process, with sufficient time to participate in setting the review schedule to ensure there is time for ethics review and clarity about how papers will be referred and evaluated. Ideally, this should also leave time for the next EAC to get their bearings and determine in consultation with the PC whether any revisions should occur in the review process (for instance, the ones we suggest below).

Workflow. We agree that the PC needs a one-stop contact point for ethical concerns, for simplicity. It may be worthwhile to restructure the phases of where contact with the committee occurs, including performing many reviews at the start of the technical review process so ethics review will be available to authors and ACs at the same time as technical reviews. If ACs or SACs don’t have sufficient information after author response, or PCs have remaining concerns, we could then do additional expedited reviews in a second round for a smaller set of papers to help clarify unresolved issues. For the smaller set of papers where EAC review deems it necessary to conditionally accept (or even more rarely, reject) a paper, we can proceed with the existing process to do so.

One suggested workflow is the following:

• During review assignment:
  • ACs and SACs flag papers during review assignments or the early part of the review period, instead of waiting until the technical reviews are done. These should include with their recommendations brief prompts for why the recommendation for review was made (e.g. “new dataset with crowd workers” or "uses medical data").
  • The EAC receives flagged papers, verifies the recommendations, and assigns ethics reviewers either at the same time or just after technical reviewers are assigned. These reviewers will have access to the notes from the ACs for the reasons for review.

• At the technical review due date/rebuttal period:
  • ALL reviews from this round will be due at the same time.
  • Rebuttal can concern both technical and ethical reviews, perhaps with separate boxes

• At AC decision time:
  • Ethics reviews can be consulted in the review discussion if needed.
  • Regardless of EAC status, ACs and SACs may ask for more clarification from the EAC to interpret ethics reviews or provide ethics feedback. EAC chairs can help consult or answer questions, allowing the ACs to use ethics review information to inform their decisions.
  • For papers where ACs and EACs conflict, where outstanding questions remain, or where an ethical question arose after the reviews were submitted, a further expedited ethics review addressing the specific remaining questions can be initiated.
  • EACs will continue to track papers where ethics reviewers recommended conditional accept or reject decisions to track through the remaining process.

• At PC decision time:
  • For papers that were recommended for acceptance by the SACs that were referred to the EAC, the EAC will provide ethics recommendations to the PCs (either Accept, Conditional Accept, or Reject). These decisions will be determined from ethics reviews, rebuttal, discussion with the ethics reviewers and AC, and (if needed) expedited further review to resolve uncertainties.
  • If PCs differ with the SACs by accepting a paper that was (a) recommended for rejection by the SAC and (b) reviewed by the EAC, the PCs will give the EAC chairs time to review the paper’s reviews and provide a decision for it as well.
  • Once PCs and EAC chairs have confirmed agreement on the decisions on EAC-reviewed papers, EAC chairs will write meta-reviews for all papers that were (i) Accepted after EAC review or where the EAC affected the paper’s final decision (i.e., (ii) Conditional Acceptance or (iii) Rejection on Ethical Grounds). Note that if a paper is rejected through the technical review process, no EAC meta-review will be written, even if the meta-review from the AC suggests ethical grounds.

• After decisions are released:
  • EAC chairs will be available for authors of papers with (ii) Conditional Acceptance decisions to help ensure their paper is prepared prior to the camera-ready. Each of the meta-reviews for these papers will have a list of criteria for the authors to meet. For simplicity, we suggest that the EAC chair responsible for the meta-review respond to authors for that paper when possible.
  • At the camera-ready deadline, EACs will peruse the submitted papers in consultation with the meta-review guidance to ensure all conditions of acceptance are met. If so, the condition will be removed; if not, the EAC will reach out to the authors and confirm that their paper will be rejected if changes aren’t made. If timing allows, it may be helpful to budget 1-2 days here in case of late revisions or misunderstandings with ethics review.

Adjusting to new submission workflows. In the future, we recommend that the EAC and PC chairs strategize on how to deal with auxiliary submission processing. When submissions may come from outside the current standard review process (such as rolling reviews and TACL papers), the EAC review process needs to be thought through.

If the rolling review process usurps the rest of the normal submitting process (at some point), we need to ensure that ethical reviews are considered (e.g., imposing the same strict deadline for papers to come under the regular submission timeline.)

ACs should be flagging papers at the point of assigning reviewers to papers. I.e., the papers to the ethics review committee should be flagged for EAC review at the point of reviewer assignment. This will help address the time pressure for the EAC review process.

Standard areas of concern. We recommend that future EAC chairs take special attention to specific areas that have outsized exposure to areas that produce more ethical and privacy concerns These include NLP Applications, Language Resources, Computational Social Science, and Generation. This could be reflected in a better balancing between number of papers loaded to reviewers and/or chairs of the EAC when reviewing papers in these special areas.

In line with the above, there were many "standard" concerns that we addressed by constructing boilerplate review texts that we applied to consistently message common concerns, such as dual use, crowd worker compensation, data privacy, as well as copyright and IP.

We recommend that future EAC chairs also categorise such concerns, and prepare common texts, scoping and workflow for handling such cases. We are happy to provide our resources for this to future chairs.

Datasets. At present, it is not clear whether situations where legal rights to redistribute data are in question fall within the purview of the EAC. We would recommend that the authors take the responsibility for the legal liability of publicizing private or commercially copyrighted data or intellectual property. While the ethics committee can help check and call into question privacy concerns about datasets, ownership and copyright of datasets are legally fraught topics that fall outside the expertise we expect of EAC reviewers.

We recommend that a checkbox on the START submission form be provided to allow authors to aver that they have legally binding permission to publish their data; and to indemnify the ACL and committees for any legal repercussions.

Recruitment. The list of ethics reviewers this year was compiled by finding an international group of experienced reviewers with demonstrated interest in supporting ethics review. While we were able to find sufficient reviewers for our modified protocol, if we were to give two reviewers for every correctly referred paper (which would need to happen if we didn’t already have reviews by the time of assigning ethics reviewers), we would have fallen significantly short for reviewer needs. If this structure is to be revisited, we would encourage a forward-thinking strategy to identify additional high-quality reviewers from a broader swath of the research community. This may require having EACs collect lists of reviewers who referred ethical issues and who might be qualified for ethics review based on high-quality past technical reviewing. This might be implemented where the EACs poll the SAC and ACs for names within each area that did due diligence for ethical concerns. We also recommend that EAC reviewers have term limits (where possible) to motivate and ensure that EAC reviewers come from a wide and diverse swath of the memberships and not by a fixed coterie of incumbent reviewers.