2021Q3 Reports: NAACL-HLT 2021

Conference summary

• We originally planned an in-person NAACL 2021 but switched to a fully virtual format (the decision was finalized less than three months before the conference).
• The NAACL virtual conference hosted 6 tutorials, 17 system demonstrations, 39 industry track papers, 499 main conference and CL/TACL papers, 6 plenary invited talks (4 main track and two industry), 2 panels (Startups in NLP and Careers in NLP), 22 workshops, and a large set of social and thematic gatherings. More details on the conference structure are at https://2021.naacl.org/blog/conference-structure/.
• 2,622 participants registered for NAACL. We reduced registration rates further compared to recent virtual conferences: student registration was $25 early/ $50 late, and regular registration was $125 early/ $175 late.
• We chose Underline (https://underline.io/events/122/reception) for our virtual infrastructure, following the experience of COLING 2020. We also used the Whova conference app for text-based chat and based the review process on START.

Main Innovations

• We followed a more well-defined process for ethics reviews, proposed by the ethics co-chairs Emily Bender and Karën Fort, and refined and implemented in coordination with the program co-chairs and the demo and industry track chairs. Key changes were allowing additional space in submissions to discuss ethical considerations and establishing a category of papers accepted conditionally on addressing ethical concerns together with a timeline and process for an additional stage of review of re-submissions. See the reports of the ethics chairs (https://2021.naacl.org/blog/ethics-review-process-report-back/) and the program co-chairs (https://2021.naacl.org/blog/ethics-review-process/) for more details.
• There was no distinction between oral and poster presentation at the main conference, and all papers were presented in both assigned video calls for groups of five-six papers, and in poster sessions in Gather.town.
• The D&I committee in collaboration with the NAACL Exec put together a D&I Grant Initiative (https://2021.naacl.org/blog/dei-grants/). A report by the D&I committee including ideas on how to improve financial accessibility further in the future is at https://2021.naacl.org/blog/dni-subsidies/. The full schedule of social events the committee organized can be seen at https://2021.naacl.org/program/social/; recordings of some of the discussions are available at https://www.youtube.com/channel/UCbdq7M48OgNoPIa9Uj_IBCw.
• The publication chairs developed new tools to automatically check and report formatting violations in an easy-to-understand way (https://github.com/yz-joey/ACLPUB/blob/master/aclpub_check/formatchecker.py).

Virtual Infrastructure

We sought a virtual infrastructure solution that would enable pre-recorded talks, interactive poster sessions, live group video calls, text-based asynchronous chat, and live-streaming of plenary sessions. Based on a small scale survey prior to the conference, a large majority of potential attendees and authors were interested in being able to interact in small groups around posters, and it was important to us that we provide that. Other considerations were having flexible video recording software, and sufficient technical support for live presentations.

The components of the virtual infrastructure were:

• Pre-recorded talks based on Screencast-O-Matic. The paper talk submission deadline was about four weeks before the conference. Underline staff checked each submission and communicated with authors to get any issues resolved.
• Website provided by Underline, including a conference schedule displayed in a local time zone and organized in tracks.
• Posters, discussion sessions, social lounge, and sponsor spaces in Gather.town. The spaces were configured by Underline. Our virtual infrastructure committee only needed to provide maps for the poster sessions.
• Groups of papers organized in sessions, together with their recorded videos, poster pdfs, text-based chat, and video call information (Zoom-based), set-up by Underline.
• Plenary talks which were produced and live-streamed by Underline using StreamYard.
• Sponsor booths including chat, video calls, and spaces in Gather.town (depending on sponsorship tier).
• The Whova conference app was used for asynchronous person-to-person text chat and threaded discussions on channels. Through Whova, 97 meetups were organized, there were 58 discussion channels (with 1,080 posted messages), 77 job postings, and 246 break-the-ice messages.
• Recordings of the live sessions and processed pre-recorded talks were provided to us after the conference. Recordings of authors that give permission will be uploaded to the ACL Anthology.

We had a positive experience working with Underline as they handled all aspects of the infrastructure and were also very open to our requests to customize the experience (provided changes were feasible in the given timeframe).

Suggestions for process improvements

• There were data errors on dozens of papers -- missing authors, wrong titles etc. Perhaps it would be possible to specify the metadata automatically though download from the the review system (e.g. START).

• If configuration of posters in Gather.Town can be programmatic instead of manual, inconsistencies and last minute missing posters could perhaps be avoided.

• It was very difficult to send announcements to all authors across tracks (main, demos, industry, SRW, workshops), and to check that all authors have completed required tasks. This would be a valuable feature for future conferences.

• We had several instances of session chairs that did not attend their assigned session. We had both a session chair and a volunteer assigned to each session, and in one instance both were missing. It would be good to have emergency on-call session chairs and volunteers.

• Having a slack channel for communication among conference organizers, session chairs, volunteers, and Underline staff during the conference was very useful. We could have set more clear expectations on when people need to be available on that channel.

• Some workshops did not finalize their schedules till 1-2 days before the conference. This made it very difficult to coordinate and assign volunteers, and to update the website schedule on time. Workshops should be required to set at least their start and end times (program overview) a week or more before their start day.

• NAACL decided to not provide pre-recorded talks to workshop papers, but some workshop organizers asked their authors to submit pre-recorded talks. The NAACL 2021 workshop co-chairs did a great job and worked very hard to coordinate the diverse workshop requirements. To make the process more efficient, automation to gather requirements and enforce policies would be useful.