2020Q3 Reports: General Chair

ACL 2020 post-conference report: General Chair

Dan Jurafsky

The 58th annual meeting of the Association for Computational Linguistics (ACL) took place online from July 5th through July 10th, 2020.

Summary of what happened

Positive:

• We ran a virtual ACL with only a few months notice, and it wasn't a disaster!
• Largest ever in every way (including attendance)!
• Also most diverse ACL ever!
• The new tracks (Ethics, Interpretation, Taking Stock, etc) were very popular; they were the most watched talks (https://twitter.com/yoavgo/status/1282459579339681792)
• People had lots of great discussion on chat; ACL conferences should ramp up use of chat media.
• Mentoring of reviewers was a good start
• The team was awesome!! Every set of chairs was amazing and rose to the occasion to deal with the shift to virtual.
• This year we added publicity team in China, important for reaching out to a large audience there.

Negative:

• The infrastructure team had to work way too hard, especially the last two weeks + the conference, it was a completely unreasonable load on them. Beware! Advice on infrastructure below.
• People in China had to use VPNs to watch the pre-recorded talks.
• All the software (but especially SlidesLive, but also Zoom) had all sorts of problems. We probably don't want to reuse SlidesLive, but Zoom we may have no choice.
• Ethics and PCC stuff caused headaches that could have been avoided with ethics statements as part of submissions and best paper awards.
• Communication with authors was very very difficult; there was no one emailing list that got every author at main, demo, SRW, tutorials, workshops. This caused innumerable problems.
• Mentoring of reviewers was a good start but needs tweeking
• Having the demos and SRW at the main conference, but not run by the PC, was problematic, in terms of scheduling (it would have been better to have demos and main papers scheduled by same person), communication (every message had to be sent through 3 separate groups of chairs), ethics (papers and awards weren't checked consistently). Probably we should rethink this structure; maybe make demo chairs a kind of area chair? And move SRW to the workshop days?

Overview of suggestions future conferences

Switching to Virtual: Advice on Personnel and Chairs

• Either pay for a completely professional conference organizer that does all infrastructure themselves, or if we're going to continue to use the miniconf website that we used this year, then:
  1. Have double the number of virtual infrastructure chairs than we did. We had essentially 9 (6 virtual infrastructure chairs led by Hao Fang and Sudha Rao, plus 2 captioning chairs, plus 1 volunteer chair). Probably you need 18, including specific chairs for live Video (Zoom or whatever), the virtual website, talk recordings, the non-virtual website, chat, and more.
  2. Assign completely separate infrastructure chairs for the workshops, who can be closely in contact with the workshop chairs at all times.
  3. Volunteer coordinator should be more tightly integrated from the beginning with the infrastructure chairs, since the volunteers are mainly helping with infrastructure
• Either way the workshops will need stricter deadlines for finalizing schedules and materials, much earlier than with a physical conference, and did I mention strictly enforced?
• Create a Virtual Social chair to find ways to get people to meet each other more (see https://www.aclweb.org/adminwiki/index.php?title=2020Q3_Reports:_Publicity_Chairs). Esther Seyffarth volunteers to do it for EMNLP or COLING.
• Probably need an ethics chair.
• Pub chair info needs to be gotten to them much earlier; pub chairs need to talk to previous pub chairs immediately and work on new FAQs.
• Diversity and Inclusion chairs created many subchairs to run various D&I committees. This worked well, and should be done again.

  All the non-virtual-infrastructure chairs were also overworked. Probably we need more chairs, 3 instead of 2 chairs for various things, 4 PCs instead of 3, or even possibly 2 general chairs with specific duties.

Switching to Virtual: Advice on Software Partners

• We think the ACL should probably not use SlidesLive going forward. They caused a number of problems:
  • They initially gave us the impression they would handle creating all the zoom rooms for the conference, then changed their mind
  • Then they gave us the impression they would at least handle creating all the zoom rooms for the workshops, then changed their mind at the last minute, too late to create all the zoom rooms for the workshops.
  • They initially gave us the impression that they would do human captioning of live sessions, but then changed to ASR captioning
  • They clearly struggled with processing 1500+ video presentations in just around 2 weeks; they also changed their commitment to what they were planning to deliver in terms of captioning multiple times during the pre-conference period\\\
  • Their setup for allowing authors to correct captions was not easy to use.
  • Their recording setup didn't allow any author editing, which made it frustrating to have to record in single takes
  • We only realized later that they were willing to paste together multiple segments, so tutorials could have been recorded in multiple chunks.
  • Our contact, Alex Chandler, wasn't technical and didn't seem to be a good listener, so complaints often were not addressed.
  • One problem occurred on the day before the workshops when they repeatedly ignored requests for live session ids for the workshops, forcing the Infra Chairs to stay up until the wee hours desparately trying to get them on the phone or email.
  • They did not do well at helping speakers navigate the 40-second delay between the live zoom rooms, and the livescreen window (failing, for example, to explain to speaker that they had to turn off the audio on the livescreen when they were entering the zoom room). It was their lack of clear explanations and dry runs that caused the much-commented-upon 40-second delayed echo in the LTA and in many other speakers, and was especially a problem in the workshops.
• However, we don't know if there are strong alternatives:
  • One option is to find a professional virtual conference organizer that runs everything (including running the website), as well as the videos. We were unable to find such, so I don't know if this is poissible
  • SlidesLive seemed better for our needs than the alternative we considered (we chose Slideslive over Underline and Freeman when it was still a physical conference, and then they were the conference for ICLR and the ICLR chairs seemed happy with them; we didn't find a lot of alternative options at that point).
  • We assume by now there are probably other options, but inclusion is extremely important to keep in mind (making it possible to record for folks in countries without ability to buy video recording software), as is having professionals deal with the livestream events.
  • Slideslive also checked every video for sound or other problems
• Live human captions (CART) were done by ACS Captions who were great. we should use them.

Switching to Virtual: Other Advice on Infrastructure

• QA sessions could possibly be improved by incorporating more poster-like elements (like having speakers do 5-minute spiels every 30 minutes or etc)
• Speakers should upload their slides as well as the talks
• How to do virtual live sessions: work out methods for taking questions, probably involving 2 moderators who screen and present questions (and so as to avoid the "more of a comment than a question" people)
• if you use slideslive or any such operator that uses a delay for transcription, need more training of each speaker about how to deal with delay.
• minor: if you use miniconf, fix so that the whole website can shift timezone.
• Nitin's suggestion: "use one of the many bot frameworks (Rasa, BotKit, Botpress, Hubot) to make a simple FAQ bot that can answer repeated questions"

Getting more Author Permissions:

• Decide whether to ask/require people for permission to record their chat conversations
• Explicitly require people to give permission to store the captions of their videos
• Somehow encourage people to correct the captions of their videos.

Global Inclusion:

• For videos (and docs and chats), decide whether to solve problem of viewability in China, Indonesia, Vietnam, etc., or else continue to require VPNs.
  • Vimeo (used by ACL Anthology and also SlidesLive pre-recorded talks) not visible in China, Indonesia, Vietnam
  • SlidesLive livestream (but not pre-recorded, which uses Vimeo) is is visible in China
• Iran is a particularly difficult case, can't use Google or Microsoft either
• Proposal: we should assume that any single technology could be banned at any time by some government, so we should set up mirrors.

Ethics:

• Require an ethics statement of all papers, add ethics to reviewing form, and add ethics, and also diversity & inclusion to all paper award decision criteria.
• Probably need an ethics chair

Historical note: switching to Virtual: Timeline of Decision to go online:

• Feb 5: Concern about coronavirus expressed by Seattle Convention and Tourist board.
• Mar 2: Companies begin to restrict travel, Stanford switches to virtual classes
• Mar 4: ACL Exec meeting, I briefly fill Exec in that virtual is an option, but still low-probability
• Mar 6: Things worsen. We change Mar 11 site visit to be a virtual meeting with hotel
• Mar 11: At meeting, Hyatt refuses to discuss refunds or cancellations until mid-April
• Mar 17: Things worsened more, I ask ACL Exec advice, they mostly advise going virtual, we wait to hear about hotel $$$
• Mar 25: Priscilla gets word that we can probably delay our Hyatt contract til NAACL 2021, removing the final obstacle to going virtual. We delay a few more days for ACL Exec to confirm all this.
• April 2: We announce the decision to go virtual.