2020Q3 Reports: General Chair

ACL 2020 post-conference report: General Chair

Dan Jurafsky

The 58th annual meeting of the Association for Computational Linguistics (ACL) took place online from July 5th through July 10th, 2020.

Summary of what happened

• We ran a virtual ACL with only a few months notice, and it wasn't a disaster!
• ACL was the largest ever, with record number of submissions (3,429), accepted papers (778), and attendees (4,972).
• It was also the most diverse ACL ever!
• For our virtual infrastructure we chose to mostly follow the ICLR model, with some modifications:
  • pre-recorded talks, recorded using the SlidesLive web-based tools, with ASR-based captions hand-corrected by authors.
  • a website dashboard based on the miniconf package, with a schedule page that can be adjusted to each timezone, a separate webpage for each paper that gives its pre-recorded video, paper, zoom channel, and Rocketchat link.
  • Livestreamed plenary sessions, also run by SlidesLive, with a mix of recorded talks and live presentations
  • Rocketchat sessions associated with each talk
  • Two 1-hour Zoom Q&A sessions for each of the talks.
  • Tutorials that were a mix of live sessions and pre-recorded talks
  • Workshops that also used a mix of live sessions and pre-recorded talks

Other Positive Results:

• The new tracks (Ethics, Interpretation, Taking Stock, etc) were very popular; they were the most watched talks (https://twitter.com/yoavgo/status/1282459579339681792)
• People had lots of great discussion on chat; ACL conferences should ramp up use of chat media.
• Mentoring of reviewers was a good start
• The team was awesome!! Every set of chairs was amazing and rose to the occasion to deal with the shift to virtual.
• This year we added publicity team in China, important for reaching out to a large audience there.
• This year (as last year) we got CC-BY4.0 permissions from authors to upload their videos of talks to the ACL Anthology. New this year, we also uploaded transcripts of the talks, making the videos more accessible, first asking authors to correct the ASR transcripts of their talks. We need a way to encourage more authors to correct their transcripts.

Negative:

• The infrastructure team had to work way too hard, especially the last two weeks + the conference, it was a completely unreasonable load on them. Beware! Advice on infrastructure below.
• People in China had to use VPNs to watch the pre-recorded talks.
• All the software (but especially SlidesLive, but also Zoom) had all sorts of problems. We probably don't want to reuse SlidesLive, but Zoom we may have no choice.
• Ethics and PCC stuff caused headaches that could have been avoided with ethics statements as part of submissions and best paper awards.
• Communication with authors was very very difficult; there was no one emailing list that got every author at main, demo, SRW, tutorials, workshops. This caused innumerable problems.
• Mentoring of reviewers was a good start but needs tweeking
• Having the demos and SRW at the main conference, but not run by the PC, was problematic, in terms of scheduling (it would have been better to have demos and main papers scheduled by same person), communication (every message had to be sent through 3 separate groups of chairs), ethics (papers and awards weren't checked consistently). Probably we should rethink this structure; maybe make demo chairs a kind of area chair? And move SRW to the workshop days?
• Workshops were much less well dealt with on our virtual site than the rest of the conference, since they all finalized their scheduled very late and we weren't able to get their infrastructure (web pages, schedules, zoom links) created sufficiently.
• It would have been great to have been able to make the website visible a few days earlier, to let people start watching papers.

Overview of suggestions future conferences

Switching to Virtual: Advice on Personnel and Chairs

• Either pay for a completely professional conference organizer that does all infrastructure themselves, or if we're going to continue to use the miniconf website that we used this year, then:
  1. Have double the number of virtual infrastructure chairs than we did. We had essentially 9 (6 virtual infrastructure chairs led by Hao Fang and Sudha Rao, plus 2 captioning chairs, plus 1 volunteer chair). Probably you need 18, including specific chairs for live Video (Zoom or whatever), the virtual website, talk recordings, the non-virtual website, chat, and more.
  2. Assign completely separate infrastructure chairs for the workshops, who can be closely in contact with the workshop chairs at all times.
  3. Volunteer coordinator should be more tightly integrated from the beginning with the infrastructure chairs, since the volunteers are mainly helping with infrastructure
• Either way the workshops will need stricter deadlines for finalizing schedules and materials, much earlier than with a physical conference, and did I mention strictly enforced?
• Create a Virtual Social chair to find ways to get people to meet each other more (see https://www.aclweb.org/adminwiki/index.php?title=2020Q3_Reports:_Publicity_Chairs). Esther Seyffarth volunteers to do it for EMNLP or COLING.
• Probably need an ethics chair.
• Pub chair info needs to be gotten to them much earlier; pub chairs need to talk to previous pub chairs immediately and work on new FAQs.
• Diversity and Inclusion chairs created many subchairs to run various D&I committees. This worked well, and should be done again.
• All the non-virtual-infrastructure chairs were also overworked. Probably we need more chairs, 3 instead of 2 chairs for various things, 4 PCs instead of 3, or even possibly 2 general chairs with specific duties.

Switching to Virtual: Advice on Software Partners

• While using a video-presentation company like SlidesLive had many advantages, in the end our advice is that ACL should try to find alternatives to SlidesLive going forward. They caused a number of problems:
  • They initially gave us the impression they would handle creating all the zoom rooms for the conference, then changed their mind
  • Then they gave us the impression they would at least handle creating all the zoom rooms for the workshops, then changed their mind at the last minute, too late to create all the zoom rooms for the workshops.
  • They initially gave us the impression that they would do human captioning of live sessions, but then changed to ASR captioning
  • They clearly struggled with processing 1500+ video presentations in just around 2 weeks; they also changed their commitment to what they were planning to deliver in terms of captioning multiple times during the pre-conference period
  • Their setup for allowing authors to correct captions was not easy to use.
  • Their recording setup didn't allow any author editing, which made it frustrating to have to record in single takes
  • We only realized later that they were willing to paste together multiple segments, so tutorials could have been recorded in multiple chunks.
  • Our contact, Alex Chandler, wasn't technical and didn't seem to be a good listener, so complaints often were not addressed.
  • One problem occurred on the day before the workshops when they repeatedly ignored requests for live session ids for the workshops, forcing the Infra Chairs to stay up until the wee hours desparately trying to get them on the phone or email.
  • They did not do well at helping speakers navigate the 40-second delay between the live zoom rooms, and the livescreen window (failing, for example, to explain to speaker that they had to turn off the audio on the livescreen when they were entering the zoom room). It was their lack of clear explanations and dry runs that caused the much-commented-upon 40-second delayed echo in the LTA and in many other speakers, and was especially a problem in the workshops.
• However, it's possible that they will begin to improve as they grow as a company, and frankly we don't know if there are strong alternatives:
  • One option is to find a professional virtual conference organizer that runs everything (including running the website), as well as the videos. We were unable to find such, so we don't know if this is possible.
  • SlidesLive seemed better for our needs than the alternative we considered (we chose Slideslive over Underline and Freeman when it was still a physical conference, and then they were the conference for ICLR and the ICLR chairs seemed happy with them; we didn't find a lot of alternative options at that point). We assume by now there are probably other options.
  • In general, a recording company like SlidesLive has advantages over asking participants to just record themselves: consistency, affordability to participants (no need for folks to buy video recording software), professionals dealing with problems in livestream. slideslive also checked each of the 1500+ videos by hand for sound and composition problems, and ran them through ASR captioning..
• Live human captions (CART) for the livestream plenary events were done by ACS Captions who were great. We should reuse them.

Switching to Virtual: Other Advice on Infrastructure

• Better ways to create a list of "papers to see" and download entire schedules (instead of downloading one paper-slot at a time)
• QA sessions could possibly be improved by incorporating more poster-like elements (like having speakers do 5-minute spiels every 30 minutes or etc)
• Speakers should upload their slides as well as the talks.
• How to do virtual live sessions: work out methods for taking questions, probably involving 2 moderators who screen and present questions (and so as to avoid the "more of a comment than a question" people)
• if you use slideslive or any such operator that uses a delay for transcription, need more training of each speaker about how to deal with delay.
• Fix miniconf so that the whole website (including workshops, plenary pages, etc) can shift timezone, not just the schedule page.
• Perhaps building an FAQ bot that can answer repeated questions using a standard toolkit (Rasa, BotKit, Botpress, Hubot); suggestion from Nitin Madnani.

Getting more Author Permissions:

• This year (as last year) we got CC-BY4.0 permissions from authors to upload their videos of talks to the ACL Anthology. In addition, we asked authors to correct the ASR transcripts of their talks, and we are also uploading the transcripts to the Anthology.
• Decide whether to ask/require people for permission to record their chat conversations
• Explicitly require people to give permission to store the captions of their videos
• Somehow encourage people to correct the captions of their videos.

Global Inclusion:

• For videos (and docs and chats), decide whether to (a) solve problem of viewability in China, Indonesia, Vietnam, etc., or (b) continue to require that people in those countries use VPNs.
  • Vimeo (used by ACL Anthology and also SlidesLive pre-recorded talks) is not visible in China, Indonesia, Vietnam
  • SlidesLive livestream (but not pre-recorded, which uses Vimeo) is visible in China
• Iran is a particularly difficult case, can't use Google or Microsoft either
• Proposal: we should assume that any single technology could be banned at any time by some government, so we should set up mirrors for videos.

Ethics:

• Require an ethics statement of all papers, add ethics to reviewing form, and add ethics and also diversity & inclusion to all paper award decision criteria.
• Probably need an ethics chair (and then need to decide which things are PCC and which are ethics)

Historical note in case useful: switching to Virtual: Timeline of Decision to go online:

• Feb 5: Concern about coronavirus expressed by Seattle Convention and Tourist board.
• Mar 2: Companies begin to restrict travel, Stanford switches to virtual classes
• Mar 4: ACL Exec meeting, I briefly fill Exec in that virtual is an option, but still low-probability
• Mar 6: Things worsen. We change Mar 11 site visit to be a virtual meeting with hotel
• Mar 11: At meeting, Hyatt refuses to discuss refunds or cancellations until mid-April
• Mar 17: Things worsened more, I ask ACL Exec advice, they mostly advise going virtual, we wait to hear about hotel $$$
• Mar 25: Priscilla gets word that we can probably delay our Hyatt contract til NAACL 2021, removing the final obstacle to going virtual. We delay a few more days for ACL Exec to confirm all this.
• April 2: We announce the decision to go virtual.