2020Q3 Reports: Virtual Infrastructure Chairs

The virtual conference infrastructure of ACL2020 consists of 4 major components:

• SlidesLive for pre-recorded talks & video live-streaming
• Zoom for live QA sessions
• RocketChat for async text-based discussions and announcements
• the virtual conference website as the central platform to present all information about the conference

Due to limited time, we follow the infrastructure setup of ICLR2020, with some necessary extensions including the plenary sessions, tutorials, virtual sponsor booths, etc. In the remaining sections of this report, we provide a high-level summary of each component and discuss potential improvement can be made in future. At the end, we provide some general advice for future virtual conferences based on our experience.

SlidesLive

SlidesLive is a company that has helped with several previous conferences including ICLR, ICML, etc for recording the videos of the talks, post processing them and putting the slides and the presenter side by side. We initially negotiated with several companies including them for the physical conference in Seattle. The companies we discussed with with their Point Of Contact (POC) were:

• SlidesLive, POC: Katherine (Katie) Millar, katherine@slideslive.com
• Underline, POC: Sol Rosenberg, sol@underline.io
• GloCastLive, POC: Arran Moffat, arran@glocast.com
• BASH FILMS, POC: Mark Bashian, mbashian@bashfilms.com
• Freeman, POC: Brian Holm, Brian.Holm@freemanco.com

We have put all the information we collected from these companies including the quotes here for your reference. Please note that these are for the time when the conference was supposed to be physical, nevertheless many of them also offer virtual experience.

After deciding to have the conference fully virtual we signed a contract with SlidesLive and relied on them for our keynotes and workshops besides the post processing of the videos. Below we list “our experience” working with them, noting these before signing a contract with them should be very helpful to have a great experience working with them, if you choose them as the company you want to work with.

General suggestions

Here we summarize our general suggestions, especially before signing a contract with them.

• We strongly suggest enforcing an early deadline for a finalized program of conference and specially workshops. This is very important as it helps to reduce communication overhead between SlidesLive and the rest of the conference.
• It is very important to clarify with SlidesLive “before signing the contract” how many live zoom QA sessions they are going to cover. For example, they did not cover “any” of the QA sessions for papers in main conference, demo sessions, sponsors or D&I events, and it was not clearly communicated to us pre-signing the contract. They also did not cover any of the parallel QA sessions happening in our workshops. We ended up creating our own zoom team to handle these. To prevent this from happening we suggest asking them to include a technical staff as well during discussions, marketing representatives might not be aware of these types of details.
• Slideslive does its best to be responsive and usually tries to answer our questions as much as they can. We recommend asking them to assign more number of “technical staff” especially during the conference. We sometimes felt having only two contacts from them is absolutely not enough especially given the fact that one of them was not as technically confident as the other one.
• There was a 8 sec delay in live presentations (the video in zoom vs the one streaming on the SlidesLive website) which our general chair (Dan) brought up during SlidesLive demo session. Please request them to resolve this.

Accessibility

Please note that they only provide captioning for the videos / live sessions they are handling, i.e., if you end up having to create your own zoom meetings they are not going to provide captioning for them. Another note, when we made the contract with Slideslive, it wasn’t clear in the contract that they only provided automatic captioning service instead of high quality human-edited captioning. Please make sure to go through all these details before signing a contract with them.

Non-profit discount

SlidesLive provides a 25% discount for Non-Profit Organizations, please make sure to ask for this if you are eligible to use it.

SlidesLive demo

We “strongly” advise to request a demo session from them “before signing the contract”. You can use part of the ACL 2020 schedule as an example if you do not have the full conference program yet. This is very helpful for you to get a sense of the experience you will have and gives you an opportunity to include your requests from them quite early on. It might take for them some time to include the features you need for your conference and this will give them a reasonable amount of time to address your requests in their framework pre-conference.

Zoom

General suggestions

• (repeating as it is quite important) We strongly suggest enforcing an early deadline for a finalized program of conference and specially workshops. This is very important as it helps to reduce communication overhead for creating all virtual sessions beforehand.
• We strongly suggest generating Zoom meetings well ahead of time to make sure you have enough time to debug and verify them. We have provided detailed documentation on how to create bulk set of meetings in zoom in this documentation. We also have provided another step by step guide for users (authors/hosts of the meetings) here.
• You need to have several volunteers during the conference who are specifically trained for zoom to be on call during the conference in case any incident happens. We recommend at least 2 volunteers per time slot. Note that only these volunteers should have access to your master zoom account.
• Verifying separate accounts need to be crowdsourced pre-conference (it is a pretty quick process), we asked volunteers to do it pre-conference.
• We strongly advise to have a contact person for zoom meetings of each part of conference, e.g., having a couple of POCs for main papers track, demos track, sponsors, etc ahead of time. This will reduce the amount of communication overhead a lot.

We suggest having a separate team for zoom meetings of workshops, there are usually several workshops with a much more diverse set of programs (it is like a mini conference by itself).

• If a meeting got hacked in progress, on-call volunteers should delete the meeting link and create a new link and announce it at rocketchat immediately.
• Each meeting should have a unique main host. If you are ‘host’ for a meeting and start that meeting, and then hop off and try to start another one, you will get an error message from Zoom stating that you already have a meeting in progress and you need to end the previous one in order to start this one. If you click yes, ZOOM WILL END THE PREVIOUS MEETING AND KICK EVERYONE OFF.
• Attendees can join a meeting on browser only when the meeting is not requiring registration.
• Do not share the master account outside of the virtual infrastructure team.
• Note that it is likely that API credentials will expire after a certain amount of time. If this happens, you need to follow the instruction to regenerate it. (Before you run the script to create Zoom Meeting, you need to login to the zoom account and create JWT credentials in Zoom App Marketplace.)
• Virtual infrastructure team can maintain a spreadsheet to store all the credentials of authors, so on-call volunteers will be able to access every meeting and offer help for all the meetings.
• Paid accounts: you can pay Zoom for a business level account and add phone dial-in plans. This is a good idea for anyone who may have a spotty internet, so that they can dial in via phone/audio as well as video -- that way, they will always have audio connection for Q&A and it won’t drop them.
• When a meeting link is broken during the conference, on-call volunteers can login ACL2020 Master Zoom account, and select a specific user from “User Management” -> “Users”, and then create a new meeting for that user (the user will be the host).

Zoom attendees statistics

We made an analysis of Zoom attendees statistics for the main conference Q&A Session. You can also find more detail and the script in the folder below here.

• Total # of Unique Participants in Each Q&A Session:

• Distribution of participants over time 20 minutes after start

• 35 minutes after start

• 50 minutes after start

RocketChat

RocketChat is a platform (similar to Slack) that provides async text-based chat functionality. In our workspace (acl2020.rocket.chat), we can create channels for different purposes.

Channels

Here we provide a summary of channels we have created for ACL2020.

• #announcements: This is a read-only channel which is used by organizers to make conference-wide announcements. Only authorized users can post messages in this channel. Other users can only react with emoji to individual threads.

• #general: This channel is used to discuss general conference topics. The conversation can be about anything related to ACL2020. During the conference, this channel was used to promote their live sessions by authors, workshop organizers, social events organizers, etc.

• #live: This channel is only used for live plenary sessions. It was read-only for the majority of the time. During the live plenary sessions, we lift the read-only setting and people can use this channel for live QA. People have also used this channel to congratulate award winners, etc.

• #firsttimers/#introduction: We initially only created the #firsttimers for people who attend ACL for the first time to say Hi to each other. At the very beginning of the conference, someone created the #introduction channel and many people joined that channel instead to introduce themselves.

• #helpdesk: This channel was used by all ACL2020 attendees to request tech support for the ACL2020 virtual conference. We ask people to prefix their questions with a topic, e.g., [Zoom], [RocketChat], etc. On-call volunteers always start a thread when responding to a question. And continue the conversation in the thread. Once the question is resolved, they will mark it with :checkmark: emoji. See the screenshot below for a demonstration.

• #presenter-helpdesk: This channel is same as #helpdesk, except that it is dedicated to authors and sponsors. It turns out most attendee still ask questions in #helpdesk so this channel may not be necessary in future.

• #incidents: This channel was used to report incidents in the conference. Conference attendees can request the on-call volunteer in the channel to join your Zoom meetings as a co-host to handle disruptions or delete messages in RocketChat. They can simply click “Reply” button to start a private conversation with the on-call volunteer.

• #professional-conduct-committee: This channel was used to contact the Professional Conduct Committee if you need our assistance (e.g. related to a concern that might fall under the anti-harassment policy). PCC members will post a message in this channel to let people know which of them are available. Then the attendee can contact the PCC member via RocketChat private message (similar to the #incidents).

• #social-media-posts: This channel contains cross-posted social media posts published on various platforms by our official live tweeters and microbloggers, managed by our publicity chairs.

• #new-channels: This channel was used to announce new channels made by conference attendees. It turns out many people have created different channels during the conference, including some topic-based channels (eg., #bio-clinical-nlp), university-based channels (e.g., #uwnlp) to get to know different research groups, location-base channels (e.g., #singapore), etc.

• #job-listing: This channel was used to post job opportunities. We only get 10 posts in this channel throughout the conference though.

• #oncall-volunteers: This channel is dedicated to on-call volunteers for them to coordinate with each other during the conference.

• #sponsor-xxx: For each sponsor, we create a dedicated channel for them to post announcements and interact with conference attendees. These are “broadcast” channels (similar to #incidents, #professional-conduct-committee), i.e., only the channel owner can post messages. Visitors can only start a private conversation with the message author. This setting is due to the privacy concerns -- we do not want to expose who has visited a specific sponsor booth.

• #sponsor-registration-desk: This channel is used as the virtual registration desk for sponsors. We need this channel because we need to set the sponsor account as the owner of their channels, and we didn’t develop an automatic way to do this. Since there are only ~20 sponsors, we just ask them to post a message in this channel when they arrive, and we just manually add them to the #sponsor-xxx channel and set them as the owner. After we add the sponsor account to their channel, we post a message “@username Welcome! I have set you as the owner of the channel. You can delete this message with your owner’s permission” so that they can familiarize themselves with the RocketChat usage.

• #organizers-registration-desk: Similar to #sponsor-registration-desk, this channel is dedicated to organizers who need to be the owner of specific channels.

• #paper-xxx: For each paper, we create a channel where the author can interact with visitors to answer questions.

• #keynote-xxx/#workshop-xxx/#tutorial-xxx: For each keynote/workshop/tutorial, we create a channel where attendees can interact with the keynote speaker and the workshop/tutorial organizers.

• #business-meeting/#review-meeting: These two channels are used for discussions about the business meeting and the review meeting, respectively.

Other considerations

• Emoji: We figure this is a way to attract more engagement from conference attendees. We uploaded the ACL logo, together with some other emoji. The ACL logo is a must-have one as we (organizers) usually by default add this emoji reaction to an announcement.
• Bad word filtering: We initially enabled the default bad word filtering in RocketChat during the internal test. However, it turns out it blocks many words such as “queer”, “gay”, etc., and we disable this feature during the conference. It seems to be okay as we didn’t receive any report about profanity in RocketChat.
• Permission to @here and @all: We initially allow all users to @here (mention all online users in the channel) and @all (mention all users in the channel). But it turns out this makes the space very noisy as people have extensively used @here/@all to announce their QA sessions. Therefore, we disable this permission and only allow channel owners to @here/@all.

Other platforms

• Slack might be an alternative to RocketChat, which has a better UI. But it is more expensive. Also, we don’t have the scripts to batch create channels for 800+ papers. Furthermore, we didn’t have time to investigate whether there are some specific features (e.g., broadcast channels) in RocketChat that are not available in Slack. In particular, we need to embed the channel as an iframe on a webpage, which seems to be unsupported natively by Slack.
• Dory.app and Slido could be used in addition to RocketChat for collecting questions & votes. This is perhaps one of the major limitations of RocketChat, i.e., it is not very ergonomic to collect questions/votes, partially because all messages in a thread are also shown in the channel with a smaller font, making it hard to read. AKBC2020 has used Slido, and one of the tutorials in ACL2020 has used Dory.app. Both seem to work well in their scenarios. However, we have tried Dory.app for the live plenary sessions on July 7, and it turns out that their server cannot handle the large amount of traffic and it crashed during the conference. So we have to switch back to the #live channel for collecting questions.

Recommendations

In general, we have relatively good experience with RocketChat. It does have some issues but the support team is very responsive, so I would recommend it for future conferences. The Slack might be a better alternative if it can meet the necessary requirements. I would highly recommend finding a better platform for collecting questions & votes to augment RocketChat. The co-founder of Dory.app (Ena ena.zheng@dory-4098177ab9b4.intercom-mail.com) has reached out to me and she explained the situation. They can easily increase the capacity of their server/database for a specific period if we can provide them an early notice. They are very happy to support the conference, so the EMNLP2020 organizers can directly reach out to Ena to discuss further details.

Virtual Conference Website

The virtual conference website is the central platform to present all information about the conference. As pointed out by ICLR2020 organizers, a portal bringing together all the different tools needed for the virtual conference was not available, and they have developed their own website. Thus, we adopted their codebase Mini-Conf https://github.com/Mini-Conf/Mini-Conf and made some extensions in our own fork https://github.com/acl-org/acl-2020-virtual-conference to meet our specific needs. After the conference, we have removed all RocketChat channels from the website, but you can check out the acl2020-v7.0 release of the repo and spin up the website locally to see the version we used during the conference. We also have an introduction video that explains how to use the website.

Overall design

The website has 12 top-level pages.

• Home: We explain how to use the website and embed the #announcement channel on this page.
• Schedule: This page shows a calendar view of all events during the conference.
• Livestream: This page is used for the livestream of plenary sessions.
• Plenary: This page contains the information about plenary sessions. Each plenary session also has its own page to display the pre-recorded videos and optionally an associated RocketChat channel.
• Papers: This page displays all main conference papers (including the demo papers and student research workshop papers). Users can browse papers by track, search by author/title, and use the visualization tool to find a cluster of papers. Each paper also has its own page, where we display the detailed information about the paper, the live session slots, the pre-recorded video, and the associated RocketChat channel.
• Tutorials: This page displays all tutorials on July 5, using a calendar view. Each tutorial also has its own page.
• Workshops: This page displays all workshops, and each workshop has its own webpage.
• Sponsors: This page displays all sponsors by their levels. Most sponsors have their own dedicated page to display their video, downloadable content, contact, and a RocketChat channel #sponsor-xxx described in Section 3.
• Socials: This page displays all social events, but each event does not have a dedicated page.
• Chat: Here we simply embed the RocketChat workspace.
• Organizers: Here we display all conference organizers with their photos.
• Help: Here we show the FAQ and Code of Conduct.

Tutorials

• We found that each tutorial is different. It would be useful to consult with each tutorial organizer beforehand to understand their needs and customize their virtual tutorial page accordingly. This year we had a common template for all tutorials but in hindsight it could have been slightly different for each.
• Some tutorial teachers did not know that tutorial videos can be recorded in parts (using SlidesLive) and had to record their 2+ hours of videos in one go. We should let tutorial teachers know beforehand that video can be recorded in parts.
• Attendees were not clear on what was expected of them before the tutorial. It would have been useful if each tutorial teacher explained what preparation is required in this virtual setup.
• The tutorial chairs posted a great blog on different modalities of tutorials - https://acl2020.org/blog/detailed-modalities-of-tutorials/. It might have been more useful if posted even earlier.

Workshops

• There should be a separate team of workshop-virtual-chairs dedicated to only thinking about virtual infrastructure needs for workshops. With 20 workshops, some as big as a mini-conference, it becomes important to have someone think about the workshop infrastructure explicitly.
• Each workshop paper should have a page of its own (similar to main conference papers) with link to paper, pre-recorded talk, rocketchat and zoom link for Q&A
• With the new virtual format, each workshop should be asked to write a paragraph explaining their virtual setup. There was a lot of confusion this year since each workshop had a different setup. The paragraph should include
  • Whether they have parallel Q&A session on different zoom links
  • Whether the pre-recorded talks will be played again on the live stream OR do they expect attendees to watch them beforehand.
  • If the pre-recorded talks are going to be played in the livestream, they should ask attendees to stay on the website for the talk part and enter the zoom meeting only after the talk for the Q&A session. This year, many attendees were waiting in the zoom room thinking the pre-recording talks can be seen there as well (not knowing that they should go to the website to watch it)
  • Like the main conference, perhaps the keynote of workshops should also allow questions only on rocketchat. Because some workshop organizers had trouble fielding questions from different sources (zoom chat, zoom hand raising, rocketchat channel)
• It seems like workshops that were single track worked better than those that had multiple tracks and this was because with parallel poster sessions, it was confusing how to go to each paper Q&A session separately.
• Workshops organizers should be given a deadline after which they should not be able to update their schedule. Last minute schedule updates this year caused SlidesLive delay in preparing their livestream session.
• Setting such a deadline will also help infra chairs to create all zoom meetings using individual zoom accounts separately as they did for main conference papers.
• The main zoom meeting should have a dedicated moderator who can take care of assigning speakers to be co-hosts.
• Each workshop should have a separate read only rocketchat channel (similar to main conference #announcement channel) where organizers can make important announcements. This year, with a common channel for all communication, the announcements from organizers were getting drowned within other chat.
• Many attendees had questions about what material will be available before/during/after the conference. The tutorial chairs posted a great blog on this - https://acl2020.org/blog/detailed-modalities-of-tutorials/. It would be great if workshop chairs did something similarly.

Virtual booths for sponsors

• We already have a pretty good template for virtual booths. The hardest part is to communicate with the sponsors and collect information from them.
• In ACL 2020, we only communicate with the sponsors through email. It’s very hard to keep track of the changes sponsors make and sponsors always want to change information at the last minute. It would be great if we could automate some information collection process in the future. Here are some suggestions for future reference:
  • Sponsors always want to change Zoom schedules at the last minute, and it would be inconvenient for them to email us every time they change their schedule. So, it would be great if we could ask them to input the information into a shared spreadsheet, so that they can change their schedule by modifying the spreadsheet. We can write a code to load the spreadsheet to the yml file so that we only need to do a refresh before the main conference, which will significantly reduce our effort.
  • For all the files and resources, it’s better to ask sponsors to provide us a static url link to the file, so that as long as the link does not change, they can update their file anytime they want. We can also ask sponsors to put the <filename, url link> pair into a shared spreadsheet so that we don’t need to manually input the information based on emails.
  • For some files, e.g. logos and pdfs, we can create a shared Google drive for them to update their logos. Again in this ACL we send all files through email which is very inconvenient.
• We should give sponsors a hard deadline on when they can not update the information. For this ACL, we got emails even after the main conference where the sponsors want to change schedules/materials in the virtual booth.
• This year we included the sponsors in the internal dry period, which turns out to be necessary since the sponsors will be able to check and test their virtual booths before the main conference. It’s very important since we can do one round of iteration to fix the information in the virtual booth based on the sponsors’ feedback before the main conference.
• This year we have 19 sponsors and only 2 volunteers to build the virtual booth, so each volunteer needs to build 9-10 sponsors. The workload is too heavy for each of the volunteers (each of them take around 4 days full time). I think the best balance would be 4-5 sponsors per volunteer. There should be one more volunteer that is available during the main conference to fix the website if there are urgent requests from the sponsors.
• We created one admin account for each sponsor that’s valid through the end of the conference, and a test visitor account that’s valid before the conference so that the sponsors can test the interaction between visitors and sponsors (especially on RocketChat). This is very important for the sponsors to learn how to make announcements and how to monitor the channel when visitors send questions.
• It’s important to provide sponsors with a step-by-step instruction on how to login, how to use Zoom, and how to use RocketChat. The direct contact for each module of the virtual booth should also be listed in the instruction, e.g. provide contact for login related issues, Zoom related issues, RocketChat related issues and wrong information in the virtual booth. This way we can significantly reduce the number of repeated questions from sponsors and save a lot of time. Link to the instruction this year: https://docs.google.com/document/d/1paG7hY7E9qKEBZrXZloyfnI3Uxt_723J23JNWBKOMCI/edit
• We should also decide on a date to stop getting new sponsors. It’s not realistic to build new virtual booths at the last minute (which happened a few times for ACL 2020).
• Double check the sponsors’ list with the list on the external website in case there are missed sponsors.

User login

• We basically follow this AWS solution to do the login authentication. But we have used a slightly different version here so we can use the customized auth domain https://signin.acl2020.org. See here for the changes from the original version in the blog post.
• This solution allows us to use OAuth to implement the sign-on for the website and RocketChat. See here for instructions.
• By default, the AWS Cognito pool can only send 50 emails per day. Need to follow this instruction to increase the limit and this instruction to switch to use the Amazon SES.
• For people who registered before July 3, we create the accounts on July 3 using the excel file provided by Prisicilla.
• To handle registration during the conference, we use Zapier + AWS Lambda. See https://github.com/acl-org/acl-2020-virtual-conference/issues/55#issue-636817449 for more details.

Although this solution works reasonably well, it does have some issues. Actually, 95% questions sent to acl2020-virtual-helpdesk@googlegroups.com are about login problems, and we probably have handled around 200 such requests.

• The AWS Cognito sent out login emails on behalf of acl2020virtual@gmail.com. It turns out the emails are easily classified as spam. Also some university email server blocks it (e.g., we found consistent issues with @xxx.i.kyoto-u.ac.jp).
• Many people could not login because they have entered the wrong password. We have therefore updated several versions of our instruction https://acl2020.org/_pages/docs/ACL2020_virtual_website_login_steps.pdf with some common errors (see Step 1), which seems to be helpful.
• We announce that our website will be live on July 3, and we send out the login credentials on that day. It turns out to be a not very wise decision because we have to deal with all incoming inquiries about login issues without volunteers on July 3 & 4. (Our on-call volunteers can only start working on July 5).
• Some authors are confused with the account used for the website and the SoftConf account.
• For some unknown reason, there are many users who have registered but were not included in the excel files sent from Priscilla. We may need to check with the company who hosted the registration form about this.
• It would be helpful if we can have the registration company directly handle the login authentication, so that we don’t need to have two separate processes to create user accounts.

General Advice for Future Virtual Conferences

• It would be nice if future conference organizers can find a vendor company to run the virtual conference if possible. Otherwise, the virtual infrastructure committee would definitely need much more people and figure out a way to distribute the work more. Since all organizer committee members are officially volunteers, we cannot rely on people to work full-time or even extra hours on this (which we have to do for ACL2020 given the limited resources and short timeline). Here is a potential organization of the virtual infrastructure committee.
  • Pre-recorded videos & livestreaming (1-2 chairs + a vendor company)
  • Zoom meetings (1-2 chairs + 5-8 volunteers)
  • Captioning (1-2 chairs)
  • RocketChat (1 chair + 3-4 code development volunteers)
  • Main website development (1-2 chairs + N code development volunteers)
  • Virtual sponsor booth (1 chair + 1 code development volunteers per 5 sponsors)
  • Tutorials (1 chair + 2-3 code development volunteers)
  • Workshops (probably better to have their own virtual infrastructure chairs to customize their webpages)
  • Login authentication (1 chair + 2 code development volunteers + 5 helpdesk volunteers; or use a vendor company)
• The repo https://github.com/acl-org/acl-2020-virtual-conference should be a good start for future conferences. Due to the time limit, we didn’t maintain a very high code quality but it should not be very hard to spin up a new website. The most challenging part would be gathering all sitedata, including the paper basic information, QA session schedule, Zoom links, video URLs, etc.
• It would be useful to have a better pipeline for information collection. Currently we have to gather information from different chairs. It would be better if the website, the proceedings, the handbook, and the ACL Anthology all use the same information source such that everything is synchronized.
• GitHub Issue seems to be a good way to manage the tasks and distribute work to volunteers.
• It would be better to have volunteers to be involved earlier, and work with the volunteer coordinator on the timeline. It was not a very smooth transition because most volunteers can work either before or during the conference, not both. And there is a period that our chairs have to handle all emails in acl2020-virtual-helpdesk@googlegroups.com on our own on the first two days when there are lots of inquiries.
• There is many other useful information in the article “Gone Virtual: Lessons from ICLR2020". The section “Technical Systems and Support” is quite relevant to the virtual conference infrastructure.