2020Q3 Reports: Captioning and Sustainability Co-Chairs

Lessons Learned: ACL 2020 Captioning Co-Chairs

Overall summary

• Providing captions to conference presentations, Q&A sessions, panel discussions etc. makes the conference more accessible – not only for attendees who are hearing impaired but also for attendees who may have some difficulty following speakers who speak fast and/or whose native language is not English
• Automatic captioning (essentially: ASR) does currently not seem at a level of quality that is required for providing such accessibility and assistance to conference participants; reasons include problems recognizing technical terms and proper names, handling foreign accents etc. While it may be possible to adapt existing systems (e.g. by training it on the ACL anthology), realistically, both manual corrections/editing, as well as some level of human live captioning is likeley needed for some time to come.
• ACL 2020 provided captioning for the majority of plenary sessions, including live Q&A plenary sessions, for all video-recorded presentations of papers (some automatic, some edited by presenters), but not for other live Q&A sessions.
• The video hosting company was SlidesLive – they clearly struggled with processing 1500+ video presentations in just around 2 weeks; they also changed their commitment to what they were planning to deliver in terms of captioning multiple times during the pre-conference period which negatively impacted planning for captioning.
• Live human captions (CART) were provided by ACS Captions: their service was outstanding, they have a high level of technical expertise, and the cost is reasonable (~$90/hour) given their high accuracy and almost simultaneous live captioning capabilities
• Many authors chose to edit the automated captions provided by SlidesLive despite a very short notice just a few days before the start of the conference; this resulted in their presentations having high quality captions for the conference participants.

Specific suggestions

1. Time line:
   • Paper authors had to upload their presentation videos by June 17; SlidesLive promised processing by June 30 but was still processing presentations 2 days later, so that some authors had to edit their captions within a very short time (or were running late, compared to when their presentations were viewed by conference participants).
   • We recommend asking authors to upload their video presentations around 1 month before the start of the conference, in case a service such as SlidesLive is used that semi-automatically processes all prerecorded presentations and adds captions to them.
   • Authors should have at least 1 week before the start of the conference to edit automatically generated captions for their presentations.
   • All of these services by the hosting company – video upload, caption download/video viewing, edited caption upload etc. – should be tested before a contract is signed to ensure that the respective company has the capability to deliver the services and has sufficient time to make suggested changes, e.g., to their platforms they use for uploading/downloading of files etc. (e.g. there were issues with access rights; easy search functions for authors; organization of files etc.)

   
   

2. Reaching out to presenters:
   • Since ACL is organized in multiple strands: main conference, demos, tutorials, student research workshop etc., it was cumbersome and difficult to quickly reach out to all ACL presenters, e.g., in order to inform them about the caption editing process.
   • We recommend that a conference establish a list of all presenters (mailing list) that can be easily used for purposes of communication that are pertinent to all presenters and that should go out via email and not just as blog post (e.g., the PCs would have sole access to this list to avoid spamming).
   • Currently, the option provided by SlidesLive to let the presenters download and upload captions was just through a Google Drive. This made it difficult for the presenters to find their caption files, as well as not being very secure. We recommend that SlidesLive either establish a portal for presenters to securely upload and download caption files for their talk, or take on the responsibility of personally emailing captions to presenters, which was not possible given the tight timeline.
3. Contract with CART company:
   • We recommend using services by ACS Captions as they are highly professional and reasonably priced.
   • We also recommend using a contract model that asks for a fixed number of hours for CART services (e.g., covering plenary live sessions) and makes it possible to add additional hours of service after the registration deadline; that way, any participants who express a need for such
   • We also recommend that the CART service provider interact early with the video hosting company or whoever provides the interactive online meeting spaces to ensure a seamless integration of CART services.
4. Automated captioning:
   • Depending on the platform used, it may make sense to investigate multiple automated captioning systems; priority may be given to a service that allows for adaptation of their ASR language model, e.g., by processing the ACL anthology.
   • As stated above, even the best of automatic services is not likely to meet high standards of accuracy expected by ACL participants who need to rely predominantly on such captions for following presentations or live Q&A sessions.