2020Q4 Reports: Student Research Workshop Chairs

The AACL-SRW is the youngest sibling of ACL-SRW, EACL-SRW, and NAACL-SRW, and was successfully held in conjunction with the first Conference of the Asia-Pacific Chapter of ACL (AACL-IJCNLP 2020).

The workshop followed previous SRWs, offering pre-submission and post-accept mentoring programs. 47 papers were submitted, representing a diverse range of NLP topics, as well as student backgrounds, languages and locales. 24 papers were accepted (51% accept rate), and were presented in 6 sessions.

In addition to following the SRW traditions, we also introduced new program features. We believe these features add to the workshop in terms of value, impact, visibility, and attractiveness. The features include:

Keynote speech
Oral presentations
Best paper award
New review form

The mentorship program is at the core of the SRW. To help in assessing the effectiveness of the SRW and improving the offering, we also conducted post-workshop online anonymous author and mentor feedback surveys. The results are available for future SRW chairs.

Top Recommendations

Carry over new program features ( (keynote, best paper award, oral presentations) to future SRWs
Consider the use of the new SRW review form
Conduct post-workshop surveys, share with future chairs to keep improving SRWs
Expand SRW to high school students and explore the possibility of an ACL high-school outreach program
Investigate START's "Multi Track Conferences" for easier SRW management

Introduction

The AACL-IJCNLP 2020 Student Research Workshop (SRW) was held in conjunction with the AACL-IJCNLP 2020 conference. The SRW gives student researchers in Computational Linguistics and Natural Language Processing the opportunity to present their work and receive constructive feedback and mentorship by experienced members of the ACL community. Previous SRW editions were held in conjunction with ACL, NAACL, and EACL. As AACL-IJCNLP 2020 was the first conference of the Asia-Pacific Chapter of the ACL, this SRW was also the first Asia-Pacific edition of the Student Research Workshop.

The workshop was held virtually on December 7, 2020.

This report discusses the various aspects of organizing the SRW at ACL 2020, including:

Mentoring Programs
Submissions and Reviews
Website and Social Media
Virtual Conference
Recommendations

Organizers

The student co-chairs were:

Yin Jou Huang (Kyoto University, Japan)
Boaz Shmueli (Academia Sinica, Taiwan)

The faculty advisors were:

Lun-Wei Ku (Academia Sinica, Taiwan)
Vincent Ng (University of Texas at Dallas, USA)

Mentoring Programs

Following previous SRWs, we provided two mentoring programs for participants. The first program was the pre-submission mentoring program which offered authors the opportunity to get feedback prior to submitting their papers for review. The second program was the post-submission mentoring program in which a mentor was assigned to each accepted paper to help the student authors with the preparations of camera-ready submission and the presentation materials.

Pre-submission mentoring

We followed previous SRW in offering an optional round of pre-submission mentoring. This was designed to give students an opportunity to improve their submission, particularly the writing and presentation of the paper, before submitting their papers to the workshop for review. Authors who submitted their papers for the pre-submission received feedback from their assigned mentors one week before the final submission deadline, giving them time to integrate the mentor’s feedback into their final submission. A total of 11 papers participated in the pre-submission mentoring program. We recruited 11 mentors, all of whom are well-experienced researchers in the field, for the pre-submission program. Papers were matched to mentors based on research areas. The mentoring feedback was sent to authors anonymously through the SRW organizers. Also, under mutual consent of the participants, the authors and the mentors can also communicate directly via email or private Slack channel.

Post-accept mentoring

We also offered the post-acceptance mentoring program. After the acceptance notice, each accepted paper was assigned a post-acceptance mentor to provide feedback to the authors about their camera-ready drafts and conference presentations. 24 mentors were recruited, i.e., one for each accepted paper.

Grammarly Premium

During the post-acceptance mentoring programs, writing assistance in the form of vouchers for one month's free use of Grammarly Premium were sent to all participating authors with the help of the D&I committee.

Submission and Reviewing

Timeline

Pre-submission mentoring deadline: August 28, 2020
Pre-submission mentoring feedback: September 18, 2020
Paper submission deadline: September 25, 2020
Acceptance notification: October 23, 2020
Camera-ready deadline: November 6, 2020

Submission procedure

This year's submissions were organized into two submission tracks: research papers and thesis proposals. Research papers describe completed work, or work in progress with preliminary results whose topics are inline with the topics of interest of the AACL-IJCNLP 2020 main conference. Thesis proposals are intended for advanced students who have decided on a thesis topic and wish to get feedback on their proposal and broader ideas for their continuing work. The page limit for both types of submissions was 5 pages of content, with unlimited pages for references. For the camera-ready drafts of accepted papers, one additional page was permitted, thus upping the page count limit to 6.

Submissions were managed through the START conference system.

Submissions and Authors

We received 47 submissions in total: 9 thesis proposals and 38 research papers. We accepted 24 papers, including 22 research papers and 2 thesis proposals, resulting in an acceptance rate of 51%.

Submission came from a very diverse group of authors in terms of geography (Asia, Europe, America, Australia) and educational qualification. In particular, we were delighted to receive a submission from students of a top STEM high school in the US. Through our interaction with these students, this may be an indication that there is further interest in CL/NLP among high school STEM students. We recommend expanding SRW CFP language to specifically address also high school students.

Based on our experience, we recommend exploring the possibility of a school outreach program for CL/NLP, as this has the potential to benefit the ACL community. Similar programs exist for other professional societies, such as the IEEE CIS High School Outreach Subcommittee.

Program committee

We were fortunate to recruit 132 members for the SRW Program Committee. Each paper was assigned 4 reviewers. No reviewer was assigned more than 2 papers. Each paper had at least 3 reviews. The average number of reviews per paper was 3.5.

Reviewing and Review Form

We used the “explicit bidding” capability on the START system, where reviewers could indicate their preferences for reviewing each paper. Submissions were then automatically assigned to reviewers based on these bids.

We believe that the existing review forms used by *ACL conferences are unnecessarily complicated and can introduce ambiguity for the reviewer. We thus decided to introduce a new review form for SRW, with the following goals in mind:

Reduce unnecessary reviewer load
Easy to understand
Unambiguous
Constructive feedback for authors
Can be easily adapted to other workshops/conferences

The main features:

Only two text boxes:
Paper summary (info for the chairs)
Feedback (info for the authors)
Use of the familiar Likert agree/disagree scale for the various criteria
Four paper criteria: Generalizability, Interestingness, Structure (in addition to Readability), Ethical Data Collection.

The review form is available at https://aacl2020-srw.github.io/pdfs/aacl2020_srw_review_from.pdf

To evaluate the effectiveness of the new review form, we included the following Likert-scale statement in the review form itself "Overall, I found this review form to be useful and easy to understand". Of the total 164 votes, 142 of the reviews had either "Agree" (4) or "Strongly Agree" (5) while only 7 chose "Disagree" (2) or "Strongly disagree" (1). The remaining 15 votes went to "Neither agree nor disagree" (3). We thus conclude that the new review form was highly effective and recommend using this form in future SRWs.

"Multi Track Conferences" for Easier Management

SRW management requires the handling of three committees: pre-submission mentors, submission reviewers, and post-accept mentors. As in previous years, we managed the pre-submission and post-submission programs via spreadsheets, emails and online forms. We found the process to be tedious and to require a lot of manual work. We believe that one possibility to ease the management of the three programs in future SRWs is to use START's "Multi Track Conferences", and use a different "track" for each of the programs. Although we did not test this option, we believe that this option is worth exploring.

Website and Social Media

We created a website (https://aacl2020-srw.github.io), which was the primary source of information about the SRW. The website includes information regarding important dates, author guidelines, workshop program and list of organization members. Additionally, we managed a Twitter account (@AACL_SRW) for posting updates to social media. The Twitter account is mainly used to convey up-to-date information of the SRW and to improve the overall reachability. The call for papers was publicized through the ACL mailing list as well as the Twitter account.

We note that since Twitter is blocked in China, this limited the effectiveness of our social media outreach as we did not run any social media accounts in China (WeChat/Weibo/etc). This issue also applies to other conferences. To expand the outreach of the workshop to geographies where Twitter is not accessible, we recommend close coordination with the main conference publicity chair.

Virtual Workshop and Proceedings

Due to the COVID-19 pandemic, the workshop was held online in conjunction with the AACL-IJCNLP 2020 main conference (AACL-IJCNLP 2020, AACL-IJCNLP 2020 Virtual Conference). The Zoom platform was used for the virtual meeting. All accepted papers were included in the SRW proceedings and presented orally (either live or by recording) at the online workshop on December 7, 2020.

Keynote Session

We introduced a new program feature to SRW: a keynote talk. We invited Prof. Emily M. Bender (University of Washington) to give the inaugural SRW keynote. The title of the talk was “Societal Impacts of NLP: How and When to Integrate Them into Your Research (and How to Make Time for That)”. The talk was followed by a Q & A session. We received very positive feedback about the talk, e.g. one member noted that the talk was one of the highlights of the AACL. The tweet announcing the keynote speech gathered ~55,000 views by Dec 15, 2020.

Oral Presentations

To our knowledge, this is the first SRW with oral presentations, as previous SRWs used the poster session format. We believe that this added a unique and important experience to the students and an important opportunity to hone their oral presentation skills in a friendly atmosphere. The papers were arranged into six sessions according to topic. Each presentation consisted of an ~8 minute live talk following a short live Q&A. We also collected pre-recorded videos from the authors. A pre-recorded video was used in case that the presenter cannot deliver the talk live or due to technical difficulty. We had six session chairs, one for each session.

Best Paper Award

As a new feature of SRW, we introduced a Best Paper Award into the program. We used the following process to select the best paper: first, we narrowed down the list of 24 accepted papers to a list of eight candidates by compiling information from the paper reviews, and then further down to a shortlist of four papers. These four papers were shared with the Best Paper Award committee. This seven-member committee was composed of the student chairs, the student advisors, and three additional members. Each member was asked to rank the four papers from best to worst. As a result of the feedback, we decided to award both a Best Paper and a Runner Up. The results were announced at the closing session and also on social media, and attracted a lot of attention (the corresponding tweet gathered ~18,000 views by Dec 15, 2020). The winners were also announced on the workshop website.

The new program features add to the workshop in terms of value, impact, visibility, and attractiveness. We recommend carrying them over to future SRW editions in all regions.

Post-Workshop Feedback Surveys

To assess the effectiveness of the mentoring program, we ran two post-workshop surveys: one survey for authors, and one for mentors. We received responses from 15 of the mentors and 15 of the authors. The full results of the surveys are available below.

We believe that this feedback can be used to improve future SRWs and we suggest running similar surveys at the end of each SRW.

Easier Communication

Post-acceptance mentoring was conducted by connecting the authors with the mentors via email, with encouragement to the authors to send their camera-ready drafts and presentations. While we received feedback from multiple authors that this program was very useful to them, some mentors informed us that the authors never communicated with them. Future SRWs can consider a more formal system of sending camera-ready drafts to the mentors and facilitating discussions between mentors and authors.