2015Q3 Reports: Student Research Workshop Chairs

Organizers

The student co-organizers were Kuan-Yu Chen (National Taiwan University), Angelina Ivanova (University of Oslo), and Ellie Pavlick (University of Pennsylvania, USA).

The faculty advisors were Emily M. Bender (University of Washington), Chin-Yew Lin (Microsoft), Stephan Oepen (University of Oslo)

Mentoring programs

We offered two mentoring programs this year. Pre-submission mentoring is available to all authors who want feedback before submitting papers. In-person mentoring is available to authors of accepted papers, to improve their presentation and their networking opportunities while at the workshop and the main conference.

Pre-submission mentoring

We followed last year’s SRW in offering an optional round of pre-submission mentoring. This was designed to give students an opportunity to improve their submission, particularly the writing and presentation of the paper, before submitting their papers to the workshop for review. Authors who submitted their papers in time for the the pre-submission deadline received feedback from their assigned mentors before the final submission deadline, giving them time to integrate the mentor’s feedback into their final submission. We had 14 papers submitted to the pre-submission mentoring program, up from 6 submitted to last year’s workshop. We had 16 volunteers to be pre-submission mentors. We manually assigned one mentor to each submitted paper (2 mentors were not assigned any papers). Mentoring was not anonymous. Mentors were given the email address of the corresponding authors for the paper, and instructed to communicate with the author directly.

In-person mentoring

Each accepted paper is assigned a mentor who will attend the workshop in person. Mentors are experienced researchers (faculty and postdocs) who are asked to read the papers in advance of the workshop, to attend the oral and poster presentations, and to ask focused questions during the presentations. Since we had 11 mentors volunteer and only 7 accepted papers, we chose to assign two mentors to each of 4 the research papers and a single mentor to each thesis proposal.

Commentary/Notes to next year’s organizers

For both mentoring assignments (pre-submission and in-person), we assigned papers to mentors manually. This was manageable given the small number of submissions, but in future years, especially if the number of submissions grows, it would be better to use a formal bidding system as we did when assigning reviewers to papers. This year, the pool of reviewers was separate from the pools of mentors (i.e. no reviewer was a mentor and no mentor was a reviewer). This was mostly to avoid problems with objectivity and anonymity in the reviewing. The recruiting would have been more manageable had we recruited volunteers of all three types (reviewer/pre-sub. mentor/in-person mentor) using a single invitation, allowing volunteers to chose how they wanted to participate in the SRW, and then used conflicts-of-interest during bidding to manage problems with objectivity.

Submissions

Submission procedure

As in the 2014 ACL Student Research Workshop, this year’s SRW consisted of two submission tracks: research papers and thesis proposals. Research papers were intended to encompass completed work as well as works-in-progress from junior graduate students, masters students, and advanced undergraduates. Thesis proposals were for intended to be a venue for senior graduate students to get feedback on their thesis proposal and the broader ideas surrounding the appropriateness and impact of their chosen topic. The limit for both types of papers was 6 pages of content and any number of pages for references. Research papers could contain any number of authors as long as the first author was a student, and had to be completely anonymized. Students who had already presented a research paper at a previous ACL/EACL/NAACL SRW were not allowed to first author another SRW research paper submission, but could still submit to the Thesis proposal track. Thesis proposals had to be single-authored by a student. Thesis proposals were allowed to reference the author’s own prior work and had to be submitted along with an up-to-date CV, and so were not required to be anonymous.

Number of submissions

We received 18 valid submissions for review (23 submissions initially, 5 of which were later withdrawn). 5 of the submissions were thesis proposals and 13 were research papers.

Accepted papers

We accepted 3 thesis proposals (60%) and 4 research papers (31%) leading to an acceptance rate of 39% overall.

Commentary/Notes to next year’s organizers

The distinction between the two tracks should have been made more clear to the submitting authors. Many reviewers of Thesis proposals complained that the scope was too narrow, focusing on a single solution to a single problem/application rather than motivating a broader area of research. The anonymization of the Thesis proposals was also an area of confusion for both students and reviewers. The idea was that the review of a Thesis proposal should take into account the appropriateness of the proposal given the author’s particular background and experience, and therefore should be considered alongside the student’s CV and prior publication record. However, this intention was not clear, so many students did not appropriately design their proposals to highlight their own prior work/planned future work, and many reviewers did not considered or mention the CV during their review. The acceptance rate was lower than in previous years (last year was 50% overall), due to a large number of negative reviews. We tried to keep up the acceptance rate while not reducing the standards of the workshop. In general, when reviewers’ consensus score for “overall impression” was below 3, we did not accept the paper. However, we read through all of the reviews ourselves, and made a few exceptions when the low scores seemed more due to writing quality/clarity than to the scientific/research content.

Reviewing

Program committee

Our committee comprised of 47 reviewers, 42 from academic institutions and 5 industry researchers. Each paper was assigned 3 reviewers. No reviewer was assigned more than 2 papers.

Commentary/Notes to next year’s organizers

Of the 152 invited to be part of the program committee, 47 (roughly 30%) agreed to review for the SRW. For many who agreed to review, the promise of no-more-than-two papers was an important condition of their participation.

Format

Timeline

Our timeline was as follows: 10-12-2015: First CFP sent out 11-02-2015: Second CFP sent out (reminder for pre-sub deadline) 24-02-2015: Submission deadline for pre-submission mentoring 10-03-2015: Authors receive feedback from pre-submission mentors. 23-03-2015: Final CFP send out (reminder for final deadline) 07-04-2015: Final submission deadline 09-04-2015: Bidding begins 15-04-2015: Bidding ends, reviewing begins 14-05-2015: Reviews due 18-05-2015: Notification of acceptance 27-05-2015: Camera ready deadline

Format of the workshop

This year, all of the accepted papers will be presented as posters alongside the main conference short paper posters on the second day of the conference. In addition, authors will each give a short 5 minute oral presentation to advertise their work. The oral presentations will take place in their own session from 4:30-5:30pm on the second day, immediately preceding the poster session. We request that mentors attend the oral presentation and prepare a question to follow their mentee student’s talk. We also ask that the mentors visit their mentee’s poster during the poster session.

Commentary/Notes to next year’s organizers

We had to adjust our timeline several times during the process. In order to increase participation, we extended the pre-submission mentoring deadline by two weeks (it was originally February 10). Again, due to a low number of submissions for the final deadline, we extended the deadline for final submissions by one week (it was originally scheduled for March 31). As a result, we shifted the reviewing deadlines accordingly. We had originally promised to give final acceptance decisions on March 15th, although reviews were not due until the 14th. This did not provide enough buffer period, and many reviews were late or missing. We had to push the acceptance notification deadline until the 18th, but this caused extra stress for many of the authors. In the future, we would be conscious to allow enough time between the review deadline and the notification of acceptance to accommodate these kinds of delays/errors.

Funding

Once student chairs had been appointed, the main role if the faculty chairs was to secure funding to contribute to the cost of SRW participation by presenters. The three main sub-tasks in this effort were to (a) understand which types of funding are traditionally available and how the flow of funds is organized; (b) submit a proposal to the National Science Foundation (NSF) to support SRW participants coming from the US; and (c) monitor the anticipated need of funding and approach candidate industry sponsors. Initially, there was some mis-understanding about left-over funds from the ACL 2014 SRW, which we expected to be able to draw on. The 2014 faculty chairs eventually clarified that all 2014 funds had been dispersed (in part by supporting student presenters in the regular conference), and that unused funding from previous years actually had been left over from 2012, but that this money had in the meantime been earmarked to support the student research workshop at NAACL 2015. Support to SRW tends to come from several sources, of which some run directly through the ACL (e.g. company donations), while in particular the traditional NSF grant to support US-based students needs to go through a US university, typically the home institution of one of the SRW faculty chairs. The distribution of funds to SRW presenters is handled by the ACL (Priscilla Rasmussen), and the typical model appears to take the form of a stipend (rather than reimbursement of documented expenses), where presenters are offered a lump sum to contribute to actual cost of participation, and it is sufficient for presenters to make plausible that they actually had legitimate expenses on at least that order. Seeing as travel to and participation in the SRW at ACJ-IJCNLP will be somewhat costly for all student authors of accepted papers (three from Europe, two from Japan, and two from the US), we decided to offer each presenter a stipend of (up to) US$ 1,500. Furthermore, the same offer was made to the SRW student chairs, but it appears that those who participate in ACL-IJCNLP 2015 had other sources to cover their cost of participation. Given earlier mis-communication about available funds, the anticipated need for NSF funding only became clear in late April. Emily M. Bender then submitted a grant proposal to the NSF for six stipends and a contribution to the cost of publishing the proceedings and holding the workshop (which appears to be a traditional element in SRW support from the NSF); with some good will on the side of the NSF programme officer, the proposal was approved for funding in about four weeks. Future SRW faculty advisors should be aware that the typical time for submission of a proposal of this type to NSF is during the month of January. In addition to securing the NSF contribution, faculty chairs sought to line up additional donations from large corporations. But as almost all of the plausible contacts already were official ACL-IJCNLP 2015 sponsors, we did not succeed in securing additional funding. We did not push very hard, either, as it was clear at the time already that there would be a comparatively small number of presenters and sufficient available funds from existing sources. Stephan Oepen discussed with the EACL executive board the possibility of supporting SRW student participation, and the EACL is open to this in principle. However, as sufficient funding for all SRW presenters at ACL-IJCNLP is available already, we see no need to invoke this mechanism for 2015.