2017Q3 Reports: EACL 2017
EACL 2017 Conference Report
Introduction
Mirella Lapata (General Chair)
EACL 2017 was the largest ever EACL in terms of the number of papers being presented. The scientific program, included 14 workshops, six tutorials, a demos session, and a student research workshop. EACL received a record number of sumbissions this year, approximately 1,000 long and short papers combined, which reflects how broad and active our field is. The conference had three invited speakers: David Blei (University of Columbia), Devi Parikh (Virginia Tech), and Hinrch Schütze (LMU Munich).
The Program Committee Chairs, Alexander Koller and Phil Blunsom put together a team of 27 area chairs who in turned assembled many reviewers and handled a large number of papers. The Workshop Chairs, Laura Rimmell and Richard Johansson, coordinated with the workshop chairs for ACL 2017 and EMNLP 2017 and succeeded in putting together an exciting and broad programme including 14 workshops. The student research workshop was organised by the student members of the EACL board -- John Camilleri, Mariona Coll Ardanuy, Uxoa Iñourrieta, and Florian Kunneman. With the help of Barbara Plank (Faculty advisor), they issued the call, organised a team of reviewers, assigned papers, coordinated and mediated among reviewers, and finally constructed a schedule consisting of 12 papers. The Tutorial Chairs, Lucia Specia and Alexandre Klementiev, put together a very strong programme of six tutorials, all well attended.
The publication chairs, Maria Liakata and Chris Biemann undertook the complex task of producing the conference proceedings. The conference also show cased 30 demos, selected by the Demo Chairs Anselmo Peñas and André Martins. David Weir, our publicity chair, actively advertised conference announcements in appropriate vernues. And the ACL business manager, Priscilla Rassmussen, offered advice on many issues ranging from invited speakers to the conference handbook.
The local organising committee worked tremendously hard to make EACL 2017 a success. The Local Chair, Paolo Rosso and Andrea Aldea from Groupo Pacifico, brought together a fantastic local team and dealt with many of the day-to-day tasks arising in organizing such a large conference expertly and efficiently. They also designed and printed the conference handbook. Several people commented on how good the organisation was and enjoyed the conference dinner and fireworks!
Program Chairs report
Alexander Koller and Phil Blunsom
Innovations
As compared to previous EACL conferences, this year's main innovations were:
- We selected four “outstanding papers” rather than two best papers.
- The areas were reorganized into a smaller number of more coarse-grained areas.
- Instead of 20-minute talks, long papers were presented in 15-minute talks, plus 5 minutes for questions and changing speakers. Short papers were given 12 minutes plus 4 for questions and transition (similar to ACL 2016).
- We introduced a new area labeled “Other” to handle papers that did not fit into traditional areas, as well as a few COI papers (similar to ACL 2016).
Rationale
Outstanding papers were added because the community is growing and so is the number of papers at each conference, including the number of papers that deserve recognition as being of particular importance and quality.
The areas were reorganized because some of the areas of EACL 2014 were rather fine-grained and received fewer submissions than others. With the more coarse-grained areas, we managed a more balanced reviewing load per area, while still ensuring the necessary expertise among the area chairs.
The “Other” area was introduced to give papers in non-traditional research areas a better chance at fair review. Rather than conventional bidding, the “Other” area chairs were given access to the reviewer pool directly (while bidding was taking place for papers in the rest of the areas), so that these papers were given the first chance at the most appropriate reviewers.
The shortened talks were necessary to squeeze the considerably larger number of accepted papers (compared to earlier EACLs) into the program, while avoiding a fourth parallel track.
Submissions and Presentations
EACL 2017 received a total of 1043 valid submissions, of which 488 were long papers and 555 were short papers. 6 long papers and 37 short papers were rejected without review due to non-anonymity or formatting issues. 40 long and 14 short papers were withdrawn before reviewing was completed. The remaining 946 submissions were each assigned to the 15 areas, and managed by a program committee of 27 area chairs and 624 reviewers. 119 long papers (27%) and 120 short papers (24%) were selected for presentation at the conference. Of the accepted long papers, 78 were selected for oral presentation; of the accepted short papers, 47 were selected for oral presentation; the rest were presented as posters. The oral versus poster decision was made mainly based on the recommendations of reviewers, filtered by our judgment of the generalized appeal of each paper. In addition, four TACL papers were be presented at EACL, all as talks.
The table below shows the number of reviewed submissions in each area for long and short papers, as well as the number of papers accepted in each area.
| area | long reviewed | long accepted | short reviewed | short accepted | total submissions | percentage of total submissions | total accepted | percentage of total accepted | area acceptance rate | outstanding papers |
|---|---|---|---|---|---|---|---|---|---|---|
| Semantics | 44 | 17 | 66 | 18 | 110 | 11.6% | 35 | 14.6% | 31.8% | 1 |
| Information extraction, text mining, and question answering | 64 | 16 | 51 | 10 | 115 | 12.2% | 26 | 10.9% | 22.6% | 0 |
| Social media, sentiment analysis, and opinion mining | 53 | 9 | 59 | 10 | 112 | 11.8% | 19 | 7.9% | 17.0% | 0 |
| Machine translation and multilinguality | 39 | 9 | 62 | 14 | 101 | 10.7% | 23 | 9.6% | 22.8% | 0 |
| Syntax, parsing, grammar formalisms, and grammar induction | 21 | 10 | 22 | 7 | 43 | 4.5% | 17 | 7.1% | 39.5% | 2 |
| Phonology, morphology, word segmentation, tagging, and chunking | 21 | 5 | 31 | 5 | 52 | 5.5% | 10 | 4.2% | 19.2% | 1 |
| Document analysis including text categorization, topic models, and retrieval | 42 | 10 | 51 | 13 | 93 | 9.8% | 23 | 9.6% | 24.7% | 0 |
| Machine learning | 30 | 10 | 27 | 7 | 57 | 6.0% | 17 | 7.1% | 29.8% | 0 |
| Resources and evaluation | 47 | 11 | 43 | 11 | 90 | 9.5% | 22 | 9.2% | 24.4% | 0 |
| Discourse and pragmatics | 11 | 2 | 16 | 3 | 27 | 2.9% | 5 | 2.1% | 18.5% | 0 |
| Generation and summarization | 21 | 5 | 26 | 5 | 47 | 5.0% | 10 | 4.2% | 21.3% | 0 |
| Dialogue, interactive systems, and speech | 21 | 6 | 25 | 9 | 46 | 4.9% | 15 | 6.3% | 32.6% | 0 |
| Vision, robots, and other grounding | 6 | 0 | 3 | 1 | 9 | 1.0% | 1 | 0.4% | 11.1% | 0 |
| Cognitive modeling and psycholinguistics | 7 | 2 | 8 | 3 | 15 | 1.6% | 5 | 2.1% | 33.3% | 0 |
| Other | 15 | 7 | 14 | 4 | 29 | 3.1% | 11 | 4.6% | 37.9% | 0 |
| Total | 442 | 119 | 504 | 120 | 946 | 100.0% | 239 | 100.0% | 25.3% | 4 |
The top five areas with the highest number of submissions this year were Information extraction, text mining, and question answering; Social media, sentiment analysis, and opinion mining; Semantics; Machine translation and multilinguality; and Document analysis including text categorization, topic models, and retrieval.
Review Process
The page limit was 8 pages for long paper submissions and 4 pages for short paper submissions (each with unlimited pages for references). Camera-ready versions were given one additional page: 9 plus ∞ for long papers, 5 plus ∞ for short papers.
We opted for a “traditional” model of assigning reviewers to areas, in that the chairs of each area could invite reviewers to their area. After the submission deadline, we asked the area chairs to flag papers that they felt should be in a different area. We then went through the list of suggestions and reassigned papers as needed. We also assigned all papers where an area chair had a conflict of interest to the “Other” area.
Despite the high number of submissions, load balancing across areas was not a major issue for us. This may be because we extrapolated the required number of reviewers from the EACL 2014 and ACL 2016 data rather generously. Some areas ended up with reviewers who were not assigned any papers, and no areas were short on reviewers. The chair for the “Other” area was given access to the entire reviewer pool, and got to choose reviewers one day before the other area chairs.
We asked the area chairs to provide brief meta-reviews of all papers that were not a clear accept or reject. We found this to be extremely helpful in deciding on borderline papers.
Outstanding and Best Papers
The area chairs nominated 12 papers (7 long and 5 short) from 9 areas as outstanding papers. Out of these, the paper awards committee selected four outstanding papers (3 long and 1 short). These were presented in semi-plenary talks (two parallel sessions) at the end of the conference.
It was noted by members of the community that all 15 authors of the outstanding papers were male. Indeed, the proportion of papers with male first authors grows steadily as we move from submitted papers to accepted papers to papers that were nominated as outstanding to the outstanding papers, as the following table indicates:
| long | short | |
|---|---|---|
| submitted | 76.7% | 72.1% |
| accepted | 78.3% | 76.9% |
| nominated for outstanding | 85.7% | 80.0% |
Given the small overall number of outstanding papers, the fact that all outstanding papers were authored by men may be a statistical accident. However, the proportional loss of papers with female first authors from step to step is disconcerting. It may make sense to track this data in the future.
Presentations
The oral presentations are arranged in up to three parallel sessions. There were two afternoon poster sessions, each of which included a part of the demo session, held in a separate room.
Last-minute changes were required to ensure that each poster was given sufficient space for a comfortable presentation. In the future, care should be taken to ensure the communication between the program, demo, student, and local chairs regarding the setup for the poster and demo session, so such misunderstandings can be avoided.
Timeline
Our aim was to push the long paper deadline back as far as possible, to give authors sufficient time to produce new research after ACL (August 2016). At the same time, we wanted the short notification deadline to be sufficiently before the ACL 2017 short paper deadline (6 Feb 2017). This forced us to accelerate the reviewing timeline for the short papers, and thus send reviewing assignments to reviewers as a “Christmas gift” on Dec 23 and skip the author response period. This was feasible because the program and area chairs and reviewers had had a chance to “warm up” with the long papers, but a looser reviewing timeline would make sense too.
The complete timeline is given below.
- 30 Sep 2016: long paper submission deadline
- 1-5 Oct 2016: desk rejects, sorting paper into areas
- 6-8 Oct 2016: long paper bidding
- 9-11 Oct 2016: papers assigned to reviewers
- 12-31 Oct 2016: long paper reviewing
- 1-7 Nov 2016: author response (longer than usual because parallel with EMNLP)
- 8-15 Nov 2016: reviewer discussion
- 15-21 Nov 2016: area chairs make acceptance recommendations
- 21 Nov - 2 Dec 2016: program chairs make acceptance decisions
- 2 Dec 2016: acceptance notifications
- 10 Jan 2017: long camera-ready deadline
- 16 Dec 2016: short paper submission deadline
- 17-18 Dec 2016: desk rejects, sorting paper into areas
- 19-20 Dec 2016: short paper bidding
- 21-23 Dec 2016: papers assigned to reviewers
- 24 Dec 2016 - 13 Jan 2017: short paper reviewing
- 14-19 Jan 2017: reviewer discussion
- 20-22 Jan 2017: area chairs make acceptance recommendations
- 23 Jan - 1 Feb 2017: program chairs make acceptance decisions
- 1 Feb 2017: acceptance notifications
- 10 Feb 2017: short camera-ready deadline
This timeline left 52 days between the last camera-ready deadline and the first day of the conference (April 4).
Recommendations
1. Many reviews were late. For instance, at the time the long author response started, about 10% of reviews were still missing, and there were multiple papers that did not have a single review. This is a real problem, because authors cannot respond to reviews they can’t see. After two days of intense bullying of delinquent reviewers, almost all reviews were in; the rest were supplied by the area chairs. Thus we recommend leaving a two-day gap between the end of reviewing and the start of the author response.
2. Gender and country data should be tracked over time. We see statistics about both at the opening session of every conference, but practical consequences follow from such data only if we can observe whether the balance of genders and countries are moving in the right direction over time. We recommend that EACL records such statistics for different years in a single spreadsheet. In addition, we found it quite time-consuming and unreliable to identify some authors’ genders from their names. Thus we recommend that authors are asked to specify the first author’s gender in the START system submission form.
3. One remarkably difficult and controversial issue was to judge whether papers exceeded the page limits. Some authors use tricks (such as reducing line spacing) to fit their text on eight or four pages, or deviate from the official EACL style file in more or less obvious ways. At the same time, we desk-rejected papers that did not use such tricks, but were eight pages plus five lines long. This situation does not reward the desired behavior, and it would be helpful for future program chairs to be given clear guidance.
4. The PC chairs’ reports from earlier years are fantastic, and we would have learned a lot from reading them before starting our work. Perhaps a more systematic and reliable way could be found to ensure that new PC chairs know about and read such reports early.
Organization chairs report
Paolo Rosso, Francisco Casacuberta (Co-chair), and Jon Ander Gómez (Co-chair)
The local organization welcomed and catered for a total of 667 participants (14% of those only registered for workshops and tutorials, around 40% were students). Local organization was in cooperation with a PCO: Pacifico Meetings. Tasks undertaken by the local committee included meetings with several PCOs; selection of Grupo Pacifico as PCO; visit of possible venues for the conference and selection of Valencia Conference Center as the venue of EACL 2017; selection of "Campo Anibal" for the social dinner; several meetings with PCO and Valencia Conference Center about the organization of logistics, social program, website, and about budget; communication with General Chair, Program Chairs and Workshop Chairs; creation and maintenance of EACL 2017 Twitter account; providing ten student volunteers to help during the workshops and conference; and daily contact with local technicians and PCO for solving unexpected issues.