2015Q3 Reports: Program Chairs

Innovations

As compared to ACL conferences in prior years, the main innovations this year were:

1. To ensure fairness for all authors and as suggested by the ACL executive board, this year all members of the PC chairs' research teams, including students and researchers, were not allowed to submit any long or short papers.
2. We introduced the item "MENTORING" in the review form to indicate whether a paper needs the help of a mentor in its writing, organization or presentation.
3. Around 18.2% of reviewers were recommended as outstanding reviewers and acknowledged in the proceedings.

Submissions and Presentations

ACL-IJCNLP 2015 received a total of 1340 submissions, of which 692 were long papers and 648 were short papers. 19 long papers and 65 short papers were rejected without review due to non-anonymity or formatting issues. The remaining submissions were assigned to one of 18 areas/tracks, and managed by a program committee of 37 area chairs and 754 primary reviewers and 137 secondary reviewers.

173 (25.0%) of the 692 qualifying long papers and 144 (22.2%) of the 648 qualifying short papers were selected for presentation at the conference. Of the accepted long papers, 105 were selected for oral presentation, and 68 for poster presentation. Of the accepted short papers, 50 have oral and 95 have poster presentation slots. The oral versus poster decision was based not on the quality of the work, but the estimated appeal to a wide audience.

In addition, 12 TACL papers will be presented at ACL -- 7 as talks and 5 as posters. Including TACL papers, there will be 164 oral and 165 poster presentations at the main ACL conference. The table below shows the number of reviewed submissions in each area for long and short papers, as well as the number of papers accepted in each area. The table also shows the number of qualifying long and short papers that were withdrawn prior to the completion of the review process (16 long and 8 short papers were withdrawn).

Area Long Received Long Accepted Short Received Short Accepted Total Submissions Percent of Total Submissions Total Accepts Percent of Total Accepts Area Accept Rate Discourse, Coreference and Pragmatics 32 9 28 7 60 4.48% 16 5.04% 26.67% Information Retrieval 14 4 21 5 35 2.61% 9 2.84% 25.71% Information Extraction and Text Mining 80 20 61 11 141 10.52% 31 9.78% 21.99% Language and Vision 14 2 11 4 25 1.87% 6 1.89% 24% Language Resources and Evaluation 34 5 34 9 68 5.07% 14 4.42% 20.59% Lexical Semantics and Ontology 35 7 20 4 55 4.10% 11 3.47% 20.00% Linguistic and Psycholinguistic Aspects of CL 14 5 16 4 30 2.24% 9 2.84% 30.00% Machine Learning and Topic Models for Language Processing 61 19 53 15 114 8.51% 34 10.73% 29.82% Machine Translation and Multilinguality 70 16 72 15 142 10.60% 31 9.78% 21.83% NLP Applications and NLP-enabled Technology 36 8 46 7 82 6.12% 15 4.73% 18.29% NLP for the Web and Social Media 41 7 44 10 85 6.34% 17 5.36% 20.00% Phonology, Morphology and Word Segmentation 12 3 16 5 28 2.09% 8 2.52% 28.57% Question Answering 14 4 11 3 25 1.87% 7 2.21% 28.00% Semantics 61 17 48 11 109 8.13% 28 8.83% 25.69% Sentiment Analysis and Opinion Mining 51 13 49 11 100 7.46% 24 7.57% 24.00% Spoken Language Processing, Dialogue and Interactive Systems, and Multimodal NLP 20 5 22 6 42 3.13% 11 3.47% 26.19% Summarization and Generation 27 6 40 7 67 5% 13 4.10% 19.40% Tagging, Chunking, Syntax and Parsing 60 23 48 10 108 8.06% 33 10.41% 30.56% withdraw 16 0 8 0 24 1.79% 0 0.00% 0.00% total 692 173 648 144 1340 100% 317 100.00% 23.66%

Review process

We used Mark Dredze's tool for reviewer assignment this year, but we didn't use its all functions. For example, we didn't use the function that the tool automatically assigns submissions to each reviewer according to his/her preference and bidding. We asked the area chairs to manually make the final assignments. We recommend integrating the tool into the START system. Thanks to the tool and the cooperation of area chairs, we organized the review process smoothly with over 750 expert reviewers. Using the successful experience of last year, we divided the submissions into different categories and used different review forms for empirical/data-driven, theoretical, applications/tools, resources/evaluation, and survey papers (however, authors need to be educated about the different categories). For the short papers we encouraged to submit papers on negative results, which was explicitly mentioned in the CFP, though we did specially had the negative results category for submission and review.

We changed the review forms slightly. We added the item "MENTORING" to the review form to show whether a paper needs the help of a mentor in its writing, organization or presentation. 6 accepted long papers and 9 accepted short papers need mentoring.

Based on a decision of the conference committee, the page limit was 8 pages for long paper submissions and 4 pages for short paper submissions (each with 2 additional pages for references), which was identical with NAACL and EMNLP this year. Camera-ready versions were given one additional page: 9 plus 2 pages for long papers, 5 plus 2 pages for short papers.

Formatting and anonymity were carefully checked with the help of the area chairs and a young researcher volunteer Jiajun Zhang who didn't have submission. More than about 10% of the submissions had author names in the properties of their files. We had to manually remove this identifying information, but it would be great if that could be automated as Kristina Toutanova and Hua Wu, PC Co-chiars of ACL'2014, said in their 2014Q3 Reeports. For long papers that had author names listed under the title, we asked the authors to remove their names from their manuscript without change of contents and send us within 24 hours. Otherwise, the submissions were rejected without review. For short submissions violating the anonymity, we didn't give authors any chance to change their manuscripts. All submissions violating the anonymity, format requirements and length limit (we allowed for about 5 extra lines) were rejected without review without exception.

Best paper awards

The area chairs nominated seventeen papers from 9 areas for the long best paper award. We selected a list of five candidates which were ranked by a best paper committee, consisting of 5 members, all non-area chairs. Based on the ranking and comments by the committee members, we selected a best student paper and a best paper. These two papers will be presented in a plenary session at the end of the conference. We didn't select the best short paper this year.

Presentations

The oral presentations are arranged in five parallel sessions. There are two large poster sessions including dinner on two evenings of the conference. We manually grouped the papers in groups based on the area and theme for sessions. We did not have the time to perform a scheduling survey (most likely because we lost three weeks between final submission deadline and conference because the conference was rescheduled).

Recommendations

We would like to make the following recommendations for future years:

1. As we mentioned before, we hope that the tool for reviewer assignment can be integrated into the START system. This would allow for a smoother process.
2. We allocated some reviewers in two or three areas according to their preferences. This made it difficult for us to control the quota limit of reviews since two or three areas assigned them review tasks at the same time. We had to coordinate with area chairs. If the START system could automatically check whether the review task assigned to a reviewer exceeds the limit, PC chairs would receive fewer complaints.
3. Add tools to START for checking non-anonymity, formatting issues and removing names containing in properties of file. (Carried over from last year :)
4. Enable merging of the schedules for long, short, and TACL papers in START, as well as uploading metadata and papers for TACL papers, to simplify the process of deriving the conference program for the website, the conference handbook, and the downloadable proceedings.