2010Q3 Reports: Program Chairs
ACL Program Chair Report
Sandra Carberry and Stephen Clark
June 21, 2010
ACL 2010 received 987 submissions, a record number for the
conference. This was a surprise, since the submission deadline was identical for both long and short papers. Thus we expected that the total number of submissions would be lower than for ACL 2009, where authors of rejected long papers had the opportunity to resubmit them as revised short papers.
655 papers were submitted to ACL 2010 as long papers, and 332
papers were submitted as short papers. After filtering out those papers that did not satify the submission requirements, for example exceeded the length limitations or were not anonymous, 956 papers were distributed to the Area Chairs for reviewing, of which 646 were long submissions and 310 were short submissions. Due to the large number of submissions, a fifth parallel session was added to the program. We accepted 7 long papers as short papers, since these were considered worthy of acceptance, but not as long papers, and it was thought that the content could be presented as short papers. Counting these 7 submissions as short papers, the overall acceptance rate was 25% for long papers and 22% for short papers. The submissions roughly fell into the following categories, with an attempt made to prevent some categories, for example Machine Translation, from becoming too large (which would have placed an unreasonable burden on the MT Area Chairs):
Long Paper Submissions: AREA SUBMITTED ACCEPT-LONG ACCEPT-SHORT %-ACCEPTED
--------- ----------- ------------ ----------
Bioinformatics 10 1 10.0% Discourse 38 11 28.9% Formal semantics 19 6 31.6% Generation/summarization 39 10 1 28.2% Information extraction 44 8 18.2% Information retrieval 24 6 25.0% Lexical semantics 59 16 27.1% Machine learning 57 13 22.8% Machine translation 64 15 23.4% Mathematical linguistics 23 10 43.5% Multimodal 13 4 30.8% Parsing 68 16 2 26.5% Psycholinguistics 14 5 35.7% Question answering 22 5 2 31.8% Resources and evaluation 28 8 28.6% Sentiment analysis 46 9 1 21.7% Speech 18 4 22.2% Tagging 37 10 27.0% Text mining 23 7 1 34.8%
Short Paper Submissions:
AREA SUBMITTED ACCEPTED %-ACCEPTED
--------- -------- ----------
Bioinformatics 2 0 0.0% Discourse 21 5 23.8% Formal semantics 4 1 25.0% Generation/summarization 17 2 11.8% Information extraction 11 3 27.3% Information retrieval 12 1 8.3% Lexical semantics 30 6 20.0% Machine learning 30 8 26.7% Machine translation 38 10 26.3% Mathematical linguistics 6 0 0.0% Multimodal 5 1 20.0% Parsing 24 5 20.8% Psycholinguistics 5 1 20.0% Question answering 18 2 11.1% Resources and evaluation 25 5 20.0% Sentiment analysis 26 6 23.1% Speech 15 3 20.0% Tagging 14 3 21.4% Text mining 7 1 14.3%
The ACL 2010 program contains a wide variety of papers, ranging
from theoretical papers to analysis papers to empirical papers. Of particular interest is the presence of 3 challenge papers and 3 survey papers on the ACL 2010 program; unfortunately, although we sought position papers, none of the submissions in this category were judged to warrant acceptance.
ACL 2010 will award 3 best paper prizes: best long paper, best long
paper by a student author, and best short paper. The selection of the prize recipients was made by a small panel consisting of selected area chairs and other senior members of the research community. Although the selection of a single paper in each prize category was difficult, we chose to award only one prize in each category since we felt that it lends more prestige to the prize than if the prize were shared by several papers.
All submissions that were accepted as long papers were allocated 9
pages of content in the proceedings, with the authors being granted an extra page for the final version compared to the 8 pages of content allowed at submission time. All submissions that were accepted as short papers were allocated 5 pages of content in the proceedings, again with an extra page allowed for the final version compared to the 4 pages at submission time. In both cases, authors were allowed an unlimited number of extra pages for references. The extra page of content in the final papers was an experiment this year, in order to allow authors to better improve their papers by addressing the comments and suggestions of the reviewers, without having to cut essential parts of their original submissions. Long papers will be presented either as 25 minute oral talks or as 10 minute oral talks followed by a poster presentation. Short papers will be presented either as 10 minute oral talks followed by a poster presentation or just as a poster presentation. The decision about presentation mode was made by the program chairs based on the quality of the paper, input from the area chairs, and our own judgement about how the paper might best be presented.
In order to attract and appropriately review a wider variety of
papers, we experimented this year with different review forms for the different categories of papers. For example, the review criteria (and thus the questions on the review form) for theoretical papers were different from those for empirical papers. There were 10 different review categories (analysis, challenge, empirical, negative result, paradigms, position, resources, survey, systems, and theoretical), which, combined with the long/short distinction resulted in 20 different review forms; the set of review forms can be found at:
http://acl2010.org/reviewforms.html
In retrospect, we believe that the different review forms helped immensely in obtaining appropriate reviews for the different types of papers.
However, despite the submission page explicitly directing authors to
examine the review forms posted on the ACL web site before selecting their review category, it is clear that authors did not do so. Thus the program chairs and area chairs examined every paper and changed the review category for papers where an inappropriate category had been selected. (Note that the Call for Papers stated that the Program Chairs reserved the right to change the review category.) This was an enormous amount of work. If this experiment is continued next year, we recommend that the submission web page do one of the following:
1. contain a question associated with each category (such
as "Does this paper present a system that has been deployed
in an industrial or research setting and includes reports
of tests with actual users?" for a systems paper), where
the user must reply "Yes" in order to select that submission
category.
2. pop up the review form for the category that the author selects,
along with the statement "I have read this entire review form and
believe that it is appropriate for this submission". The author
would need to respond "Yes" in order to finalize the category
selection.
We would like to note that Rich Gerber was extremely helpful in modifying the START system to accommodate our needs. For example, the START system has been modified so that when a reviewer clicks on the review form for a paper, he or she gets the appropriate review form for that category of paper.
Most of our Area Chairs and reviewers were outstanding. However,
a few reviewers did not provide the high-quality reviews that we expect for ACL submissions. It appears that these individuals often review for ACL. Although we do not have a solution, we do believe that some mechanism should be developed for keeping track of reviewers whose work is below norm so that future program chairs and area chairs avoid inviting them to be part of the program committee.
While we hope that at least one author of each accepted paper will
attend the conference and present the paper, it appears that in a few cases, the authors may not be making a serious attempt to ensure that at least one author is present at the conference. Some conferences require that at least one author register for the conference in order for a paper to appear in the proceedings; other conferences prohibit authors from submitting to the conference in the future if at least one author of an accepted paper does not attend the conference. We recognize that these are either difficult to enforce or somewhat draconian measures, we do suggest that ACL adopt the policy that the submission page contain the following statement:
If this submission is accepted for the ACL conference, we commit to at least one of the authors attending the conference and presenting the paper during the main technical session on <dates>.
with the requirement that the authors click "YES" in order for the submission to be successful.
We also encountered problems with authors expecting that the
submission deadline would not be enforced. We had several authors (some who are senior researchers and active in ACL) who were very unhappy that the START system refused their papers after the submission deadline. One spoke of a "traditional 1 hour or more grace period" for submissions. After consultation with the ACL Exec, we did not accept these late submissions since we felt it was the only way to be fair to all authors. (We did keep the START system open for 15 minutes after the deadline in order to avoid cutting off authors who were in the process of submitting prior to the deadline.) We most strongly recommend that ACL adopt the following (or something similar) as policy, that it be included in the Call for Papers in future years, and that it be added to the ACL conference handbook:
"The ACL submission deadline will be extended only in the event that the START system crashes near the deadline. The START system will automatically shut down at the deadline, and it is ACL policy that late submissions will not be allowed."
Without such a policy, exceptions that are informally granted by Program Chairs are unfair to other authors (who are not aware of the possibility of an exception) and cause problems for subsequent Program Chairs.
ACL 2010 will have two outstanding invited talks:
Andrei Broder: vice-president of Yahoo and both an ACM Fellow
and an IEEE Fellow. He will present a talk on
the emerging field of computational advertising,
with an emphasis on issues relevant to computational
linguistics and natural language processing.
Zenzi Griffin: professor of psychology at the University of Texas
at Austin. She will present a talk on the
psycholinguistics of social interaction, with an
emphasis on issues in language processing.
Our goal was to select invited speakers whose talks would be related to computational linguistics, but would broaden the perspective of the conference attendees. In addition, we sought individuals who had a reputation for excellent presentational skills. We believe that both of these individuals satisfy these criteria and will present excellent and exciting invited talks at ACL.