ACL PROGRAM COMMITTEE REPORT 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the ACL July 7-12, 1997 Universidad Nacional de Educación a Distancia (UNED) Madrid, SPAIN Phil Cohen and Wolfgang Wahlster This year, the ACL-EACL'97 meeting experimented with a hierarchical style of program committee, with 5 Area Chairs, a large international program committee of 53 experts (who read from 7 - 15 papers) supplemented by 54 expert reviewers. The one day program committee meeting was attended by only the Area Chairs and the PC Chairs, to minimize costs and increase interaction. The papers were reviewed blind, even by the Area Chairs, who ranked the papers based on the reviews. OVERVIEW 264 papers were submitted, an increase of 47% over the ACL'96 meeting. Of these, 63 were accepted for an acceptance rate of 24%, in keeping with prior ACL standards. There was a large number of very good papers that could not be included. 10 papers were withdrawn during the review process, since they were accepted by ANLP. These withdrawls caused considerable overhead, since the reviewing process for our conference had already been started before the final list of acceptances for ANLP was forwarded to us. It is highly recommended that the schedules for ANLP, ACL and EACL are better synchronized for future events. 88 submissions came from North America (33%), 129 from Europe (49%), and 47 from Asia (18%). 32 accepted papers come from North America (51%), 25 from Europe (40%) and 6 from Asia (9%). We attempted to give Asia adequate representation in the PC with 4 PC Members (Japan, China). ACL-EACL'97 is a truly international conference: authors from 16 different countries contribute accepted papers. The top three contributing countries are: USA (29), Germany (9), and Japan (5). Canadian, French and British authors contribute 3 papers each, and Dutch and Spanish 2 papers each. The rest of the accepted papers come fom Ireland, Israel, Italy, Romania, Switzerland, and Taiwan (1 paper each). It is interesting to note that 20 of the accepted papers (32%) were authored by reseachers working in industrial labs. This clearly shows the industrial and economic relevance of our field. There was also an increase of accepted papers (6) based on transatlantic collaboration between US researchers and European researchers. The following table gives a rough distribution of the number of submitted papers by topic: Area 1: Morphology, Lexicon, and Finite State Technology 45 Area 2: Grammar and Formalisms for Parsing and Tactical Generation 57 Area 3: Semantics, Pragmatics, and Discourse 50 Area 4: Uses of Language Processing 59 Area 5: Statistical Language Processing 53 Authors could state that a paper fit in multiple areas, and often did. With our own judgements, approximately 36% of papers fell into multiple areas. We diligently attempted to broaden the topics covered by the meeting. For example, the Sadek invited talk was an attempt to answer the question "whatever happened to dialogue?" Another topic to encourage breadth was "Uses of Language Processing," which received most of the submissions (59 or 87 including multiple area classifications), even though there was an ANLP meeting in the same year. THE SUBMISSION PROCESS Based on historical comments about PC workload, we had committed to restrict each PC member to read a maximum of 15 papers. Unfortunately (or fortunately, depending on your perspective), we received a record 264 paper submissions. This required us to more than double the size of the program committee, which, because of other deadlines and commitments, was accomplished in less than a week. Fortunately, with diligent work by the Co-chairs and Area Chairs, we were able to meet both the deadline and our paper reviewing commitments. Papers were accepted both on paper and electronically, using the scripts and LaTex processing set up at prior ACL meetings. Although the automated LaTex processing worked quite well, the whole process needs rethinking. Unless papers will be distributed electronically to the PC members, there is little reason to submit them electronically. We ended up making over 10,000 photocopies, at ACL expense, in order to send printed copies of the electronically submitted papers to the PC members. Given the record number of submissions, this turned out to be quite burdensome, and could have led to numerous difficulties. Luckily, we had only one potential problem, which was properly resolved. However, there are numerous points of failure in the electronic submission process, including papers being deleted before they are backed up by the nightly backup run, unprintable papers, the gradual move from LaTeX to other formatters (e.g., Word), etc. Until the computational world evolves to more standardized environment, e.g, where papers are posted on Web sites and the conference retrieves them, the cost and hassle can be minimized by using ordinary paper submissions. THE REVIEW PROCESS We had 3 reviewers for each paper. Since a number of PC members did not complete their reviewing assignment (in the first round, 48 papers were reviewed by only 2 reviewers), this meant that several papers did not receive the desired full set of reviews. Luckily, when extra reviews were needed, the Area Chairs were able to fill in, even though each Area Chair had 50 - 70 papers to handle, and could not read all of them. In the future, if we want more than 3 reviews, we need to move to an even larger program committee, with more Area Chairs and more reviewers. Note that the number of submissions this year is the same as CHI, who has twice as many reviewers and Area Chairs, but accepts the same percentage of papers. In fact, if we had done a more thorough job of publicity (more on that later), we believe there would have been another 50% increase in submissions. In other words, we are on the verge of both success, and success disaster. We need to scale up our entire ACL organization, offer the conferences in larger venues, and be prepared for attention. In our view, it's about time! In managing the submission of reviews, we used an interactive Web form designed by DFKI, so that reviewers could write their reviews online. The special feature of this Web form was that all reviewers were forced to add a few lines as a justification for negative ratings >=3. The form solicited information as follows: - ------------------------------------------------------- PAPER-CODE: PAPER-TITLE (AT LEAST FIRST FOUR WORDS OF THE TITLE): REVIEW PREPARED BY: Please give a justification for ratings >= 3 for each item and use exactly one of the integers 1, 2, 3, 4 for rating (i.e. ratings like 3.5 or 2-3 are not possible). ORIGINALITY: 1 = Strikingly original 2 = Original 3 = Derivative 4 = Very derivative JUSTIFICATION: APPROPRIATENESS FOR ACL/EACL-97: 1 = Extremely relevant 2 = Relevant 3 = Marginal 4 = Inappropiate for this conference JUSTIFICATION: TECHNICAL ACCURACY AND COMPLETENESS: 1 = Accurate 2 = Minor flaws 3 = Major flaws 4 = Invalid JUSTIFICATION: DEGREE OF IMPLEMENTATION/EVALUATION/VALIDATION: 1 = Fully 2 = Partially 3 = Proposed 4 = None JUSTIFICATION: PRESENTATION AND ORGANIZATION: 1 = Excellently presented 2 = Well presented 3 = Readable 4 = Badly presented JUSTIFICATION: CHANGES REQUIRED FOR THE FINAL VERSION: 1 = No changes required 2 = A few changes required 3 = Many changes required 4 = Paper must be rewritten JUSTIFICATION: MY OVERALL RECOMMENDATION IS: 1 = Accept 2 = Marginal 3 = Leaning to reject 4 = Reject THE ABOVE OVERALL JUDGEMENT IS: 1 = Confident 2 = Fairly confident 3 = Somewhat uncertain 4 = Uncertain CONFIDENTIAL COMMENTS FOR THE AREA CHAIRS: FURTHER COMMENTS FOR THE AUTHOR(S): - ----------------------------------------------------------------- Area Chairs could use a password to access their reviews as postscript files. The reviews were sorted by paper code, and automatically stored in a review database. Although a 100 MB ATM link was used for the Web server, some reviewers preferred to send their reviews via email. This caused some overhead, since in these cases the reviews could not be processed automatically. All papers for which reviewers indicated low confidence in their judgements were explicitly discussed during the one day PC meeting on Sunday, March 2 at Brighton, UK. The PC meeting was quite economical. We spent 3400 USD for 7 people (2 from the US, 4 from Europe, 1 living in Brighton). THE PRODUCTION OF THE PROCEEDINGS 8 proceedings pages were reserved for each accepted paper. 7 authors used less than the allocated 8 pages (6 or 7 pages). There was an increased page charge of $250 USD for each extra page. 3 authors used 1 extra page and 2 authors used 2 extra pages. The proceedings consists of 540 pages. The printer was LEd Language Editing in Utrecht, Netherlands. The cost for one copy of the proceedings is about $10 USD (including shipping to the conference venue). 700 copies were produced for sale at the conference. The price per page of the proceedings (700 copies) is $13 USD. There was an increase of 10% wrt the first quote of the production costs, since some authors delivered their camera-ready version on recycled paper that requires special processing. In future instructions to the authors of accepted papers, the use of recycled paper should be explicitly ruled out. The different paper size (A4 in Asia and Europe vs. Letter Size in North America) caused some problems in the final pagination and formatting process. The final pagination was quite tedious, since every page of a camera-ready paper had to be fed through a laser printer to achieve a uniform layout. PUBLICITY The call for papers was essentially only mailed to people on the ACL mailing list. SIG members who are not members of ACL (yes, there are those) did not receive mailings. Moreover, to our knowledge, we did not systematically attempt to publicize the CFP or the program itself within the speech, information retrieval, HCI, cognitive science, and general AI communities. For example, we should have had a summary page in the AAAI magazine, and one in the CACM. Given the turnout we received, this was mercifully lucky. However, we believe we do ourselves a disservice by not broadening the appeal. In particular, we believe ACL should absorb the ANLP meeting, and present a larger scale meeting, with applied papers, demonstrations, lively panels, lab overviews, etc., in addition to our usual high quality theoretical work. We would do well to emulate the CHI meetings, which are always enjoyable and have something for everyone. We already have a two-track and occasionally three-track meeting, so the days of single track highly technical meetings are now a thing of the past. THANKS We would like to thank everyone who helped to bring about the richness and high standards of the ACL-EACL'97 conference. We thank all of the authors of submitted papers for choosing ACL-EACL'97 as a forum for the communication of their research results. We have been constantly impressed by the dedication of the Area Chairs and other Program Committee Members, who worked very hard to get the best possible job done. Special praise goes to our workshop coordinator, Harald Trost from the Austrian Research Institute for AI, who ably arranged the post-conference workshops, and to Megumi Kameyama from SRI International, who put together an excellent slate of tutorial speakers. We are very grateful to Pamela Jordan and Johan Bos, who organized two most interesting student sessions. We would like to express our gratitude to Kathy McKeown and Priscilla Rasmussen, who provided support and guidance for our work. In addition, we are most appreciative of the outstanding job performed by Felisa Verdejo, the local arrangements chair, and her Local Organizing Committee in Madrid. We would like to thank the German Research Center for Artificial Intelligence (DFKI) for providing internal financial support for both administrative and technical tasks related to the conference. Also commended for their excellent support are Ira Smith from OGI, who handled the electronic submissions and designed and maintained the submission database; Jochen Mueller from DFKI, who designed the interactive Web form for reviewing, and maintained the review database and formatted the front pages; Gabi Jacquinot from DFKI, who maintained the PC database and helped with the Proceedings, and Vivienne Wicks, who arranged a very pleasant PC meeting in Brighton.