ACL PROGRAM COMMITTEE REPORT


       35th Annual Meeting of the Association for Computational Linguistics and
               8th Conference of the European Chapter of the ACL


                               July 7-12, 1997
          Universidad Nacional de Educación a Distancia (UNED)
                              Madrid, SPAIN


                    Phil Cohen and Wolfgang Wahlster



This year, the ACL-EACL'97 meeting experimented with a hierarchical style
of program committee, with 5 Area Chairs, a large international program
committee of 53 experts (who read from 7 - 15 papers) supplemented by 54
expert reviewers. The one day program committee meeting was attended by
only the Area Chairs and the PC Chairs, to minimize costs and increase
interaction. The papers were reviewed blind, even by the Area Chairs, who
ranked the papers based on the reviews.


OVERVIEW

264 papers were submitted, an increase of 47% over the ACL'96 meeting. Of
these, 63 were accepted for an acceptance rate of 24%, in keeping with
prior ACL standards. There was a large number of very good papers that
could not be included. 10 papers were withdrawn during the review process,
since they were accepted by ANLP. These withdrawls caused considerable
overhead, since the reviewing process for our conference had already been
started before the final list of acceptances for ANLP was forwarded to us.
It is highly recommended that the schedules for ANLP, ACL and EACL are
better synchronized for future events.

88 submissions came from North America (33%), 129 from Europe (49%), and 47
from Asia (18%).  32 accepted papers come from North America (51%), 25 from
Europe (40%) and 6 from Asia (9%). We attempted to give Asia adequate
representation in the PC with 4 PC Members (Japan, China).

ACL-EACL'97 is a truly international conference: authors from 16 different
countries contribute accepted papers. The top three contributing countries
are: USA (29), Germany (9), and Japan (5). Canadian, French and British
authors contribute 3 papers each, and Dutch and Spanish 2 papers each. The
rest of the accepted papers come fom Ireland, Israel, Italy, Romania,
Switzerland, and Taiwan (1 paper each).

It is interesting to note that 20 of the accepted papers (32%) were
authored by reseachers working in industrial labs. This clearly shows the
industrial and economic relevance of our field. There was also an increase
of accepted papers (6) based on transatlantic collaboration between US
researchers and European researchers.

 The following table gives a rough distribution of the number of submitted
papers by topic:

Area 1:	Morphology, Lexicon, and Finite State Technology		45
Area 2:	Grammar and Formalisms for Parsing and Tactical Generation 	57
Area 3:	Semantics, Pragmatics, and Discourse				50
Area 4:	Uses of Language Processing   					59
Area 5:	Statistical Language Processing					53

Authors could state that a paper fit in multiple areas, and often did. With
our own judgements, approximately 36% of papers fell into multiple areas.

We diligently attempted to broaden the topics covered by the meeting. For
example, the Sadek invited talk was an attempt to answer the question
"whatever happened to dialogue?" Another topic to encourage breadth was
"Uses of Language Processing," which received most of the submissions (59
or 87 including multiple area classifications),  even though there was an
ANLP meeting in the same year.


THE SUBMISSION PROCESS

Based on historical comments about PC workload, we had committed to
restrict each  PC member to read a maximum of 15 papers. Unfortunately (or
fortunately, depending  on your perspective), we received a record 264
paper submissions. This required us  to more than double the size of the
program committee, which, because of other deadlines and commitments, was
accomplished in less than a week.  Fortunately, with diligent work by the
Co-chairs and Area Chairs, we were able to meet both the deadline and our
paper reviewing commitments.

Papers were accepted both on paper and electronically, using the scripts
and LaTex  processing set up at prior ACL meetings. Although the automated
LaTex processing  worked quite well, the whole process needs rethinking.
Unless papers will be  distributed electronically to the PC members, there
is little reason to submit them  electronically. We ended up making over
10,000 photocopies, at ACL expense, in order to send printed copies of the
electronically submitted papers to the PC members. Given the record number
of submissions, this turned out to be quite burdensome, and could have led
to numerous difficulties. Luckily, we had only one potential problem, which
was properly resolved. However, there are numerous points of failure in the
electronic submission process, including papers being deleted before they
are backed up by the nightly backup run, unprintable papers, the gradual
move from LaTeX to other formatters (e.g., Word), etc. Until the
computational world evolves to more standardized environment, e.g, where
papers are posted on Web sites and the conference retrieves them, the cost
and hassle can be minimized by using ordinary paper submissions.


THE REVIEW PROCESS

We had 3 reviewers for each paper. Since a number of PC members did not
complete their reviewing assignment (in the first round, 48 papers were
reviewed by only 2  reviewers), this meant that several papers did not
receive the desired full set of reviews.  Luckily, when extra reviews were
needed, the Area Chairs were able to fill in, even though each Area Chair
had 50 - 70 papers to handle, and could not read all of them.

In the future, if we want more than 3 reviews, we need to move to an even
larger program committee, with more Area Chairs and more reviewers.  Note
that the number of submissions this year is the same as CHI, who has twice
as many reviewers and Area Chairs, but accepts the same percentage of
papers. In fact, if we had done a more thorough job of publicity (more on
that later), we believe there would have been another 50% increase in
submissions. In other words, we are on the verge of both success, and
success disaster.  We need to scale up our entire ACL organization, offer
the conferences in larger venues, and be prepared for attention. In our
view, it's about time!

In managing the submission of reviews, we used an interactive Web form
designed by DFKI, so that reviewers could write their reviews online. The
special feature of this Web form was that all reviewers were forced to add
a few lines as a justification for negative ratings >=3.

The form solicited information as follows:

- -------------------------------------------------------
PAPER-CODE:

PAPER-TITLE (AT LEAST FIRST FOUR WORDS OF THE TITLE):

REVIEW PREPARED BY:

Please give a justification for ratings >= 3 for each item and use
exactly one of the integers 1, 2, 3, 4 for rating (i.e. ratings
like 3.5 or 2-3 are not possible).

ORIGINALITY:
1 = Strikingly original
2 = Original
3 = Derivative
4 = Very derivative

JUSTIFICATION:

APPROPRIATENESS FOR ACL/EACL-97:
1 = Extremely relevant
2 = Relevant
3 = Marginal
4 = Inappropiate for this conference

JUSTIFICATION:

TECHNICAL ACCURACY AND COMPLETENESS:
1 = Accurate
2 = Minor flaws
3 = Major flaws
4 = Invalid
JUSTIFICATION:

DEGREE OF IMPLEMENTATION/EVALUATION/VALIDATION:
1 = Fully
2 = Partially
3 = Proposed
4 = None

JUSTIFICATION:

PRESENTATION AND ORGANIZATION:
1 = Excellently presented
2 = Well presented
3 = Readable
4 = Badly presented

JUSTIFICATION:

CHANGES REQUIRED FOR THE FINAL VERSION:
1 = No changes required
2 = A few changes required
3 = Many changes required
4 = Paper must be rewritten

JUSTIFICATION:

MY OVERALL RECOMMENDATION IS:

1 = Accept
2 = Marginal
3 = Leaning to reject
4 = Reject

THE ABOVE OVERALL JUDGEMENT IS:

1 = Confident
2 = Fairly confident
3 = Somewhat uncertain
4 = Uncertain

CONFIDENTIAL COMMENTS FOR THE AREA CHAIRS:

FURTHER COMMENTS FOR THE AUTHOR(S):
- -----------------------------------------------------------------

Area Chairs could use a password to access their reviews as postscript
files. The reviews were sorted by paper code, and automatically stored in a
review database.   Although a 100 MB ATM link was used for the Web server,
some reviewers preferred to send their reviews via email. This caused some
overhead, since in these cases the reviews could not be processed
automatically.

All papers for which reviewers indicated low confidence in their judgements
were explicitly discussed during the one day PC meeting on Sunday, March 2
at Brighton, UK.

The PC meeting was quite economical. We spent 3400 USD for 7 people (2 from
the US, 4 from Europe, 1 living in Brighton).


THE PRODUCTION OF THE PROCEEDINGS

8 proceedings pages were reserved for each accepted paper. 7 authors used
less than the allocated 8 pages (6 or 7 pages). There was an increased page
charge of $250 USD for each extra page. 3 authors used 1 extra page and 2
authors used 2 extra pages.

The proceedings consists of 540 pages. The printer was LEd Language Editing
in Utrecht, Netherlands. The cost for one copy of the proceedings is about
$10 USD (including shipping to the conference venue). 700 copies were
produced for sale at the conference. The price per page of the proceedings
(700 copies) is $13 USD.

There was an increase of 10% wrt the first quote of the production costs,
since some authors delivered their camera-ready version on recycled paper
that requires special processing. In future instructions to the authors of
accepted papers, the use of recycled
paper should be explicitly ruled out.

The different paper size (A4 in Asia and Europe vs. Letter Size in North
America) caused some problems in the final pagination and formatting
process. The final pagination was quite tedious, since every page of a
camera-ready paper had to be fed through a laser printer to achieve a
uniform layout.


PUBLICITY

The call for papers was essentially only mailed to people on the ACL
mailing list. SIG  members who are not members of ACL (yes, there are
those) did not receive mailings. Moreover, to our knowledge, we did not
systematically attempt to publicize the CFP or  the program itself within
the speech, information retrieval, HCI, cognitive science, and general AI
communities.

For example, we should have had a summary page in the AAAI magazine, and
one in the CACM. Given the turnout we received, this was mercifully lucky.
However, we believe we do ourselves a disservice by not broadening the
appeal. In particular, we believe ACL should absorb the ANLP meeting, and
present a larger scale meeting, with applied papers, demonstrations, lively
panels, lab overviews, etc., in addition to our usual high quality
theoretical work.

We would do well to emulate the CHI meetings, which are always enjoyable
and have something for everyone. We already have a two-track and
occasionally three-track meeting, so the days of single track highly
technical meetings are now a thing of the past.


THANKS

We would like to thank everyone who helped to bring about the richness and
high standards of the ACL-EACL'97 conference. We thank all of the authors
of submitted papers for choosing ACL-EACL'97 as a forum for the
communication of their research results. We have been constantly impressed
by the dedication of the Area Chairs and other Program Committee Members,
who worked very hard to get the best possible job done. Special praise goes
to our workshop coordinator, Harald Trost from the Austrian Research
Institute for AI, who ably arranged the post-conference workshops, and to
Megumi Kameyama from SRI International, who put together an excellent slate
of tutorial speakers. We are very grateful to Pamela Jordan and Johan Bos,
who organized two most interesting student sessions.

We would like to express our gratitude to Kathy McKeown and Priscilla
Rasmussen, who provided support and guidance for our work. In addition, we
are most appreciative of the outstanding job performed by Felisa Verdejo,
the local arrangements chair, and her Local Organizing Committee in Madrid.

We would like to thank the German Research Center for Artificial
Intelligence (DFKI) for providing internal financial support for both
administrative and technical tasks related to the conference.

Also commended for their excellent support are Ira Smith from OGI, who
handled the  electronic submissions and designed and maintained the
submission database; Jochen  Mueller from DFKI, who designed the
interactive Web form for reviewing, and maintained  the review database and
formatted the front pages; Gabi Jacquinot from DFKI, who  maintained the PC
database and helped with the Proceedings, and Vivienne Wicks, who arranged
a very pleasant PC meeting in Brighton.