NAACL-2000

NAACL-2000 Program Chair's Report
Janyce Wiebe

27 April 2000

1.  The program

In its first year, NAACL received 166 submissions.  43 papers were
accepted, giving an acceptance rate of 26%.  

With input from the NAACL and ANLP program committees, I chose to
invite Len Talmy to give a plenary talk.

Providing a detailed country breakdown for submissions and acceptances
is difficult, because many papers are multi-authored.  However, a
rough idea can be gained from the data we have on host countries of
the first authors.  This gives us a submission profile of:
    
   93 from North American (56% of submissions)
   54 from Europe         (33% of submissions)
   18 from Asia and       (11% of submissions)
      New Zealand
   1  from Brazil         ( 0% of submissions)

The acceptance profile is:

   27 from North America  (29% acceptance rate)
   15 from Europe         (28% acceptance rate)
    1 from Asia           ( 0% acceptance rate)
    0 from Brazil         ( 0% acceptance rate)

   
Apparently, more North Americans and Europeans than Asians decided to
submit their better work to NAACL.  Perhaps this is due to the fact
that ACL is in Asia this year.

I invited program committee members to be session chairs, starting
with area chairs, and filling the remaining slots with reviewers.

2. The Program Committee

The program committee was hierarchical, with a senior program
committee composed of 8 area chairs and a program chair, and 113
reviewers who reported to the area chairs.  All members of the senior
program committee are from North America (7 from the USA and 2 from
Canada).  The reviewers represent 20 different countries, and all are
experienced reviewers.  72 (64%) of them are from North America; 30
(27%) are from Europe; and 11 (10%) are from Asia.

The lists of reviewers were coordinated, such that only in a few
exceptional cases were reviewers asked to review for more than one
area.

The area chairs cannot be thanked enough for the conscientious and
painstaking jobs they performed.  The reviewers were also excellent,
providing careful and timely reviews of the papers they were assigned.

3.  Areas

The NAACL areas are

    Discourse, Dialogue, and Pragmatics
    Semantics and the Lexicon
    Syntax, Morphology, and Phonology
    Generation and Summarization
    Spoken Language
    Corpus-Based and Statistical Natural Language Processing
    Cognitive Modeling and Human-Computer Interaction
    Multilingual Natural Language Processing

They include areas traditionally well-represented at ACL conferences,
and also areas we would like to expand.  The feeling among the senior
program committee is that having separate areas in Spoken Language and
Multilingual Natural Language Understanding helped attract quality
submissions in those areas.  Unfortunately, the Cognitive Modeling and
Human-Computer Interaction area was not as successful.

Following is breakdown of submissions and acceptances by area:

                                       Submissions  Acceptances  Acceptance 
                                                                 Rate

Discourse, Dialogue, and Pragmatics    23           4            17.4
Semantics and the Lexicon              25           7            28.0
Syntax, Morphology, and Phonology      28           9            32.1
Generation and Summarization           17           3            17.6
Spoken Language                        15           4            26.7
Corpus-Based and Statistical 
     Natural Language Processing       33          11            33.3
Cognitive Modeling and 
     Human-Computer Interaction         7           1            14.3
Multilingual Natural Language 
     Processing                        18           4            22.2

4.  Call for Papers

In its first year, it was advantageous to co-locate NAACL with ANLP, an
established conference.  To coordinate the two conferences,
submissions focusing on end-applications were invited to ANLP 2000,
while submissions focusing on methodology were invited to NAACL 2000.
Future NAACL conferences may encourage both types of submissions.

Although I do not endorse the split into "end-applications" and
"methodology", this split posed no particular problems for NAACL.

5.  Reviewing and Acceptance Decisions

Reviewing was blind to reviewers and area chairs.  Information about
authors' identities was given to program committee members in a few
cases when it was needed to confirm conflicts of interest.

Papers were reviewed by 3 reviewers each.  Once the reviews were complete,
the reviewers discussed among themselves the papers for which there was 
disagreement. Based on the reviews and subsequent discussions,
the area chairs identified the papers that appeared to be clear acceptances, 
clear rejections, and borderline cases.  

At the senior program committee meeting held in Crystal City,
Virginia, January 29, 2000, 9AM to 6PM, all of the papers that had
been judged to be apparent borderline and accepted papers were
discussed, and final acceptance decisions were made.

6.  Process

Extending Henry Thompson's electronic review form for EACL '99,
staff at the Computing Research Laboratory at NMSU developed software to 
enable electronic handling of the reviewing processes.  Reviewers obtained 
the papers electronically from the Web; Area chairs were provided with an
on-line system for assigning papers to reviewers; and the reviewers submitted
their reviews electronically.

The electronic process probably required more of the reviewers' time
than a hardcopy process would have.  Anticipating this, I kept the
reviewer load low, with an average of roughly 4 papers per reviewer.
As mentioned above, all of the reviewers are experienced reviewers.

ACL is at an awkward stage in its growth.  It is too small to support
office staff to handle submissions, distribution to reviewers, etc.,
but it is large enough that handling the submissions and reviews is a
significant burden for the program chairs.  Between the two
conferences, NMSU had to handle almost 300 submissions, with little
time to prepare.  In addition, the time requirements for area chairs
are quite significant.  The electronic software made the tasks of the
program and area chairs feasible.  In my view, the reviewers had to
spend a modest amount of extra time to deal with the software, which
was mitigated by the lower than average reviewer load.  Although
specifying the software and overseeing its development was
time-consuming, overall the savings in my and the area chairs' time
was significant.

I am grateful to the Computing Research Laboratory for donating
significant internal administrative and technical support to
ANLP-NAACL.  Specifically, Linda Fresques worked on all aspects of the
process, from managing the submissions to preparing the proceedings,
and Eli Kane developed and maintained the review assignment and
submission software. Lori Allen and Jacob Feddersen also helped with
the submission process.

We received more positive than negative feedback about the software.
We received negative feedback from two people.

In the instructions to reviewers, the reviewers were given the option
to receive hardcopies of the papers.  One person, Karen Sparck-Jones,
took us up on this offer, after having been unsuccessful printing the
papers from the Web.  At she  requested, she was also able to submit her
reviews in e-mail.  She asked that her comments be passed to the
executive committee, so I append them below.  

In addition, David Tram was negative about the software.  I also append
his comments below.  In my opinion, Eli did an excellent job on the
software given the time-frame available.  Perhaps David's comments
will be useful to future program chairs.

7.  Acknowledgements

Many thanks to Marie Meteer for her service and support in her role as
General Chair.  I would also like to express my gratitude to Kathy
McCoy and Priscilla Rasmussen for their consistent support and
guidance, and to the ACL and NAACL Executive Committees.