Johanna moore

Guidelines for ACL Program Committee

Conference Handbook - ACL Program Committee

Purpose of Document

Many policy questions came up during the two ACL program committee (PC) meetings I attended (1992 and 1993). In many cases, individuals who had served on the ACL PC in the past or had served on the PCs of other major conferences were able to provide suggestions that enabled decisions to be made and the meeting to progress. These individuals typically cited what was done in the past in similar situations. Unfortunately, what has been done from year to year has not always been consistent, and this led to a general sense that a set of guidelines should be collected and written down to act as a kind of institutional memory that can be passed from PC to PC each year.

To this end, an e-mail message was sent to all members of the ACL ‘93 PC and to several past PC chairs asking for views on a set of questions/issues that came up at the meeting and inviting comments on other PC-related issues. The document that follows is my summary of the responses that I received. I viewed my task to be the collection and organization of information to be passed on to the ACL Executive Committee (EC) in order to aid them in setting policy. I have tried to represent all views on the topics. Additional comments and suggestions are welcome and encouraged.

Definition of the Area

When is a paper relevant for ACL? In some cases, there were papers submitted that seemed like good work in a given discipline (e.g., "pure" linguistics or formal language theory), but it was not clear how that work related to Computational Linguistics.

Suggestions:

• We should take an historical view as to whether the subject of the paper has grown within the CL community and thus ACL is the only reasonable forum for the work. - The burden of explaining how the work is relevant to CL is on the authors and must be explained in the paper.

What to do with "Tools" papers:

There was some discussion about what to do with papers that describe tools. Those surveyed suggested that tools papers should be allowed and encouraged if:

• the tool is demonstrably useful for Computational Linguistics, or
• the tool is based on linguistically interesting methods.

In either case, the paper should not be just an advertisement, but should communicate substantial information about the technology used so that

• others could build the tool too
• it may give people ideas about how to build other useful tools

Some expressed interest in a special section in the ACL proceedings for tools papers, along the lines of the student sessions. Ultimately they’d like to see a special issue of CL on tools too, but suggested we experiment with a session in the ACL for 2--3 years first to see how it works.

Submission Deadline

Every year the PC chair receives several papers that arrive after the deadline and must decide what to do with these. There seems to have been an informal understanding among some in the ACL community that papers received within a "reasonable" time after the deadline would be reviewed and considered for publication in the conference as if they had arrived on time. The decision to review papers received after the deadline and the determination of what constitutes a "reasonable" time are typically left to the discretion of the program chair who could consult the entire committee if he or she chose. In 1993, Len Schubert consulted the committee about this issue and a large majority of the PC members felt strongly that papers received after the deadline indicated in the Call for Papers (CFP) should be returned to the authors unopened. Everyone agreed that a policy for late submissions must be set and made explicit in the CFP. There is a general sense that the rules should be well known and apply equally.

By far the majority still feels that the deadline should be strictly enforced. Many authors work long and hard to submit papers on time. They stay up late working round the clock and send papers via express mail. These papers could no doubt be improved if the author(s) had another few days or a week to polish them. The majority feels that in the interest of fairness, all authors should be subject to the same deadlines.

A very small minority feels that there should be a few days’ grace period, whose exact length should be covertly determined by the PC chair, based on her or his judgement and constraints at the time.

How the Deadline Should be Stated:

Several suggested that in the future the deadline be given in terms of a "postmarked by" date, rather than a "must be received by" date.

Advantages:

• Would allow the members to save literally thousands of dollars on express mail service.
• Would even out the discrepancy between US and overseas submissions.

Potential Problems:

• The main concern is that the worldwide airmail system varies so greatly that this will create unmanageable logistical problems for reviewing. Ordinary airmail even from Canada can take weeks ESPECIALLY in the holiday season. We would have to set the postmark deadline in mid-December to get the usual schedule to work out. This would give authors significantly less time to finish research and write good papers, and they would be doing it in the pre-holiday tumult. Given the vagaries of the world’s postal systems, how could a program chair and his or her committee ever schedule their reviewing? How long do we wait? We will need some cut-off date. For this reason, several people thought the "postmarked by date" was a TERRIBLE IDEA.

E-mail submissions?

Several members of the committee suggested that We allow electronic submissions. This would:

• cut down on shipping costs
• ease distribution to the program committee.
• make it possible for PC members to view the abstracts (or full papers) of all the papers submitted, without ACL or authors incurring massive photocopying bills.

Apparently EACL used standard style files and e-mail submission and distribution in ‘93 and it worked astoundingly well. Reviewers received all of their papers electronically in a tar file, and just had to unpack them and send the postscript images to a printer.

Some pointed out that electronic submission is feasible only for authors who have the capability to get papers into postscript form and to e-mail or ftp their papers to the PC chair. Although this is true of the majority of authors in the USA, Western Europe and Japan, we cannot assume that this is true for all authors. Therefore, we must continue to allow hardcopy submissions. However, we should try to use electronic mail to the extent possible.

For those who do not have access to appropriate e-mail tools for submission, faxing is an option.

Length Restrictions

Submissions that Violate Restrictions:

Every year the PC must tackle the question of what to do with papers that clearly violate the length restriction. While a small number of PC members felt that handling of such papers should be left up to the individual reviewers that received them, the majority felt that all papers should be checked for compliance with length restrictions by an administrator and those that are in clear violation should be returned to authors without review. Many felt strongly that this decision should NOT be left up to individual reviewers as this would lead to inconsistent treatment of papers violating length restrictions.

By far the majority feel that length restrictions must be adhered to. One of the most difficult parts of writing an abstract for something like ACL is compressing ideas into 3200 words. Some PC members were appalled by the fact that the PC reviewed, and even accepted some papers that were more than double the length specified in the CFP. This was considered extremely unfair to authors who may have lessened their chances for acceptance by cutting their papers to meet the length restrictions.

Specification of Length Restrictions:

If we are going to enforce length restrictions, then we need to specify length restrictions in such a way that we can easily check whether or not a paper conforms. Word count is a very hard measure to check. Specifications that detail the font size, margin size, page size, and number of pages seem better. To help authors make sure they meet these guidelines we should provide Latex, Scribe, and MS Word style files that meet ACL specifications. (We could get these from DARPA.)

A problem that many authors have noted is that the abstact length is often longer than the final paper length, leaving authors no room to address reviewers’ comments. If we gave out style files from the beginning, we could alleviate this problem (since authors would know the final camera-ready paper length from the outset, and/or we could require that submissions be 1/2 page shorter than final papers.)

Of course, even if we move to style files, we will need to provide specifications for authors who do not have access to Latex, Scribe, MS Word,etc. Since we expect the number of people in this category to be small and dwindling, this should not be a problem.

Submission to Multiple Conferences

A small number of PC members feel that ACL (and other conferences) should not allow multiple submissions. They feel that submitting to multiple conferences is an abuse of the reviewing process. It wastes reviewers’ time and causes problems with the conference schedule if authors retract their paper from ACL after it has been accepted.

However, the majority feel that allowing multiple submissions is good compromise between conference’s desires to get their programs finalized and authors’ desires to get their work published while it is still fresh. Due to the number of relevant conferences that have submission deadlines in November—February, authors must be allowed to submit to multiple conferences.

All agree that if multiple submissions are allowed, authors must declare this fact up front and agree to publish in only one place. Failure to explicitly declare multiple submissions should be grounds for rejection.

Most feel that it’s up to the PC chair to coordinate with other conferences (e.g., AAAI, COLING, EACL, Cognitive Science) to see that the same paper isn’t going to be presented at more than one of them, and it’s up to reviewers to make sure that papers haven’t been published (outside of workshop proceedings) before.

Managing the Schedule if Multiple Submissions are allowed:

A serious problem is what to do with multiply submitted papers that are accepted to ACL as well as one or more other conferences. There was concern about how this will affect the schedule. Here are several suggestions for the EC to consider:

• Authors of multiple submission MUST declare their intention to withdraw from the OTHER CONFERENCE(s) if their paper is accepted to ACL. This way, the ACL schedule will not be impacted, and this may cut down on multiple submissions.
• Keep some of the papers that were borderline on a waiting list to be used in case any papers are withdrawn.
• Have the program be shorter by N fewer papers if N papers are withdrawn.

What Constitutes Prior Publication?:

A number of questions arose regarding the nature of prior publication of work that would make a paper ineligible for publication in the ACL proceedings. The following suggestions were almost unanimous: Workshops were the hardest to deal with.

One person suggested that guidelines for workshops should take account of:

• the refereeing process for the workshop, and
• who may attend, and
• availability of proceedings.

Most felt that prior presentation at a workshop that did not publish a proceedings, or did not have a serious refereeing process should not prevent a paper from appearing in the ACL proceedings. For example, everyone agrees that workshops like the AAAI symposia for which there are no published proceedings should not preclude publication at ACL. For these workshops, papers are minimally reviewed, participants receive a set of photocopied notes, but these are not published and are not widely available. Some suggest that the criterion should be that "all papers published without further editing" should be eligible for ACL.

However, if papers from a workshop are reviewed, authors are given time to revise their papers, and the papers are published in a book or proceedings, then these papers should not be eligible for publication in ACL. (For example, the AI and Planning Systems Workshop is almost indistinguishable in format from ACL, but it’s called a workshop, so we can’t simply go by the title the organizers give.)

• Papers that are under review for a journal can be submitted to ACL. A gray area that the EC should discuss is how to handle papers that have been accepted to a journal, but won’t appear before ACL.
• The most objectively verifiable approach is to use some notion of what it means to be published. Some suggested that appearance in something with an ISBN or ISSN number constitutes publication. This means previous (or soon to be) appearance in a DARPA proceedings or in a Springer-Verlag-published workshop or conference proceedings counts as a publication.

Some PC members believe that there is an ordering (total?) of forums (i.e., journal > conferences > no-proceedings workshop)>, so that a publication accepted in a "higher or equal level" forum should not be submitted/published again at lower level. This seems reasonable, since ideas become refined as feedback is gained at lower level meetings, and workshops have shorter turnaournd times than conferences and journals, so one can get their work aired while it is new and exciting. However, there seems to be a lot of confusion about ordering. Most agree that it is all right to publish in a conference before a journal, but not vice versa. The EC should take this under consideration.

Whatever is decided, it has been suggested that the following are good rules of thumb:

• for conference organizers: make the policy clear to authors
• for authors: always inform the program chair, or the editor in chief of any submission or prior publication of any similar work (like another paper with a whole section in common).

Special consideration for DARPA workshop?

There is no agreement on this question. Here are some comments the EC may want to take into account when making their decision on this issue:

• The DARPA workshops pose a special problem because in some sense they are not really "optional" for DARPA contractors. With other workshops, it is possible for an author to keep their paper out of the published proceedings to save it for other, more formal, publication, but for DARPA there are political complications.
• DARPA is becoming more and more like a conference, and papers presented at DARPA should not be permitted at ACL.
• The DARPA workshop should not count as a publication, and therefore can be resubmitted. The main reason is that DARPA conferences are not open, for submission or for attendance. (Note: this should also apply to counting a DARPA publication in one’s CV.)

How to Compare Different Reviewers’ Scores:

Several people felt that different reviewers were using different metrics for scoring and therefore it was difficult to compare scores given by different reviewers.

Several suggestions about how to alleviate this problem were made:

1) Papers should be available to all committee members.

Some PC members feel that it is crucial that every PC member have a chance to browse through all the abstracts and review whichever they wish to review. This year PC members saw only the papers they were officially asked to review and, since the committee was large, this number was very small (15-20). Without the context of the other abstracts, PC members felt that they couldn’t tell whether their sample of a given field was representative, or whether a given paper was better or worse than another similar paper that they weren’t officially reviewing and didn’t even know existed until the PC meeting. Some felt that it was this problem that made the PC meeting itself seem rather extraneous, since it was very hard to participate in discussions of papers one hadn’t even seen, and it was hard to have any idea whether the decisions overall were fair.

It was generally agreed that if all papers are made available to all PC members, other PC members might provide useful additional comments on papers for which they were not the primary reviewers, and this would likely strengthen the basis for decisions. However, it was noted that this could allow very ‘aggressive’ PC members to have undue influence in the process by reading all papers and voicing their opinions on them.

2) Some past PC chairs have used spread sheets to calculate statistics such as the mean, mode, and median of each PC members’ scores. This was done at the ACL ‘92 PC meeting and the ACL Applied meeting and was very useful in interpreting reviewers’ ratings.

3) Some PC members said they were accepting papers with technical problems, because the idea was interesting, while others would reject such a paper. Clearly, we want to encourage new and interesting ideas, but being incorrect seems very different than having correct but somewhat preliminary results. We need to attempt to operationalize / standardize the meaning of ratings such as "accept," "marginal" etc.

4) The discrepancy may not be so much between individual reviewers, but between subareas of the program committee. One way to overcome this might be by having more discussion between reviewers, before the plenary program committee meeting. If reviewers in each subarea identify what they think are the best papers in that area, possibly in some kind of order, then the plenary session can be devoted to constructing a program out of that, without having to compare numbers on individual papers issued by individual reviewers.

5) One person suggested that reviewers just be forewarned that for a 5-point scale from definite accept to definite reject, experience shows that very few papers with ratings less than second-highest will ultimately be included. That should discourage reviewers from employing standards they subsequently regret.

Blind Refereeing:

Although many felt that this was less than successful because they were able to guess who the authors of papers were, most felt that ACL should continue this practice. If continued, ACL should give authors more guidelines about how to avoid recognition. In many cases, reviewers were able to identify authors because authors lack experience about how to reference their own work or deal with acknowledgements when submitting a paper for blind review.

Submission by Program Committee

Most agree that members of the PC should be allowed to submit papers and that their papers should be treated in the same manner as all other submissions, with additional care to keep reviewers anonymous.

Some PC members thought this was handled very well in ‘93, where the PC chair notified authors of accept/reject decisions by handing out slips of paper.

Selection of Invited Speakers

Many felt that this is often handled in an ad hoc fashion because it is left to the time that remains after the decisions on papers are made. One person suggested that choosing invited speakers be done by e-mail as soon as the PC is formed in November/December. This will increase our chances of getting the speakers we want. As it is now, we end up asking a relatively famous person in mid-March to commit to speaking in late June. Several times our first choice(s) are already booked.

As to the criteria for selecting invited speakers, several PC members feel that it is preferable to invite speakers outside of the general range of ACL topics.

Committee Size

Although a couple of people found the ‘93 committee consisting of 22 members to be too large, some felt that the larger size had many advantages. It made the reviewing load more modest, allowing more careful reviewing. It also allowed all of the reviewing (save a few instances of delegation) to be done within the committee. This has the crucial advantages that (a) PC members have already set aside time for the job and made a commitment to be punctual, whereas finding reviewers ad hoc and holding them to deadlines is much harder, and (b) all the papers get discussed by all the reviewers at the meeting. Even with the committee of 22, some problems arose in trying to find 3 expert reviewers for some papers.

Need For PC Meeting

Comments in favor:

The whole point of having an in-person meeting is to take advantage of as much of our expertise as possible. Yet it is virtually impossible for a person to have a reasonable opinion to contribute if he or she hasn’t seen a paper. The problems this year were largely due to the fact that most PC members saw only 15—20 of the papers. To make sure every good paper has ample chance to be discovered and every not-so-good paper has ample chance to be correctly evaluated, we should maximize papers’ exposure to members of the PC rather than concluding that we don’t need a PC meeting. We do need a PC meeting, but we nee to come to it having looked at, if not laboriously reviewed, more papers. This is what makes ACL different from other conferences, and different in a good way. It also makes the reviewing process fairer. If we make all the papers available to everyone (see above), the meeting will potentially provide new information.

This year the number of papers accepted pre-meeting was just about the right quantity. Other years might have way too many or too few papers accepted on a first pass. The meeting is useful if papers have to be reconsidered.

Many rating errors come more from personal biases that are overcome by discussion.

The sense of group ownership of the process is very important, and this would be lost in an electronic approach.

The face-to-face meeting is important­unless you have a program committee consisting solely of people who really know each other very well already. It’s perhaps only in retrospect that we think we could have done it all via email since we now know everyone on the program committee. However, I would have liked to have seen more discussion of the program that was emerging, rather than just have discussion of single papers by two or three committee members at a time.

Comments against:

The PC meeting can be held via e-mail. The ACL ‘93 PC committee assigned 2 reviewers to each paper. Reviewers were given a deadline by which they were expected to have written up their reviews and exchanged them with their co-reviewer. If the 2 reviewers could not come to consensus, they asked for a third review. The PC chair selected a third reviewer and typically a decision could be reached. A few members felt that this was sufficient and that no meeting was necessary.

Specific suggestions for how to manage an all electronic PC meeting:

• Titles for ALL papers should be advertised to the entire PC via email: that way, if there are papers that a committee member definitely wants to review, then they can alert the program chair of this. Otherwise, the program chair will assign papers according to specified areas of interest, as was done this year. (Obviously, there will be SOME cases where the program chair has to adjudicate if several people want to review the same paper.)
• As was done this year, the pair of reviewers for each paper will come to a consensus by a specified date, or else will ask for a third reviewer if they cannot reach a consensus. After the third reviews are all in (again by a specified date), the decisions for ALL of the papers will be sent to the entire committee via email (modulo some mechanism to handle the anonymity issue­X may not want Y to know X reviewed Y’s paper). Anyone who wants to object to a decision should have an opportunity to do so, and if necessary should be able to get a FAX of the relevant paper: however, since said person would have had the opportunity the first time around to pick the papers for review, such instances should be relatively infrequent.

Commitment by Program Committee

Several PC members stated that based on their own ACL experiences, and comments they have heard from others at different ACL program committee meetings, it is imperative that all PC members be present at the PC meeting. Having the review is not the same as having the reviewer for the difficult cases.

Makeup of Program Committee

In recent years there has been an attempt to bring new faces onto the ACL PC. While everyone agrees this is important for the community, we must be careful to ensure continuity. Therefore several have suggested that there be an attempt to balance the number of new PC members with persons that have served on the ACL PC in the past.

Panels, Tutorials:

There was a proposal for a panel, yet many committee members felt that panels were not solicited in the CFP so it was unfair. However, in the past there have been panels and no solicitation, and workshops and tutorials seem to be done without a proposal solicitation. AAAI and IJCAI have explicit calls for these things, and many PC members suggest that ACL do the same. A panel, workshop, or tutorial chair could still seek out people as is done now.

Conflict of Interest

Some PC members heard of three cases where PC members were sent papers from their home institution to review. In one case the PC member knew there was a conflict and returned the paper. In another case the paper was kept even though the reviewer suspected it was from their institution (and it was). The members that reported this feel that this must be prohibited in the future. NSF has quite an explicit policy regarding what constitutes a conflict of interest, which might prove instructive.

Author: Johanna Moore, June 1993.