2011Q3 Reports: Workshop Chairs

The ACL workshop co-chairs (John Carroll and Hal Daume III) worked together with the EMNLP workshop chair (Marie Candito) to jointly select workshops for ACL/EMNLP in a single call. The initial call went out on 16 September 2010, and a final call went out on 4 November 2010. The deadline for submission was 15 Nov 2010. Workshops were asked to note whether which conferences were acceptable (ACL versus EMNLP) for their workshop and, if both, which was preferable (if either). Workshop vetting was through a gmail account we created for this process.

In all, we received 32 proposals. Of these, 8 exclusively wanted ACL and 2 exclusively wanted EMNLP. Of those 22 that were okay with either, 11 expressed a preference for ACL and 6 expressed a preference for EMNLP (5 did not express a preference). Overall, it was fairly balanced, which eased the selection process. We made decisions about how many workshops to accept based on discussion with Priscilla Rasmussen and the Conference Chair, Dekang Lin. In the end, 15 were accepted for ACL, of which three were two days long (CoNLL, BioNLP and Annotation); and 7 for EMNLP). For the most part, these were the workshops we felt were the strongest. Unlike 2010, we chose to take into account attendance and did not accept every workshop that would fit. Each workshop was scored (scale of 1 to 3) independently by the three workshop co-chairs. For the most part, the highest averaging workshops were accepted, though some shuffling was done on a phone call between the three chairs. Having the ability to route them either to ACL or EMNLP was very helpful to avoid overlapping workshop topics.

The following workshops were accepted, together with their preregistration numbers as of 12 June: CoNLL (117); BioNLP (84); SSST-5 (72); Social Media (70); Educ Applications (60); Comparable Corpora (50); Text-to-Text (46); Multiword (44); WASSA (43); TextGraphs-6 (43); The LAW V (35); RELMS (34); DiSCo (31); Summarization (31); CMCL (29); LaTeCH (29). A post-hoc analysis suggests that our overall ratings were only moderate predictors of attendance (rho=0.2626, tau=0.1015). However, our own goal in acceptance was not just to maximize attendance, but also to encourage nascent areas.

Overall the selection process worked well. It enabled us to do things like reduce redundancy, have a bigger pool of proposals, and a bigger selection committee. As a result, we recommend a joint model for workshop selection for 2012. However, there were a few issues that arose that we suggest are considered in the future.

• Involve the main conference chairs to define general guidelines on how to weight the various criteria to use for workshop selection: quality of proposal, originality, expected audience, experience of organizers ... Especially things like "Workshop X ran last year and had good attendance. Workshop Y is new but hard to predict. How do we compare the two?"

• Ensure that all notifications of rejection include a few lines of justification, perhaps with a suggestion for re-submission to future conferences. (We omitted to do this, and Hal got a number of queries from rejected workshops asking for feedback.)

• Finally, there was the issue of a workshop that was rejected, but was still very keen to go ahead and so decided to organize separately as a co-located event. We think this is a fine thing to do, so long as it does not organize during the official workshop session. However, we suggest that the ACL board consider creating a policy that strongly discourages co-located events from overlapping with the workshops. (However, it turned out not to be a problem since the workshop ended up not running anyway.)

One of the biggest headaches for the workshop process had to do with handling the workshop proceedings. Some of the groups running the workshops had not previously used ACLPUB and/or the version integrated into START, and were inadequately prepared to deal with all the nuances. The workshop co-chairs had to very quickly try to figure out how to handle this, with numerous emails back and forth to the publications chair. This problem was exacerbated by several workshop organizers not having the right tools to spot some types of incorrectly formatted papers (e.g., not all fonts embedded), and a short timeline (the publications timeline for workshops was very compressed, in comparison to the usual ACL timeline). We think the dividing line for responsibility between the workshop co-chairs and publications chair needs either to be more tightly defined in the ACL guidelines, or established by negotiation between the workshop co-chairs and publications chair in advance. We suggest that since the publications chair is likely the most knowledgable in this area, workshop publications be handled entirely by the publications chair.