Diversification of papers
Diversification of ACL Paper Types
Some first observations (some coming from a short meeting at ACL ‘08 with Alex & Kathy, some from myself, etc.) and some related questions:
− The inclusion of ‘other’ (until we decide how to call/characterise them) types of papers as pertinent to ACL Conferences reflects a change in culture. Not everyone may like it
o Does it mean that we should ask the memebership? o Or we go ahead? I’m for this
− Which are the possible ‘other’ types? Just some examples:
o Models that incorporate (more) linguistic, cognitive, psychological, insights, e.g. discourse, dialogue, generation, … o New challenging ideas, also controversial ones, potentially influential: novelty of the problem, new formulation of a task. Results not guaranteed. o Position papers o Critical survey, state-of-art o Papers dealing with applications o Language resources, data analysis, infrastructural initiatives. Big projects. Papers on evaluation methodologies. (These topics are in reality those typically covered by LREC). o Contributions to / from other fields
− How to characterise, distinguish the difference in types? Can be done along different dimensions, e.g.: o Theoretical vs. empirical o Theory vs. practice o Qualitative vs. quantitative o Exploratory research of new approaches vs. engineering of existing approaches o Surveys/strategies/infrastructures/… vs. normal research papers
Once we accept the position that “other” types of papers are welcome, what to do wrt the review forms, which is one of the major means to enable diversity:
− One review form with no need to fill all of it? Or multiple forms? o We would prefer one modular form with some alternative questions: e.g. the question on evaluation should be used only if it is pertinent, without penalising the overall score if it is not; and another question could be used for the ‘others’
− Which are the not relevant vs. relevant review questions for ‘other’ types? o No evaluation, no empirical validation; no results from corpora o There may be, for some papers, other types of evaluation, e.g. user evaluation o Should we think at possible questions in the review forms for the “other “ papers, or do we leave it to next Chairs?
→ Steps to put this in motion (the change will need some ‘education’ process):
− Announcement of the change of policy, to encourage also the ‘other’ types of submissions, more varied (providing some examples) o Possibly give visibility to this on the ACL website
− Clear message in the first new Calls, that there is a change, describe it, encourage other types, …
o New review forms (already those at last ACL start going in this direction)
o Publish the new review forms with the Call, for all to see them??
o Should authiors indicate a “paper type”? (not sure about this)
− Chairs should pay attention in the choice of reviewers, to have also some who may be sensible to the new types of submissions (maybe also some from other disciplines? this may be difficult)
o Instructions to reviewers
− To be put in place when? For 2010?
Other comments, some related to other committees, now probably to be dealt in this committee
− The ‘other’ type od papers are orthogonal wrt long & short papers, and also orthogonal wrt papers & posters (for which see below)
− This proposal may open ACL conferences to something different from only the “trend or fashion of the moment”, with the possible desired effect of being more stimulating (obviously nothing is without risks…), less boring, with more topics relevant for discussion
− Does this change go together with an attempt for ACL conferences to be more inclusive? Does it affect the acceptance rate, with a light increase? (not necessarily, but it’s another issue to be discussed, I think, together with this)
o E.g. some argue that too low acceptance hurts participation o Higher acceptance rate could be achieved gradually, with the possibility also of evaluating the results and how it is perceived by the community o Mark pointed out that even ACL conference papers are not journal papers, this should be taken into consideration
+++++++++++
I attach here also some points related to other issues, coming from some of the old mails (to some I attach a comment). I do that because obviously all these issues are interrelated, and together give one or another image of a Conference.
Orals vs. Posters; Long vs. Short; Late-breaking
Some general questions:
− Should a definition be given of the various types?
− What level of decision does pertain to the Exec? And what to individual ACL chairs?
− Acceptance rates for the various types?
Orals vs. Posters:
− There has been a proposal (in some mails) of aiming for e.g. 100 posters vs. 50 orals. This is a completely orthogonal issue.
o Do we discuss this?
− Do we want to have separate submissions tracks? Separate reviewing?
o I would prefer one submission track, authors express their preference (for oral or poster), but reviewers (just one set of reviewers) can change this proposing what type of presentation is better for each paper. o Guidelines should be given
− How oral vs. poster interacts with long vs. short papers?
o For Orals it’s clear. o For Posters? This is not clear to me. Not grouping posters by long vs short instead than by topics.
Posters
− Equal value as Orals
o Is it commonly accepted?
− Posters could/should be grouped in ‘thematic sessions’ each with a chair.
o This is what we do at LREC, and I find it good. It also gives more the impression of an equal treatment between the different papers. o Participants should have a plan of the various thematic sessions in the handbook, to facilitate navigation o Some said that this may be a disadvantage for presenters because they can’t look at the others. But still I think this is better for partipants at large o The other possibility proposed by some is to mix on purpose all the topics, but still grouping them in clusters and making them visible (e.g. with colors). This seems to me more complicated
− Posters could have some kind of summary or 1-few minutes promo during a plenary oral session, or at the poster session itself (as at EACL ’09)
o Isn’t it redundant, or giving even more space to posters? Doesn’t work with many posters. Is it more informative than looking at summaries? o It doesn’t work with parallel sessions, you are here or there …
− The poster thematic sessions could also be in parallel, as are the oral sessions, both among themselves and with orals
− Some of the ‘best papers’ could also be a poster
Long vs. Short
− Is the distinction of long vs. short papers to be retained?
o Some argue no. This should be discussed. E.g. the motivation, as reminded by Owen, that Speech people are used to 4 page papers is to be considered. o On the other side we must be careful in not multiplying categories too much (unless really useful/important), and with them maybe sub-committees.
Late-breaking
− Another category is the ‘late-breaking’ papers? Orthogonal to the others
o To be discussed depending also on other choices
Other comments
− Maybe some experiments could be done with different configurations of all these categories, as in fact it’s already done in different conferences.
o To have them all together may be confusing? And add to an image of “bureaucratic”?
− In parallel with all these distinctions is the topic of when is submission for the different types of papers, for the various conferences in a year. This could/should be well organised for 2010.