4REAL Workshop
Workshop on Research Results Reproducibility and
Resources Citation in Science and Technology of Language
28 May 2016
Portorož, Slovenia
http://4real.di.fc.ul.pt
Collocated with LREC2016
10th Language Resources and Evaluation Conference
http://lrec2016.lrec-conf.org
Important dates:
15 February 2016: Deadline for submissions
15 March 2016: Notification of authors
15 April 2016: Deadline for camera-ready
28 May 2016: Workshop
Call for papers:
The discussion on research integrity has grown in importance as
the resources allocated to and societal impact of scientific
activities have been expanding (e.g. Stodden, 2013, Aarts et al., 2015),
to the point that it has recently crossed the borders of
the research world and made its appearance in important mass media
and was brought to the attention of the general public
(e.g. Nail and Gautam, 2011, Zimmer, 2012, Begley and Sharon 2012,
The Economist, 2013).
The immediate motivation for this increased interest is to be found
in a number of factors, including the realization that for
some published results, their replication is not being obtained
(e.g. Prinz et al., 2011; Belgley and Ellis, 2012); that there may be
problems with the commonly accepted reviewing procedures, where
deliberately falsified submissions, with fabricated errors and
fake authors, get accepted even in respectable journals
(e.g. Bohannon, 2013); that the expectation of researchers vis a vis
misconduct, as revealed in inquiries to scientists on questionable
practices, scores higher than one might expect or would be ready
to accept (e.g. Fanelli, 2009); among several others.
Doing justice to and building on the inherent ethos of scientific
inquiry, this issue has been under thorough inquiry leading to
a scrutiny on its possible immediate causes and underlying factors,
and to initiatives to respond to its challenge, namely by
the setting up of dedicated conferences (e.g. WCRI – World Conference
on Research Integrity), dedicated journals (e.g. RIPR – Research
Integrity and Peer review), support platforms (e.g. COS – Center for
Open Science), revised and more stringent procedures (e.g. Nature, 2013),
batch replication studies (e.g. Aarts et al, 2015), investigations
on misconduct (e.g. Hvistendahl, 2013), etc.
This workshop seeks to foster the discussion and the advancement
on a topic that has been so far given insufficient attention
in the research area of language processing tools and resources
(Branco, 2013, Fokkens et al., 2013) and that has been an important topic
emerging in other scientific areas. That is the topic of the reproducibility
of research results and the citation of resources, and its impact
on research integrity.
We are thus inviting submissions of articles that present pioneering
cases, either with positive or negative results, of actual replication
exercises of previous published results in our area. We are interested
also in articles discussing the challenges, the risk factors,
the procedures, etc. specific to our area or that should be adopted,
or adapted from other neighboring areas, possibly including of course
the new risks raised by the replication articles themselves and
their own integrity, in view of the preservation of the reputation
of colleagues and works whose results are reported has having been
replicated, etc.
Byu the same token, this workshop is interested also on articles addressing
methodologies for monitoring, maintaining or improving citation of
language resources and tools and to assess the importance of data citation
for research integrity and for the advancement of natural language science
and technology.
Submission:
Please submit your papers at https://www.softconf.com/lrec2016/4REAL.
Accepted papers will be presented as oral presentations or as posters.
All accepted papers will be published in the workshop proceedings.
Papers should be formatted according to the stylesheet to be provided on
the LREC 2016 website and should not exceed 8 pages, including references
and appendices. Papers should be submitted in PDF format through
the URL mentioned above.
When submitting a paper, authors will be asked to provide essential
information,
if this is appropriate for their cases, about resources (in a broad sense,
i.e. also technologies, standards, evaluation kits, etc.) that have been
used
for the work described in the paper or are a new result of your research.
Moreover, ELRA encourages all LREC authors to share the described LRs
(data, tools, services, etc.) to enable their reuse and replicability
of experiments (including evaluation ones).
Organization committee:
António Branco (University of Lisbon)
Nicoletta Calzolari (ILC)
Khalid Choukri (ELRA)
Program committee:
António Branco (University of Lisbon)
Iryna Gurevych (Universität Darmstadt)
Isabel Trancoso (INESC)
Joseph Mariani (CNRS/LIMSI)
Justus Roux (NWVU)
Khalid Choukri (ELRA)
Maria Gavrilidou (ILSP)
Marko Grobelnik (Jozef Stefan Institute)
Marko Tadic (University of Zagreb)
Mike Rosner (University of Malta)
Nicoletta Calzolari (ILC)
Nick Campbell (Trinity College Dublin)
Senja Pollak (Jozef Stefan Institute)
Stelios Piperidis (ILSP)
Steven Krauwer (University of Utrecht)
Thierry Declerck (DFKI)
Torsten Zesch (University of Duisburg-Essen)
Yohei Murakami (Language Grid Japan)
References:
Aarts, et al., 2015, “Estimating the reproducibility of psychological
science”, Science
The Economist, 2013, Unreliable Research: Trouble at the Lab, The Economist,
October 19, 2013, online
Begley, 2012, In cancer science, many "discoveries" don't hold up,
Reuters, March 28th, online
Begley and Ellis, 2012, Drug development: Raise standards for preclinical
cancer research, Nature
Bohannon, John, 2013, Who's Afraid of Peer Review?, Science
Branco, António, 2013, Reliability and Meta-reliability of Language
Resources: Ready to initiate the Integrity Debate?, TLT12
COS, Centre for Open Science
Fanelli,2009 How Many Scientists Fabricate and Falsify Research?
A Systematic Review and Meta-Analysis of Survey Data, PL OS ONE
Fokkens et al., 2013, Offspring from Reproduction Problems: What Replication
Failure Teaches US, ACL.
Hvistendahl, 2013, "China's Publication Bazaar”, Science.
Nail, 2011, Scientists' Elusive Goal: Reproducing Study Results,
The Wall Street Journal.
Nature, 2013, “Announcement: Reducing our irreproducibility”, Nature,
Editorial.
Prinz, et al., 2011, Believe it or not: how much can we rely on published
data on potential drug targets?, Nature Reviews Drug Discovery 10, 712.
RIPR, Research Integrity and Peer Review.
Stodden, 2013, Resolving Irreproducibility in Empirical and Computational
Research, IMS Bulletin Online.
WCRI, World Conference on Research Integrity.
Zimmer, Carl, 2012, A Sharp Rise in Retractions Prompts Calls for Reform,
The New York Times.