CALL FOR PARTICIPATION
Text Analysis Conference -- Knowledge Base Population
TAC KBP 2013
http://www.nist.gov/tac/2013/KBP/
Follow us on twitter @TACKBP2013
Task Evaluations: February - October 2013
Workshop: November 18-19, 2013
Conducted by: National Institute of Standards and Technology (NIST)
With support from: U.S. Department of Defense
INTRODUCTION
The Text Analysis Conference (TAC) is a series of evaluations and
workshops organized to promote research in Natural Language Processing
and related applications, by providing a large test collection, common
evaluation procedures, and a forum for organizations to share their
results.
The goal of TAC Knowledge Base Population (KBP) is to develop and
evaluate technologies for building and populating knowledge bases
(KBs) about named entities from unstructured text. KBP systems must
either populate an existing reference KB, or else build a KB from
scratch.
You are invited to participate in TAC KBP 2013. Organizations may
choose to participate in any or all of the TAC KBP 2013 tasks. NIST
provides test data for each KBP task, and participants run their NLP
systems on the data and return their results to NIST for evaluation.
TAC KBP culminates in a November workshop at NIST in Gaithersburg,
Maryland, USA.
All results submitted to NIST are archived on the TAC web site, and
all evaluations of submitted results are included in the workshop
proceedings. Dissemination of TAC work and results other than in the
workshop proceedings is welcomed, but the conditions of participation
specifically preclude any advertising claims based on TAC results.
TRACKS
TAC KBP 2013 offers shared tasks in 7 tracks, all aimed at improving
the ability to automatically populate knowledge bases from text:
1. Entity Linking
The entity linking task is to link names in a document collection
to entities in a reference KB, or to new named entities discovered
in the document collection. Tasks are offered in English, Spanish,
and Chinese.
( http://www.nist.gov/tac/2013/KBP/EntityLinking/ )
2. English Slot Filling
The slot filling task is to search a document collection to fill in
values for predefined slots (attributes) for a given entity in a
reference KB.
( http://surdeanu.info/kbp2013/ )
3. Temporal Slot Filling
The goal of temporal slot filling is to augment a KB with temporal
constraints on slot filling relations.
( http://surdeanu.info/kbp2013/ )
4. Cross-lingual Spanish Slot Filling
The Spanish slot filling track extends English slot filling to the
cross-lingual paradigm, in which English and Spanish documents are
used to populate an English reference KB.
( http://www.nist.gov/tac/2013/KBP/SpanishSF/ )
5. Sentiment Slot Filling
The overall goal of the Sentiment Slot Filling track is to assess
the quality of detectors for scoped and attributed sentiment.
( http://www.nist.gov/tac/2013/KBP/SentimentSF/ )
6. Slot Filler Validation
The Slot Filler Validation track focuses on the refinement of
output from slot filling systems by either combining information
from multiple slot filling systems, or applying more intensive
linguistic processing to validate individual candidate slot
fillers.
( http://www.nist.gov/tac/2013/KBP/SFValidation/ )
7. Cold Start KBP
The Cold Start track integrates entity linking and slot filling to
build a knowledge base from scratch.
( http://www.nist.gov/tac/2013/KBP/ColdStart/ )
REGISTRATION
Organizations wishing to participate in any of the TAC KBP 2013 tracks
are invited to register online by June 15, 2013. Participants are
advised to register and submit all required agreement forms as soon as
possible in order to receive timely access to evaluation resources,
including any sample and training data. Registration for a task does
not commit you to participating in the task, but is helpful to know
for planning. Late registration will be permitted only if resources
allow. Any questions about conference participation may be sent to the
TAC project manager: tac-manager [at] nist.gov.
Track registration: http://www.nist.gov/tac/2013/KBP/registration.html
WORKSHOP
The TAC KBP 2013 workshop will be held November 18-19, 2013, in
Gaithersburg, Maryland, USA. The workshop is a forum both for
presentation of results (including failure analyses and system
comparisons), and for more lengthy system presentations describing
techniques used, experiments run on the data, and other issues of
interest to NLP researchers. KBP track participants who wish to give a
presentation during the workshop will submit a short abstract in
September describing the experiments they performed. As there is a
limited amount of time for oral presentations, the abstracts will be
used to determine which participants are asked to speak and which will
present in a poster session.
SCHEDULE
Early May 2013 Track guidelines posted
Early May 2013 Distribution of document collections
June 15, 2013 Deadline for registration for track participation
July-Sept 2013 Track evaluation windows (varies by track)
Late Sept 2013 Release of individual evaluated results to participants
late Sept 2013 Deadline for workshop presentation proposals
Mid Oct 2013 Notification of acceptance of presentation proposals
Oct 27, 2013 Deadline for system reports (workshop notebook version)
Nov 18-19, 2013 TAC KBP 2013 workshop in Gaithersburg, Maryland, USA
early Feb, 2014 Deadline for system reports (final proceedings version)
ORGANIZING COMMITTEE
Javier Artiles (Rakuten Institute of Technology, javart [at] gmail.com)
Hoa Trang Dang (NIST, hoa.dang [at] nist.gov)
Jason Duncan (U.S. Department of Defense, jddunca [at] tycho.ncsc.mil)
Joe Ellis (LDC, joellis [at] ldc.upenn.edu)
Kira Griffitt (LDC, kiragrif [at] ldc.upenn.edu)
Ralph Grishman (New York University, grishman [at] cs.nyu.edu)
Heng Ji (City University of New York, hengji [at] cs.qc.cuny.edu)
Xuansong Li (LDC, xuansong [at] ldc.upenn.edu)
James Mayfield (Johns Hopkins University, james.mayfield [at] jhuapl.edu)
Paul McNamee (Johns Hopkins University, paul.mcnamee [at] jhuapl.edu)
Margaret Mitchell (Johns Hopkins University, m.mitchell [at] jhu.edu)
Boyan Onyshkevych (U.S. Department of Defense, baonysh [at] tycho.ncsc.mil)
Stephanie Strassel (LDC, strassel [at] ldc.upenn.edu)
Mihai Surdeanu (University of Arizona, msurdeanu [at] email.arizona.edu)