Jason D. Williams

Also published as: Jason D Williams, Jason Williams


2023

pdf bib
DELPHI: Data for Evaluating LLMs’ Performance in Handling Controversial Issues
David Sun | Artem Abzaliev | Hadas Kotek | Christopher Klein | Zidi Xiu | Jason Williams
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track

Controversy is a reflection of our zeitgeist, and an important aspect to any discourse. The rise of large language models (LLMs) as conversational systems has increased public reliance on these systems for answers to their various questions. Consequently, it is crucial to systematically examine how these models respond to questions that pertaining to ongoing debates. However, few such datasets exist in providing human-annotated labels reflecting the contemporary discussions. To foster research in this area, we propose a novel construction of a controversial questions dataset, expanding upon the publicly released Quora Question Pairs Dataset. This dataset presents challenges concerning knowledge recency, safety, fairness, and bias. We evaluate different LLMs using a subset of this dataset, illuminating how they handle controversial issues and the stances they adopt. This research ultimately contributes to our understanding of LLMs’ interaction with controversial issues, paving the way for improvements in their comprehension and handling of complex societal debates.

pdf bib
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)
Sunayana Sitaram | Beata Beigman Klebanov | Jason D Williams
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)

2021

pdf bib
Noise Robust Named Entity Understanding for Voice Assistants
Deepak Muralidharan | Joel Ruben Antony Moniz | Sida Gao | Xiao Yang | Justine Kao | Stephen Pulman | Atish Kothari | Ray Shen | Yinying Pan | Vivek Kaul | Mubarak Seyed Ibrahim | Gang Xiang | Nan Dun | Yidan Zhou | Andy O | Yuan Zhang | Pooja Chitkara | Xuan Wang | Alkesh Patel | Kushal Tayal | Roger Zheng | Peter Grasch | Jason D Williams | Lin Li
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers

Named Entity Recognition (NER) and Entity Linking (EL) play an essential role in voice assistant interaction, but are challenging due to the special difficulties associated with spoken user queries. In this paper, we propose a novel architecture that jointly solves the NER and EL tasks by combining them in a joint reranking module. We show that our proposed framework improves NER accuracy by up to 3.13% and EL accuracy by up to 3.6% in F1 score. The features used also lead to better accuracies in other natural language understanding tasks, such as domain classification and semantic parsing.

2020

pdf bib
Conversational Semantic Parsing for Dialog State Tracking
Jianpeng Cheng | Devang Agrawal | Héctor Martínez Alonso | Shruti Bhargava | Joris Driesen | Federico Flego | Dain Kaplan | Dimitri Kartsaklis | Lin Li | Dhivya Piraviperumal | Jason D. Williams | Hong Yu | Diarmuid Ó Séaghdha | Anders Johannsen
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

We consider a new perspective on dialog state tracking (DST), the task of estimating a user’s goal through the course of a dialog. By formulating DST as a semantic parsing task over hierarchical representations, we can incorporate semantic compositionality, cross-domain knowledge sharing and co-reference. We present TreeDST, a dataset of 27k conversations annotated with tree-structured dialog states and system acts. We describe an encoder-decoder framework for DST with hierarchical representations, which leads to ~20% improvement over state-of-the-art DST approaches that operate on a flat meaning space of slot-value pairs.

pdf bib
Improving Human-Labeled Data through Dynamic Automatic Conflict Resolution
David Q. Sun | Hadas Kotek | Christopher Klein | Mayank Gupta | William Li | Jason D. Williams
Proceedings of the 28th International Conference on Computational Linguistics

This paper develops and implements a scalable methodology for (a) estimating the noisiness of labels produced by a typical crowdsourcing semantic annotation task, and (b) reducing the resulting error of the labeling process by as much as 20-30% in comparison to other common labeling strategies. Importantly, this new approach to the labeling process, which we name Dynamic Automatic Conflict Resolution (DACR), does not require a ground truth dataset and is instead based on inter-project annotation inconsistencies. This makes DACR not only more accurate but also available to a broad range of labeling tasks. In what follows we present results from a text classification task performed at scale for a commercial personal assistant, and evaluate the inherent ambiguity uncovered by this annotation strategy as compared to other common labeling strategies.

2017

pdf bib
Demonstration of interactive teaching for end-to-end dialog control with hybrid code networks
Jason D. Williams | Lars Liden
Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue

This is a demonstration of interactive teaching for practical end-to-end dialog systems driven by a recurrent neural network. In this approach, a developer teaches the network by interacting with the system and providing on-the-spot corrections. Once a system is deployed, a developer can also correct mistakes in logged dialogs. This demonstration shows both of these teaching methods applied to dialog systems in three domains: pizza ordering, restaurant information, and weather forecasts.

pdf bib
Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning
Jason D. Williams | Kavosh Asadi | Geoffrey Zweig
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

End-to-end learning of recurrent neural networks (RNNs) is an attractive solution for dialog systems; however, current techniques are data-intensive and require thousands of dialogs to learn simple behaviors. We introduce Hybrid Code Networks (HCNs), which combine an RNN with domain-specific knowledge encoded as software and system action templates. Compared to existing end-to-end approaches, HCNs considerably reduce the amount of training data required, while retaining the key benefit of inferring a latent representation of dialog state. In addition, HCNs can be optimized with supervised learning, reinforcement learning, or a mixture of both. HCNs attain state-of-the-art performance on the bAbI dialog dataset (Bordes and Weston, 2016), and outperform two commercially deployed customer-facing dialog systems at our company.

2015

pdf bib
Fast and easy language understanding for dialog systems with Microsoft Language Understanding Intelligent Service (LUIS)
Jason D. Williams | Eslam Kamal | Mokhtar Ashour | Hani Amr | Jessica Miller | Geoff Zweig
Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue

2014

pdf bib
The Second Dialog State Tracking Challenge
Matthew Henderson | Blaise Thomson | Jason D. Williams
Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)

pdf bib
Web-style ranking and SLU combination for dialog state tracking
Jason D. Williams
Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)

pdf bib
Discovering Latent Structure in Task-Oriented Dialogues
Ke Zhai | Jason D. Williams
Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

2013

pdf bib
Proceedings of the SIGDIAL 2013 Conference
Maxine Eskenazi | Michael Strube | Barbara Di Eugenio | Jason D. Williams
Proceedings of the SIGDIAL 2013 Conference

pdf bib
Continuously Predicting and Processing Barge-in During a Live Spoken Dialogue Task
Ethan Selfridge | Iker Arizmendi | Peter Heeman | Jason Williams
Proceedings of the SIGDIAL 2013 Conference

pdf bib
The Dialog State Tracking Challenge
Jason Williams | Antoine Raux | Deepak Ramachandran | Alan Black
Proceedings of the SIGDIAL 2013 Conference

pdf bib
Multi-domain learning and generalization in dialog state tracking
Jason Williams
Proceedings of the SIGDIAL 2013 Conference

pdf bib
Discriminative state tracking for spoken dialog systems
Angeliki Metallinou | Dan Bohus | Jason Williams
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

2012

pdf bib
Integrating Incremental Speech Recognition and POMDP-Based Dialogue Systems
Ethan O. Selfridge | Iker Arizmendi | Peter A. Heeman | Jason D. Williams
Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue

pdf bib
A belief tracking challenge task for spoken dialog systems
Jason Williams
NAACL-HLT Workshop on Future directions and needs in the Spoken Dialog Community: Tools and Data (SDCTD 2012)

2011

pdf bib
Spoken Dialog Challenge 2010: Comparison of Live and Control Test Results
Alan W Black | Susanne Burger | Alistair Conkie | Helen Hastie | Simon Keizer | Oliver Lemon | Nicolas Merigaud | Gabriel Parent | Gabriel Schubiner | Blaise Thomson | Jason D. Williams | Kai Yu | Steve Young | Maxine Eskenazi
Proceedings of the SIGDIAL 2011 Conference

pdf bib
Stability and Accuracy in Incremental Speech Recognition
Ethan Selfridge | Iker Arizmendi | Peter Heeman | Jason Williams
Proceedings of the SIGDIAL 2011 Conference

pdf bib
An Empirical Evaluation of a Statistical Dialog System in Public Use
Jason Williams
Proceedings of the SIGDIAL 2011 Conference

2009

pdf bib
Estimating Probability of Correctness for ASR N-Best Lists
Jason Williams | Suhrid Balakrishnan
Proceedings of the SIGDIAL 2009 Conference

2008

pdf bib
Using Automatically Transcribed Dialogs to Learn User Models in a Spoken Dialog System
Umar Syed | Jason Williams
Proceedings of ACL-08: HLT, Short Papers

pdf bib
Demonstration of a POMDP Voice Dialer
Jason Williams
Proceedings of the ACL-08: HLT Demo Session

2007

pdf bib
Applying POMDPs to Dialog Systems in the Troubleshooting Domain
Jason Williams
Proceedings of the Workshop on Bridging the Gap: Academic and Industrial Research in Dialog Technologies

pdf bib
Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT)
Bob Carpenter | Amanda Stent | Jason D. Williams
Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT)

2005

pdf bib
Partially Observable Markov Decision Processes with Continuous Observations for Dialogue Management
Jason D. Williams | Pascal Poupart | Steve Young
Proceedings of the 6th SIGdial Workshop on Discourse and Dialogue

2003

pdf bib
Using Wizard-of-Oz simulations to bootstrap Reinforcement - Learning based dialog management systems
Jason D. Williams | Steve Young
Proceedings of the Fourth SIGdial Workshop of Discourse and Dialogue