Learning Hierarchical Structures On-The-Fly with a Recurrent-Recursive Model for Sequences

Athul Paul Jacob, Zhouhan Lin, Alessandro Sordoni, Yoshua Bengio


Abstract
We propose a hierarchical model for sequential data that learns a tree on-the-fly, i.e. while reading the sequence. In the model, a recurrent network adapts its structure and reuses recurrent weights in a recursive manner. This creates adaptive skip-connections that ease the learning of long-term dependencies. The tree structure can either be inferred without supervision through reinforcement learning, or learned in a supervised manner. We provide preliminary experiments in a novel Math Expression Evaluation (MEE) task, which is created to have a hierarchical tree structure that can be used to study the effectiveness of our model. Additionally, we test our model in a well-known propositional logic and language modelling tasks. Experimental results have shown the potential of our approach.
Anthology ID:
W18-3020
Volume:
Proceedings of the Third Workshop on Representation Learning for NLP
Month:
July
Year:
2018
Address:
Melbourne, Australia
Editors:
Isabelle Augenstein, Kris Cao, He He, Felix Hill, Spandana Gella, Jamie Kiros, Hongyuan Mei, Dipendra Misra
Venue:
RepL4NLP
SIG:
SIGREP
Publisher:
Association for Computational Linguistics
Note:
Pages:
154–158
Language:
URL:
https://aclanthology.org/W18-3020
DOI:
10.18653/v1/W18-3020
Bibkey:
Cite (ACL):
Athul Paul Jacob, Zhouhan Lin, Alessandro Sordoni, and Yoshua Bengio. 2018. Learning Hierarchical Structures On-The-Fly with a Recurrent-Recursive Model for Sequences. In Proceedings of the Third Workshop on Representation Learning for NLP, pages 154–158, Melbourne, Australia. Association for Computational Linguistics.
Cite (Informal):
Learning Hierarchical Structures On-The-Fly with a Recurrent-Recursive Model for Sequences (Jacob et al., RepL4NLP 2018)
Copy Citation:
PDF:
https://aclanthology.org/W18-3020.pdf