Structure Modeling for Natural Language Processing

Download or read book Structure Modeling for Natural Language Processing written by Jie Hao and published by . This book was released on 2020 with total page 0 pages. Available in PDF, EPUB and Kindle. Book excerpt: As the rise in availability of natural language data, the underlying language structures can be better learned and play the important roles in many natural language processing tasks. Although the neural language representation models like Transformer trained on large-scale corpora have achieved amazing performance on different natural language processing (NLP) tasks, how to further incorporate the structural knowledge information is not well explored. In this thesis, we propose to explore the structure modeling for existing powerful neural models of natural language via explicitly and implicitly ways, in order to further boost the performance of the models.We describe three general approaches for incorporating structure information into the Transformer, the state of the art model of many NLP tasks. The first method is mainly based on Recurrent Neural Networks (RNNs) and we propose a novel Attentive Recurrent Networks (ARNs) to introduce the recurrence into Transformer. The second method leverages the RNNs' variants ordered neuron Long short-term memory (ON-LSTM). The third method leverages multi granularity phrases information of the sequences, which enables Transformer to capture different segments structure from words to phrases. The linguistic representations learned as a result of structure modeling are shown to be effective across a range of downstream tasks such as neural machine translation (NMT) and text classification. We validate our approaches across a range of tasks, including machine translation, targeted linguistic evaluation, language modeling and logical inference. While machine translation is a benchmark task for deep learning models, the other tasks focus on evaluating how much structure information is encoded in the learned representations and how it can affect models. Experimental results show that the proposed approach consistently improves performances in all tasks, and modeling structure is indeed an essential method for further improving the performance of the NLP models such as Transformer. Furthermore, in the last part of the thesis, we conduct a series of experiments to analyze the importance of syntax information in NLP tasks. In detail, we investigate the role of syntax in NMT and language modeling. More specific, we adopt the On-Lstm decoder, which can be used to induce the latent structure of natural language, to integrate the syntax information into the state-of-the-art Transformer model. Then, by conducting fluency and adequacy evaluation experiments, we illustrate the role of the syntax information in such tasks. Our analysis shade the lights on the role of syntax for NLP tasks especially for the sentence generation in machine translation.