Sentence Ordering Using Recurrent Neural Networks

An application of RNN about Sentence Ordering by Lajanugen Logeswaran, Honglak Lee & Dragomir Radev in University of Michigan for ICLR2017

Abstract

• Background: Modeling the structure of conherent texts is a task of great importance in NLP.
• Task: The task of organizing a given set of sentences into a coherent order has been commonly used to build and evaluate models that understand such structure.
• Method: In this paper, we propose an end-to-end neural approach based on the recently proposed set to sequence mapping framework to address the sentence ordering problem.

• Result:

  • Our model achieves state-of-the-art performance in the order discrimination task on two datasets widely used in the literature.
  • They also consider a new interesting task of ordering abstracts from conference papers and research proposals and demonstrate strong performance against recent methods.

• Visual: Visualizing the sentence representations learned by the model shows that the model has captured high level logical structure in these paragraphs.

• Lucky: The model also learn rich semantic sentence representations by learning to order texts, performing conparably to recent unsupervised representation learning methods in the sentence similarity and paraphrase detection tasks.

Introduction

• Background: Modeling the structure of coherent texts is one of the central problems in NLP. A well written piece of text has a particular high level logical and topical structure to it. The actual word and sentence choices as well as their transitions come together to convey the purpose of the text. Our overaching goal is to build models that can learn such structure by learning to arrange a given set of sentence to make coherent text.
• Application of Sentence Ordering task: Multi-document Summarization(MDS) and retrieval based question answering involve extracting information from multiple source documents and organizing and content into a conherent summary. Since the relative ordering about sentences that come from different sources can be unclear, being able to automatically evaluate a particular order and/or finding the optimal order is essential.
• Lucky: The models that learn to order text fragments can also be used as models of coherence. It can be considered as a jigsaw puzzle of sorts in the language domain.

• Driven:

  • capturing sematics using distributed representations
  • Using RNNs for sequence modeling tasks

• Embeddings: Success in unsupervised approaches for learning embeddings for textual entities from large text corpora altered the way NLP problems are studied today. These embeddings have been shown to capture syntactic and semantic information as well as higher level analogical structure. These methods have been adopted to learn vector representations of sentences, paragraphs and entire documents. Embedding based approaches allow models to be trained end-to-end from scratch with no handcrfting.

• Recurrent Neural Networks(RNNs) have become the de facto approach to sequence learning and mapping problems in recent times. The sequence to sequence mapping framework, as well as several of its variants have fuelled RNN based approaches to a wide variety of problems including language modeling, language generation, machine translation, question answering and many others.

Related Work

• Coherence modeling and sentence ordering
  • The coherence modeling and sentence ordering tasks have been approached by closely related techniques.