Some Connectionist Models and Their Application to Automatic Speech Recognition

Download or read book Some Connectionist Models and Their Application to Automatic Speech Recognition written by Yoshua Bengio and published by . This book was released on 1990 with total page 31 pages. Available in PDF, EPUB and Kindle. Book excerpt: Abstract: "We attempt to apply some connectionist models to automatic speech recognition. To do so we first consider ways to take advantage of a-priori knowledge in the design of those models. For example we consider the influence on generalization of various preprocessing methods, of the output coding and supervision as well as the architectural design. Recurrent neural networks contain cycles that enable them to retain some information about their past history in order to better predict the next output given the current input. Hence we describe two learning algorithms for these networks, one for general architectures (but not local in time) and one for constrained architectures with self- loops only. Given the importance of cpu requirements for back-propagation algorithms, we discuss some simple methods that can greatly accelerate the convergence of gradient descent with the back-propagation algorithm. In particular we introduce an original technique that provides a different learning rate to different layers of a multi-layered sigmoid network. We then study an alternative type of networks based on Radial Basis Functions (local representation) that can be initialized very fast. We present in detail the results of several experiments with these networks on the recognition of phonemes for the TIMIT databases (speaker-independent, continuous speech database). We propose an acceleration scheme for Radial Basis Functions based on a fast search of the subset of active hidden units. After considering successful networks that combine gaussian units and sigmoid units in a network we propose a cognitively relevant model that combines both a local representation and and [sic] a distributed representation subnetworks to which correspond respectively a fast-learning and a slow-learning capability. This system is based on a reorganization phase during which the information about prototypes and outliers stored in the local subsystem is transferred to the distributed representation subsystem."