Recurrent Neural Networks

Abstract

A recurrent neural network (RNN) is applied to inputs recurrently, with the network output from one time step sending an additional input to the next time step, augmenting the input for that time step. Inputs can be observations recorded at different time steps. Recurrent application of the network enables such networks to detect temporal relationships in input data that have a material impact in modeling output. The network’s output from one time step is passed as an input to the same network at the next time step, along with inputs for that time step. This enables RNNs to pass information learned from one time step to subsequent ones. This chapter illustrates the use of recurrent neural networks by focusing on gated recurrence unit (GRU), long-short-term memory (LSTM) cell, and customized recurrent neural network layers. RNNs are trained using backpropagation through time (BPTT), which involves unrolling the network through time and using backpropagation. Vanishing gradients pose a challenge to training RNNs, as the examples will demonstrate. A LSTM network was proposed by Hochreiter and Schmidhuber in 1997. In 2007, it was applied to speech recognition with outstanding results.