Recurrent Neural Networks

Abstract

The information flow in a classical ANN is from the input layer to hidden layers and then to the output layer. An ANN with such a ow of information is referred to as a feed-forward network. Such a network, however, has a major limitation in that it is not able to capture dependencies among sequential observations such as time-series. This is because the network has no memory and, as a result, observations are implicitly assumed to be independent.

Recurrent neural networks (RNNs) are fundamentally different from the feed-forward networks in that they operate not only based on input observations, but also on a set of internal states. The internal states capture the past information in sequences that have already been processed by the network. That is to say, the network includes a cycle that allows keeping past information for an amount of time that is not fixed a-priori and, instead, depends on its input observations and weights. Therefore, in contrast with other common architectures used in deep learning, RNN is capable of learning sequential dependencies extended over time. As a result, it has been extensively used for applications involving analyzing sequential data such as time-series, machine translation, and text and handwritten generation. In this chapter, we first cover fundamental concepts in RNNs including standard RNN, stacked RNN, long short-term memory (LSTM), gated recurrent unit (GRU), and vanishing and exploding gradient problems. We then present an application of RNNs for sentiment classification.