Recurrent Neural Networks (RNN)
A typical neural network is a feedforward network which takes in a single large vector and processes in one go. In contrast, a recurrent neural network (RRN) processes sequences by iterating through the sequence elements one at a time and maintaining a state containing information relative to what it has seen so far.
An RNN is a type of neural network that has an internal loop depicted in the following figure:
A lot of time the RRN is illustrated with a sequence of unrolled blocks. The last output at time t-1 is fed into the block of time t.
Why not use a standard NN to for the sequence processing? The problems could be:
- The inputs, outputs can be different lengths in different examples.
- The standard NN does not share features learned across different positions of text.
Forward Propagation of an RNN
References
- Deep Learning with Python by Francois Chollet, Chapter 6