\documentclass[12pt]{article}
\usepackage{epic} %allows use of latex graphics
\usepackage{eepic} %allows use of latex graphics
\usepackage{amsmath}
\newcommand{\be}{\begin{equation}} %these definitions save typing
\newcommand{\ee}{\end{equation}}
\newcommand{\bea}{\begin{eqnarray}}
\newcommand{\eea}{\end{eqnarray}}
\newcommand{\pd}[2]{\frac{\partial#1}{\partial#2}}
\newcommand{\pdd}[3]{\frac{\partial^{2}#1}{\partial#2 \partial#3}}
\newcommand{\vc}[1]{\mathbf{#1}}
\textwidth=6.5in
\oddsidemargin=0in
\evensidemargin=0in
\textheight=9.0in
%\topmargin=-.625in 
\topmargin=0.in  
\parindent=0in
\begin{document}
%\pagestyle{empty}
\title{Back propagation algorithm} 
\author{Brian Fiedler}
\date{November 18, 2001}
\maketitle

\abstract{The backpropagation algorithm coded in {\tt nn.f} is briefly summarized.
The notation follows that of Chapter 11 in {\it Neural Network Design} by Hagan, Demuth
and Beale.} 


\section{The neural network (the forward model)}
\settoheight{\parskip}{G}
\parskip=2\parskip
\parindent=0in
A vector of predictors $\vc{p}$ serves as the input vector to
the forward model.
The forward model progresses through $M$ layers,
from layer $m=1$ to $m=M$.  The output from layer $m$ will be
a vector $\vc{a}^m$.  This output vector either serves as either the input vector
to a subsequent layer, or, if it is $\vc{a}^M$,
it is the predictand of the model.  The length of $\vc{a}^m$ is denoted as $N^m$.

To aid with the notation of the algorithm, the input vector is
also denoted as $\vc{a}^0$.  The action of layer $m$ can be described in
two steps.  First, there is a linear transformation of the input vector $\vc{a}^{m-1}$ 
with the multiplication by a weight matrix $\vc{W}^m$ and the addition of a
bias vector $\vc{b}^m$.  Second, each element of the resultant vector $\vc{n}^m$
is operated on by a transfer function $f^m$, which produces the elements of the output vector
$\vc{a}^m$.  Therefore, for $i=1$ to $N^m$,     
% 
\be
n_{i}^{m}=\sum_{j=1}^{N^{m-1}}w_{ij}^{m}a_{j}^{m-1}+b_{i}^{m}
\ee
%
\be
a_{i}^{m}=f^{m}\left( n_{i}^{m}\right) \ . 
\ee
A neural network will usually have at least one nonlinear transfer function,
with the logsig function being a common choice:
\be
f(x) = \frac{1}{1+e^{-cx}} \ , 
\label{ftran}
\ee
where $c$ is a value to be chosen.
Common advice in the use of neural networks
is to scale the predictors to have zero mean and a variance
near unity.  With such scaling, $c=1$ is a workable choice in (\ref{ftran}).

One of the most popular neural networks is a two-layer model,
with a nonlinear transfer function
in only the first layer.
For the second layer, $f^2(x)=x$.

Optimal values for the weight matrices and bias vectors are 
achieved with the use of the use of {\it training data},
which consists of many pairs of {\it input vectors} $\vc{p}$
and {\it target vectors} $\vc{t}$.
For each of the pairs, the error in the prediction is calculated as 
\be
\widehat{F}= \left( \vc{a}^{M}-\vc{t} \right) ^2.
\ee
The sensitivity of this error to changes in each element
of every $\vc{W}^m$ and $\vc{b}^m$ is then calculated using the
chain rule of differential calculus.
Based on the sensitivity analysis, small adjustments are then made to
the weight matrices and bias vectors.  The 
most error reduction with the least
``disruption'' of the existing element occurs if the
adjustments are proportional to the sensitivity.   

The goal of training is to reduce the average
error over all the pairs of input and target vectors. There is little purpose
in dwelling on the optimal adjustment of the weight matrices and bias vectors for a given
pair of input vector and target vector; the same weight matrices and
bias vectors will eventually have to be used for all input vectors.
The adjustment process is kept simple: a small nudge is given with
the use of each data pair.
  With many passes
through the training data set, which consists of many pairs of 
input and target vectors, the accumulation of nudges to the weight matrices
and bias vectors will also tend to reduce the average error.
This iterative procedure is the essence of training a neural network.
The craft of training usually consists of calculating an average error
for the entire training data set each after epoch of training (an epoch
is a pass through the entire data set).
A portion of data pairs is often set aside as a {\it verification data
set}, with which no training is done, but for which an average error is
also calculated.  Monitoring the average error of both the training data set
and the verification data set will may provide a {\it stopping critereon}
for the training, which is often when the error with verification data set
either begins to rise, or rise substantially relative to the training data.
Such a rise is often indicative of ``overfitting'' or
 ``fitting to noise'', meaning fitting to
relations that are not universal, but exist only in the training data set.  
   
\section{Reducing the error}
Here is shown the method to adjust $w_{ij}^{m}$ and $b_{i}^{m}$ in the 
effort to reduce $\widehat{F}$. ``Minimization'' is 
attempted with a simple {\it method of steepest descent}. For each set of $\left(
\bf{p},\bf{t}\right) $, adjustment is done with:
\be
w_{ij}^{m}=w_{ij}^{m}-\alpha \pd{\widehat{F}}{w_{ij}^{m}}
\label{steepw}
\ee
\be
b_{i}^{m}=b_{i}^{m}-\alpha \pd{\widehat{F}}{b_{i}^{m}}.
\label{steepb}
\ee
$\alpha$ is sometimes referred to as the learning rate.
Small values of $\alpha$ cause slow convergence and increase the
computational expense.  Large values of $\alpha$ may overemphasize
adjustment with the current data pair, and global minimization may suffer.
Large values of $\alpha$ may also cause instability and lack of convergence. 
An acceptable value of $\alpha$ may be determined experimentally.
A common practice is to reduce $\alpha$ during the training,
which allows for initial speed of convergence, but ultimately
a training of a neural network that is optimized over all the training data. 

Note that with the use of unscaled, dimensional data, 
the mthod of steepest descent is not dimensionally homogeneous.
Unscaled data, with very different numerical ranges, is known to
cause difficulty in the convergence of the minimization using
the method of steepest descent.
However, the method of steepest descent will work satisfactorily
with scaled variables.

The gradient in the error $\widehat{F}$ is found with
\be
s_{i}^{m}\equiv \pd{\widehat{F}}{n_{i}^{m}}.
\ee
In (\ref{steepw}) and (\ref{steepb}) we need 
\bea
\pd{\widehat{F}}{w_{ij}^{m}}&=&\pd{\widehat{F}}{n_{i}^{m}}
                    \pd{n_{i}^{m}}{w_{ij}^{m}}\\
&=&\pd{\widehat{F}}{n_{i}^{m}}a_{j}^{m-1}\\
 &=&s_{i}^{m}a_{j}^{m-1}.
\eea
and 
\bea
\pd{\widehat{F}}{b_{i}^{m}}&=&\pd{\widehat{F}}{n_{i}^{m}}
                    \pd{n_{i}^{m}}{b_{i}^{m}}\\
 &=&\pd{\widehat{F}}{n_{i}^{m}}\\
&=&s_{i}^{m}.
\eea
But how do we find $s_{i}^{m}$?

\section{Backpropagating the sensitivities}
\bea
s_{i}^{m} &\equiv& \pd{\widehat{F}}{n_{i}^{m}}\\
&=& \sum_{j=1}^{N}\pd{n_{j}^{m+1}}{n_{i}^{m}}
    \pd{\widehat{F}}{n_{j}^{m+1}}\\
&=& \sum_{j=1}^{N}\pd{n_{j}^{m+1}}{n_{i}^{m}}
    s_{j}^{m+1}.\\
\eea
But,
\bea
\pd{n_{j}^{m+1}}{n_{i}^{m}} &=& \sum_{k=1}^{N}\pd{n_{j}^{m+1}}{a_{k}^{m}}
\pd{a_{k}^{m}}{n_{i}^{m}}\\
&=& \pd{n_{j}^{m+1}}{a_{i}^{m}}
\pd{a_{i}^{m}}{n_{i}^{m}}\\
&=&w_{ji}^{m+1}\pd{f\left(n_{i}^{m}\right)}{n_{i}^{m}}.
\eea
So,
\be
s_{i}^{m}
=\pd{f^{m}\left(n_{i}^{m}\right)}{n_{i}^{m}} \sum_{j=1}^{N}s_{j}^{m+1}
w_{ji}^{m+1}.
\ee
Back propagation is started with $s_{i}^{M}$, obtained with:
\bea
s_{i}^{M}
& = & -2\left(t_{i}-a_{i}^{M}\right)\pd{a_{i}^{M}}{n_{i}^{M}}\\
&=& 
-2\left(t_{i}-a_{i}^{M}\right)
\pd{f^{M}\left(n_{i}^{M}\right)}{n_{i}^{M}}.
\eea

\end{document}
