mlpack
Public Member Functions | List of all members
RMSPropUpdate Class Reference

RMSProp is an optimizer that utilizes the magnitude of recent gradients to normalize the gradients. More...

Public Member Functions

 RMSPropUpdate (const double epsilon=1e-8, const double alpha=0.99)
 Construct the RMSProp update policy with the given parameters. More...

 
double Alpha () const
 Get the smoothing parameter. More...

 
double & Alpha ()
 Modify the smoothing parameter. More...

 
double Epsilon () const
 Get the value used to initialise the squared gradient parameter. More...

 
double & Epsilon ()
 Modify the value used to initialise the squared gradient parameter. More...

 
void Initialize (const size_t rows, const size_t cols)
 The Initialize method is called by SGD Optimizer method before the start of the iteration update process. More...

 
void Update (arma::mat &iterate, const double stepSize, const arma::mat &gradient)
 Update step for RMSProp. More...

 

Detailed Description

RMSProp is an optimizer that utilizes the magnitude of recent gradients to normalize the gradients.

In its basic form, given a step rate $ \gamma $ and a decay term $ \alpha $ we perform the following updates:

\begin{eqnarray*} r_t &=& (1 - \gamma) f'(\Delta_t)^2 + \gamma r_{t - 1} \\ v_{t + 1} &=& \frac{\alpha}{\sqrt{r_t}}f'(\Delta_t) \\ \Delta_{t + 1} &=& \Delta_t - v_{t + 1} \end{eqnarray*}

For more information, see the following.

@misc{tieleman2012,
title = {Lecture 6.5 - rmsprop, COURSERA: Neural Networks for Machine
Learning},
year = {2012}
}

Definition at line 44 of file rmsprop_update.hpp.

Constructor & Destructor Documentation

◆ RMSPropUpdate()

RMSPropUpdate ( const double  epsilon = 1e-8,
const double  alpha = 0.99 
)
inline

Construct the RMSProp update policy with the given parameters.

Parameters
epsilonThe epsilon value used to initialise the squared gradient parameter.
alphaThe smoothing parameter.

Definition at line 54 of file rmsprop_update.hpp.

Member Function Documentation

◆ Alpha() [1/2]

double Alpha ( ) const
inline

Get the smoothing parameter.

Definition at line 98 of file rmsprop_update.hpp.

◆ Alpha() [2/2]

double& Alpha ( )
inline

Modify the smoothing parameter.

Definition at line 100 of file rmsprop_update.hpp.

◆ Epsilon() [1/2]

double Epsilon ( ) const
inline

Get the value used to initialise the squared gradient parameter.

Definition at line 93 of file rmsprop_update.hpp.

◆ Epsilon() [2/2]

double& Epsilon ( )
inline

Modify the value used to initialise the squared gradient parameter.

Definition at line 95 of file rmsprop_update.hpp.

◆ Initialize()

void Initialize ( const size_t  rows,
const size_t  cols 
)
inline

The Initialize method is called by SGD Optimizer method before the start of the iteration update process.

Parameters
rowsNumber of rows in the gradient matrix.
colsNumber of columns in the gradient matrix.

Definition at line 69 of file rmsprop_update.hpp.

◆ Update()

void Update ( arma::mat &  iterate,
const double  stepSize,
const arma::mat &  gradient 
)
inline

Update step for RMSProp.

Parameters
iterateParameters that minimize the function.
stepSizeStep size to be used for the given iteration.
gradientThe gradient matrix.

Definition at line 82 of file rmsprop_update.hpp.

The documentation for this class was generated from the following file: