\section{Layer\+Norm$<$ Input\+Data\+Type, Output\+Data\+Type $>$ Class Template Reference} \label{classmlpack_1_1ann_1_1LayerNorm}\index{Layer\+Norm$<$ Input\+Data\+Type, Output\+Data\+Type $>$@{Layer\+Norm$<$ Input\+Data\+Type, Output\+Data\+Type $>$}} Declaration of the Layer Normalization class. \subsection*{Public Member Functions} \begin{DoxyCompactItemize} \item \textbf{ Layer\+Norm} () \begin{DoxyCompactList}\small\item\em Create the \doxyref{Layer\+Norm}{p.}{classmlpack_1_1ann_1_1LayerNorm} object. \end{DoxyCompactList}\item \textbf{ Layer\+Norm} (const size\+\_\+t size, const double eps=1e-\/8) \begin{DoxyCompactList}\small\item\em Create the \doxyref{Layer\+Norm}{p.}{classmlpack_1_1ann_1_1LayerNorm} object for a specified number of input units. \end{DoxyCompactList}\item {\footnotesize template$<$typename eT $>$ }\\void \textbf{ Backward} (const arma\+::\+Mat$<$ eT $>$ \&input, const arma\+::\+Mat$<$ eT $>$ \&gy, arma\+::\+Mat$<$ eT $>$ \&g) \begin{DoxyCompactList}\small\item\em Backward pass through the layer. \end{DoxyCompactList}\item Output\+Data\+Type const \& \textbf{ Delta} () const \begin{DoxyCompactList}\small\item\em Get the delta. \end{DoxyCompactList}\item Output\+Data\+Type \& \textbf{ Delta} () \begin{DoxyCompactList}\small\item\em Modify the delta. \end{DoxyCompactList}\item double \textbf{ Epsilon} () const \begin{DoxyCompactList}\small\item\em Get the value of epsilon. \end{DoxyCompactList}\item {\footnotesize template$<$typename eT $>$ }\\void \textbf{ Forward} (const arma\+::\+Mat$<$ eT $>$ \&input, arma\+::\+Mat$<$ eT $>$ \&output) \begin{DoxyCompactList}\small\item\em Forward pass of Layer Normalization. \end{DoxyCompactList}\item {\footnotesize template$<$typename eT $>$ }\\void \textbf{ Gradient} (const arma\+::\+Mat$<$ eT $>$ \&input, const arma\+::\+Mat$<$ eT $>$ \&error, arma\+::\+Mat$<$ eT $>$ \&gradient) \begin{DoxyCompactList}\small\item\em Calculate the gradient using the output delta and the input activations. \end{DoxyCompactList}\item Output\+Data\+Type const \& \textbf{ Gradient} () const \begin{DoxyCompactList}\small\item\em Get the gradient. \end{DoxyCompactList}\item Output\+Data\+Type \& \textbf{ Gradient} () \begin{DoxyCompactList}\small\item\em Modify the gradient. \end{DoxyCompactList}\item size\+\_\+t \textbf{ In\+Size} () const \begin{DoxyCompactList}\small\item\em Get the number of input units. \end{DoxyCompactList}\item Output\+Data\+Type \textbf{ Mean} () \begin{DoxyCompactList}\small\item\em Get the mean across single training data. \end{DoxyCompactList}\item Output\+Data\+Type const \& \textbf{ Output\+Parameter} () const \begin{DoxyCompactList}\small\item\em Get the output parameter. \end{DoxyCompactList}\item Output\+Data\+Type \& \textbf{ Output\+Parameter} () \begin{DoxyCompactList}\small\item\em Modify the output parameter. \end{DoxyCompactList}\item Output\+Data\+Type const \& \textbf{ Parameters} () const \begin{DoxyCompactList}\small\item\em Get the parameters. \end{DoxyCompactList}\item Output\+Data\+Type \& \textbf{ Parameters} () \begin{DoxyCompactList}\small\item\em Modify the parameters. \end{DoxyCompactList}\item void \textbf{ Reset} () \begin{DoxyCompactList}\small\item\em Reset the layer parameters. \end{DoxyCompactList}\item {\footnotesize template$<$typename Archive $>$ }\\void \textbf{ serialize} (Archive \&ar, const unsigned int) \begin{DoxyCompactList}\small\item\em Serialize the layer. \end{DoxyCompactList}\item Output\+Data\+Type \textbf{ Variance} () \begin{DoxyCompactList}\small\item\em Get the variance across single training data. \end{DoxyCompactList}\end{DoxyCompactItemize} \subsection{Detailed Description} \subsubsection*{template$<$typename Input\+Data\+Type = arma\+::mat, typename Output\+Data\+Type = arma\+::mat$>$\newline class mlpack\+::ann\+::\+Layer\+Norm$<$ Input\+Data\+Type, Output\+Data\+Type $>$} Declaration of the Layer Normalization class. The layer transforms the input data into zero mean and unit variance and then scales and shifts the data by parameters, gamma and beta respectively over a single training data. These parameters are learnt by the network. Layer Normalization is different from Batch Normalization in the way that normalization is done for individual training cases, and the mean and standard deviations are computed across the layer dimensions, as opposed to across the batch. For more information, refer to the following papers, \begin{DoxyCode} @article\{Ba16, author = \{Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey E. Hinton\}, title = \{Layer Normalization\}, volume = \{abs/1607.06450\}, year = \{2016\}, url = \{http:\textcolor{comment}{//arxiv.org/abs/1607.06450\},} eprint = \{1607.06450\}, \} \end{DoxyCode} \begin{DoxyCode} @article\{Ioffe15, author = \{Sergey Ioffe and Christian Szegedy\}, title = \{Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift\}, journal = \{CoRR\}, volume = \{abs/1502.03167\}, year = \{2015\}, url = \{http:\textcolor{comment}{//arxiv.org/abs/1502.03167\},} eprint = \{1502.03167\}, \} \end{DoxyCode} \begin{DoxyTemplParams}{Template Parameters} {\em Input\+Data\+Type} & Type of the input data (arma\+::colvec, arma\+::mat, arma\+::sp\+\_\+mat or arma\+::cube). \\ \hline {\em Output\+Data\+Type} & Type of the output data (arma\+::colvec, arma\+::mat, arma\+::sp\+\_\+mat or arma\+::cube). \\ \hline \end{DoxyTemplParams} Definition at line 65 of file layer\+\_\+norm.\+hpp. \subsection{Constructor \& Destructor Documentation} \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_a8c32833275f5be5c6b93c984cab337d1}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Layer\+Norm@{Layer\+Norm}} \index{Layer\+Norm@{Layer\+Norm}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Layer\+Norm()\hspace{0.1cm}{\footnotesize\ttfamily [1/2]}} {\footnotesize\ttfamily \textbf{ Layer\+Norm} (\begin{DoxyParamCaption}{ }\end{DoxyParamCaption})} Create the \doxyref{Layer\+Norm}{p.}{classmlpack_1_1ann_1_1LayerNorm} object. \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_a03d21dc57fa62684657612597373a677}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Layer\+Norm@{Layer\+Norm}} \index{Layer\+Norm@{Layer\+Norm}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Layer\+Norm()\hspace{0.1cm}{\footnotesize\ttfamily [2/2]}} {\footnotesize\ttfamily \textbf{ Layer\+Norm} (\begin{DoxyParamCaption}\item[{const size\+\_\+t}]{size, }\item[{const double}]{eps = {\ttfamily 1e-\/8} }\end{DoxyParamCaption})} Create the \doxyref{Layer\+Norm}{p.}{classmlpack_1_1ann_1_1LayerNorm} object for a specified number of input units. \begin{DoxyParams}{Parameters} {\em size} & The number of input units. \\ \hline {\em eps} & The epsilon added to variance to ensure numerical stability. \\ \hline \end{DoxyParams} \subsection{Member Function Documentation} \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_a78dbad83871f43db1975e45a9a69c376}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Backward@{Backward}} \index{Backward@{Backward}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Backward()} {\footnotesize\ttfamily void Backward (\begin{DoxyParamCaption}\item[{const arma\+::\+Mat$<$ eT $>$ \&}]{input, }\item[{const arma\+::\+Mat$<$ eT $>$ \&}]{gy, }\item[{arma\+::\+Mat$<$ eT $>$ \&}]{g }\end{DoxyParamCaption})} Backward pass through the layer. \begin{DoxyParams}{Parameters} {\em input} & The input activations. \\ \hline {\em gy} & The backpropagated error. \\ \hline {\em g} & The calculated gradient. \\ \hline \end{DoxyParams} \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_a797f7edb44dd081e5e2b3cc316eef6bd}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Delta@{Delta}} \index{Delta@{Delta}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Delta()\hspace{0.1cm}{\footnotesize\ttfamily [1/2]}} {\footnotesize\ttfamily Output\+Data\+Type const\& Delta (\begin{DoxyParamCaption}{ }\end{DoxyParamCaption}) const\hspace{0.3cm}{\ttfamily [inline]}} Get the delta. Definition at line 130 of file layer\+\_\+norm.\+hpp. \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_ad6601342d560219ce951d554e69e5e87}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Delta@{Delta}} \index{Delta@{Delta}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Delta()\hspace{0.1cm}{\footnotesize\ttfamily [2/2]}} {\footnotesize\ttfamily Output\+Data\+Type\& Delta (\begin{DoxyParamCaption}{ }\end{DoxyParamCaption})\hspace{0.3cm}{\ttfamily [inline]}} Modify the delta. Definition at line 132 of file layer\+\_\+norm.\+hpp. \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_af6d960193bb5db37e51416e12bf720de}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Epsilon@{Epsilon}} \index{Epsilon@{Epsilon}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Epsilon()} {\footnotesize\ttfamily double Epsilon (\begin{DoxyParamCaption}{ }\end{DoxyParamCaption}) const\hspace{0.3cm}{\ttfamily [inline]}} Get the value of epsilon. Definition at line 149 of file layer\+\_\+norm.\+hpp. References Layer\+Norm$<$ Input\+Data\+Type, Output\+Data\+Type $>$\+::serialize(). \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_a461f849bc638c15bec262dc9c3a58abe}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Forward@{Forward}} \index{Forward@{Forward}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Forward()} {\footnotesize\ttfamily void Forward (\begin{DoxyParamCaption}\item[{const arma\+::\+Mat$<$ eT $>$ \&}]{input, }\item[{arma\+::\+Mat$<$ eT $>$ \&}]{output }\end{DoxyParamCaption})} Forward pass of Layer Normalization. Transforms the input data into zero mean and unit variance, scales the data by a factor gamma and shifts it by beta. \begin{DoxyParams}{Parameters} {\em input} & Input data for the layer. \\ \hline {\em output} & Resulting output activations. \\ \hline \end{DoxyParams} \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_aaf577db350e2130754490d8486fba215}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Gradient@{Gradient}} \index{Gradient@{Gradient}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Gradient()\hspace{0.1cm}{\footnotesize\ttfamily [1/3]}} {\footnotesize\ttfamily void Gradient (\begin{DoxyParamCaption}\item[{const arma\+::\+Mat$<$ eT $>$ \&}]{input, }\item[{const arma\+::\+Mat$<$ eT $>$ \&}]{error, }\item[{arma\+::\+Mat$<$ eT $>$ \&}]{gradient }\end{DoxyParamCaption})} Calculate the gradient using the output delta and the input activations. \begin{DoxyParams}{Parameters} {\em input} & The input activations. \\ \hline {\em error} & The calculated error. \\ \hline {\em gradient} & The calculated gradient. \\ \hline \end{DoxyParams} \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_a0f1f4e6d93472d83852731a96c8c3f59}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Gradient@{Gradient}} \index{Gradient@{Gradient}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Gradient()\hspace{0.1cm}{\footnotesize\ttfamily [2/3]}} {\footnotesize\ttfamily Output\+Data\+Type const\& Gradient (\begin{DoxyParamCaption}{ }\end{DoxyParamCaption}) const\hspace{0.3cm}{\ttfamily [inline]}} Get the gradient. Definition at line 135 of file layer\+\_\+norm.\+hpp. \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_a19abce4739c3b0b658b612537e21956a}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Gradient@{Gradient}} \index{Gradient@{Gradient}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Gradient()\hspace{0.1cm}{\footnotesize\ttfamily [3/3]}} {\footnotesize\ttfamily Output\+Data\+Type\& Gradient (\begin{DoxyParamCaption}{ }\end{DoxyParamCaption})\hspace{0.3cm}{\ttfamily [inline]}} Modify the gradient. Definition at line 137 of file layer\+\_\+norm.\+hpp. \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_adc2669016a821704c4c4aeb4651e9e87}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!In\+Size@{In\+Size}} \index{In\+Size@{In\+Size}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{In\+Size()} {\footnotesize\ttfamily size\+\_\+t In\+Size (\begin{DoxyParamCaption}{ }\end{DoxyParamCaption}) const\hspace{0.3cm}{\ttfamily [inline]}} Get the number of input units. Definition at line 146 of file layer\+\_\+norm.\+hpp. \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_ac85d687fc82569ded8e31b5559d4a1d2}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Mean@{Mean}} \index{Mean@{Mean}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Mean()} {\footnotesize\ttfamily Output\+Data\+Type Mean (\begin{DoxyParamCaption}{ }\end{DoxyParamCaption})\hspace{0.3cm}{\ttfamily [inline]}} Get the mean across single training data. Definition at line 140 of file layer\+\_\+norm.\+hpp. \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_a0ee21c2a36e5abad1e7a9d5dd00849f9}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Output\+Parameter@{Output\+Parameter}} \index{Output\+Parameter@{Output\+Parameter}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Output\+Parameter()\hspace{0.1cm}{\footnotesize\ttfamily [1/2]}} {\footnotesize\ttfamily Output\+Data\+Type const\& Output\+Parameter (\begin{DoxyParamCaption}{ }\end{DoxyParamCaption}) const\hspace{0.3cm}{\ttfamily [inline]}} Get the output parameter. Definition at line 125 of file layer\+\_\+norm.\+hpp. \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_a21d5f745f02c709625a4ee0907f004a5}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Output\+Parameter@{Output\+Parameter}} \index{Output\+Parameter@{Output\+Parameter}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Output\+Parameter()\hspace{0.1cm}{\footnotesize\ttfamily [2/2]}} {\footnotesize\ttfamily Output\+Data\+Type\& Output\+Parameter (\begin{DoxyParamCaption}{ }\end{DoxyParamCaption})\hspace{0.3cm}{\ttfamily [inline]}} Modify the output parameter. Definition at line 127 of file layer\+\_\+norm.\+hpp. \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_aa530552c7ef915c952fbacc77b965c90}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Parameters@{Parameters}} \index{Parameters@{Parameters}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Parameters()\hspace{0.1cm}{\footnotesize\ttfamily [1/2]}} {\footnotesize\ttfamily Output\+Data\+Type const\& Parameters (\begin{DoxyParamCaption}{ }\end{DoxyParamCaption}) const\hspace{0.3cm}{\ttfamily [inline]}} Get the parameters. Definition at line 120 of file layer\+\_\+norm.\+hpp. \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_a9c5c5900772a689d5a6b59778ec67120}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Parameters@{Parameters}} \index{Parameters@{Parameters}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Parameters()\hspace{0.1cm}{\footnotesize\ttfamily [2/2]}} {\footnotesize\ttfamily Output\+Data\+Type\& Parameters (\begin{DoxyParamCaption}{ }\end{DoxyParamCaption})\hspace{0.3cm}{\ttfamily [inline]}} Modify the parameters. Definition at line 122 of file layer\+\_\+norm.\+hpp. \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_a372de693ad40b3f42839c8ec6ac845f4}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Reset@{Reset}} \index{Reset@{Reset}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Reset()} {\footnotesize\ttfamily void Reset (\begin{DoxyParamCaption}{ }\end{DoxyParamCaption})} Reset the layer parameters. \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_af0dd9205158ccf7bcfcd8ff81f79c927}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!serialize@{serialize}} \index{serialize@{serialize}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{serialize()} {\footnotesize\ttfamily void serialize (\begin{DoxyParamCaption}\item[{Archive \&}]{ar, }\item[{const unsigned}]{int }\end{DoxyParamCaption})} Serialize the layer. Referenced by Layer\+Norm$<$ Input\+Data\+Type, Output\+Data\+Type $>$\+::\+Epsilon(). \mbox{\label{classmlpack_1_1ann_1_1LayerNorm_a82e95c95f5e63bbcab06a042b5bdfea1}} \index{mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}!Variance@{Variance}} \index{Variance@{Variance}!mlpack\+::ann\+::\+Layer\+Norm@{mlpack\+::ann\+::\+Layer\+Norm}} \subsubsection{Variance()} {\footnotesize\ttfamily Output\+Data\+Type Variance (\begin{DoxyParamCaption}{ }\end{DoxyParamCaption})\hspace{0.3cm}{\ttfamily [inline]}} Get the variance across single training data. Definition at line 143 of file layer\+\_\+norm.\+hpp. The documentation for this class was generated from the following file\+:\begin{DoxyCompactItemize} \item /var/www/mlpack.\+ratml.\+org/mlpack.\+org/\+\_\+src/mlpack-\/3.\+3.\+2/src/mlpack/methods/ann/layer/\textbf{ layer\+\_\+norm.\+hpp}\end{DoxyCompactItemize}