\section{Introduction}\label{anntutorial_intro_anntut} There is vast literature on neural networks and their uses, as well as strategies for choosing initial points effectively, keeping the algorithm from converging in local minima, choosing the best model structure, choosing the best optimizers, and so forth. mlpack implements many of these building blocks, making it very easy to create different neural networks in a modular way. mlpack currently implements two easy-\/to-\/use forms of neural networks\+: {\bfseries Feed-\/\+Forward} {\bfseries Networks} (this includes convolutional neural networks) and {\bfseries Recurrent} {\bfseries Neural} {\bfseries Networks}.\section{Table of Contents}\label{anntutorial_toc_anntut} This tutorial is split into the following sections\+: \begin{DoxyItemize} \item \doxyref{Introduction}{p.}{anntutorial_intro_anntut} \item \doxyref{Table of Contents}{p.}{anntutorial_toc_anntut} \item \doxyref{Model A\+PI}{p.}{anntutorial_model_api_anntut} \item \doxyref{Layer A\+PI}{p.}{anntutorial_layer_api_anntut} \item \doxyref{Model Setup \& Training}{p.}{anntutorial_model_setup_training_anntut} \item \doxyref{Saving \& Loading}{p.}{anntutorial_model_saving_loading_anntut} \item \doxyref{Extracting Parameters}{p.}{anntutorial_extracting_parameters_anntut} \item \doxyref{Further documentation}{p.}{anntutorial_further_anntut} \end{DoxyItemize}\section{Model A\+PI}\label{anntutorial_model_api_anntut} There are two main neural network classes that are meant to be used as container for neural network layers that {\bfseries mlpack} implements; each class is suited to a different setting\+: \begin{DoxyItemize} \item {\ttfamily F\+FN\+:} the Feed Forward Network model provides a means to plug layers together in a feed-\/forward fully connected manner. This is the \textquotesingle{}standard\textquotesingle{} type of deep learning model, and includes convolutional neural networks (C\+N\+Ns). \item {\ttfamily R\+NN\+:} the Recurrent Neural Network model provides a means to consider successive calls to forward as different time-\/steps in a sequence. This is often used for time sequence modeling tasks, such as predicting the next character in a sequence. \end{DoxyItemize} Below is some basic guidance on what should be used. Note that the question of \char`\"{}which algorithm should be used\char`\"{} is a very difficult question to answer, so the guidance below is just that---guidance---and may not be right for a particular problem. \begin{DoxyItemize} \item {\bfseries Feed-\/forward} {\bfseries Networks} allow signals or inputs to travel one way only. There is no feedback within the network; for instance, the output of any layer does only affect the upcoming layer. That makes Feed-\/\+Forward Networks straightforward and very effective. They are extensively used in pattern recognition and are ideally suitable for modeling relationships between a set of input and one or more output variables. \item {\bfseries Recurrent} {\bfseries Networks} allow signals or inputs to travel in both directions by introducing loops in the network. Computations derived from earlier inputs are fed back into the network, which gives the recurrent network some kind of memory. R\+N\+Ns are currently being used for all kinds of sequential tasks; for instance, time series prediction, sequence labeling, and sequence classification. \end{DoxyItemize} In order to facilitate consistent implementations, the {\ttfamily F\+FN} and {\ttfamily R\+NN} classes have a number of methods in common\+: \begin{DoxyItemize} \item {\ttfamily Train()}\+: trains the initialized model on the given input data. Optionally an optimizer object can be passed to control the optimization process. \item {\ttfamily Predict()}\+: predicts the responses to a given set of predictors. Note the responses will reflect the output of the specified output layer. \item {\ttfamily Add()}\+: this method can be used to add a layer to the model. \end{DoxyItemize} \begin{DoxyNote}{Note} To be able to optimize the network, both classes implement the Optimizer\+Function A\+PI. In short, the {\ttfamily F\+NN} and {\ttfamily R\+NN} class implement two methods\+: {\ttfamily Evaluate()} and {\ttfamily Gradient()}. This enables the optimization given some learner and some performance measure. \end{DoxyNote} Similar to the existing layer infrastructure, the {\ttfamily F\+FN} and {\ttfamily R\+NN} classes are very extensible, having the following template arguments; which can be modified to change the behavior of the network\+: \begin{DoxyItemize} \item {\ttfamily Output\+Layer\+Type\+:} this type defines the output layer used to evaluate the network; by default, {\ttfamily Negative\+Log\+Likelihood} is used. \item {\ttfamily Initialization\+Rule\+Type\+:} this type defines the method by which initial parameters are set; by default, {\ttfamily Random\+Initialization} is used. \end{DoxyItemize} \begin{DoxyCode} \textcolor{keyword}{template}< \textcolor{keyword}{typename} OutputLayerType = NegativeLogLikelihood<>, \textcolor{keyword}{typename} InitializationRuleType = RandomInitialization > \textcolor{keyword}{class }FNN; \end{DoxyCode} Internally, the {\ttfamily F\+FN} and {\ttfamily R\+NN} class keeps an instantiated {\ttfamily Output\+Layer\+Type} class (which can be given in the constructor). This is useful for using different loss functions like the Negative-\/\+Log-\/\+Likelihood function or the {\ttfamily V\+R\+Class\+Reward} function, which takes an optional score parameter. Therefore, you can write a non-\/static Output\+Layer\+Type class and use it seamlessly in combination with the {\ttfamily F\+NN} and {\ttfamily R\+NN} class. The same applies to the {\ttfamily Initialization\+Rule\+Type} template parameter. By choosing different components for each of these template classes in conjunction with the {\ttfamily Add()} method, a very arbitrary network object can be constructed. Below are several examples of how the {\ttfamily F\+NN} and {\ttfamily R\+NN} classes might be used. The first examples focus on the {\ttfamily F\+NN} class, and the last shows how the {\ttfamily R\+NN} class can be used. The simplest way to use the F\+N\+N$<$$>$ class is to pass in a dataset with the corresponding labels, and receive the classification in return. Note that the dataset must be column-\/major – that is, one column corresponds to one point. See the \doxyref{matrices guide}{p.}{matrices} for more information. The code below builds a simple feed-\/forward network with the default options, then queries for the assignments for every point in the {\ttfamily queries} matrix. \begin{DoxyImageNoCaption} \mbox{\includegraphics[width=\textwidth,height=\textheight/2,keepaspectratio=true]{dot_inline_dotgraph_1}} \end{DoxyImageNoCaption} \begin{DoxyNote}{Note} The number of inputs in the above graph doesn\textquotesingle{}t match with the real number of features in the thyroid dataset and are just used as an abstract representation. \end{DoxyNote} \begin{DoxyCode} \textcolor{preprocessor}{#include } \textcolor{preprocessor}{#include } \textcolor{preprocessor}{#include } \textcolor{keyword}{using namespace }mlpack; \textcolor{keyword}{using namespace }mlpack::ann; \textcolor{keywordtype}{int} main() \{ \textcolor{comment}{// Load the training set and testing set.} arma::mat trainData; data::Load(\textcolor{stringliteral}{"thyroid\_train.csv"}, trainData, \textcolor{keyword}{true}); arma::mat testData; data::Load(\textcolor{stringliteral}{"thyroid\_test.csv"}, testData, \textcolor{keyword}{true}); \textcolor{comment}{// Split the labels from the training set and testing set respectively.} arma::mat trainLabels = trainData.row(trainData.n\_rows - 1); arma::mat testLabels = testData.row(testData.n\_rows - 1); trainData.shed\_row(trainData.n\_rows - 1); testData.shed\_row(testData.n\_rows - 1); \textcolor{comment}{// Initialize the network.} FFN<> model; model.Add >(trainData.n\_rows, 8); model.Add >(); model.Add >(8, 3); model.Add >(); \textcolor{comment}{// Train the model.} model.Train(trainData, trainLabels); \textcolor{comment}{// Use the Predict method to get the predictions.} arma::mat predictionTemp; model.Predict(testData, predictionTemp); \textcolor{comment}{/*} \textcolor{comment}{ Since the predictionsTemp is of dimensions (3 x number\_of\_data\_points)} \textcolor{comment}{ with continuous values, we first need to reduce it to a dimension of} \textcolor{comment}{ (1 x number\_of\_data\_points) with scalar values, to be able to compare with} \textcolor{comment}{ testLabels.} \textcolor{comment}{} \textcolor{comment}{ The first step towards doing this is to create a matrix of zeros with the} \textcolor{comment}{ desired dimensions (1 x number\_of\_data\_points).} \textcolor{comment}{} \textcolor{comment}{ In predictionsTemp, the 3 dimensions for each data point correspond to the} \textcolor{comment}{ probabilities of belonging to the three possible classes.} \textcolor{comment}{ */} arma::mat prediction = arma::zeros(1, predictionTemp.n\_cols); \textcolor{comment}{// Find index of max prediction for each data point and store in "prediction"} \textcolor{keywordflow}{for} (\textcolor{keywordtype}{size\_t} i = 0; i < predictionTemp.n\_cols; ++i) \{ \textcolor{comment}{// we add 1 to the max index, so that it matches the actual test labels.} prediction(i) = arma::as\_scalar(arma::find( arma::max(predictionTemp.col(i)) == predictionTemp.col(i), 1)) + 1; \} \textcolor{comment}{/*} \textcolor{comment}{ Compute the error between predictions and testLabels,} \textcolor{comment}{ now that we have the desired predictions.} \textcolor{comment}{ */} \textcolor{keywordtype}{size\_t} correct = arma::accu(prediction == testLabels); \textcolor{keywordtype}{double} classificationError = 1 - double(correct) / testData.n\_cols; \textcolor{comment}{// Print out the classification error for the testing dataset.} std::cout << \textcolor{stringliteral}{"Classification Error for the Test set: "} << classificationError << std::endl; \textcolor{keywordflow}{return} 0; \} \end{DoxyCode} Now, the matrix prediction holds the classification of each point in the dataset. Subsequently, we find the classification error by comparing it with test\+Labels. In the next example, we create simple noisy sine sequences, which are trained later on, using the R\+NN class in the {\ttfamily R\+N\+N\+Model()} method. \begin{DoxyCode} \textcolor{keywordtype}{void} GenerateNoisySines(arma::mat& data, arma::mat& labels, \textcolor{keyword}{const} \textcolor{keywordtype}{size\_t} points, \textcolor{keyword}{const} \textcolor{keywordtype}{size\_t} sequences, \textcolor{keyword}{const} \textcolor{keywordtype}{double} noise = 0.3) \{ arma::colvec x = arma::linspace>(0, points - 1, points) / points * 20.0; arma::colvec y1 = arma::sin(x + arma::as\_scalar(arma::randu(1)) * 3.0); arma::colvec y2 = arma::sin(x / 2.0 + arma::as\_scalar(arma::randu(1)) * 3.0); data = arma::zeros(points, sequences * 2); labels = arma::zeros(2, sequences * 2); \textcolor{keywordflow}{for} (\textcolor{keywordtype}{size\_t} seq = 0; seq < sequences; seq++) \{ data.col(seq) = arma::randu(points) * noise + y1 + arma::as\_scalar(arma::randu(1) - 0.5) * noise; labels(0, seq) = 1; data.col(sequences + seq) = arma::randu(points) * noise + y2 + arma::as\_scalar(arma::randu(1) - 0.5) * noise; labels(1, sequences + seq) = 1; \} \} \textcolor{keywordtype}{void} RNNModel() \{ \textcolor{keyword}{const} \textcolor{keywordtype}{size\_t} rho = 10; \textcolor{comment}{// Generate 12 (2 * 6) noisy sines. A single sine contains rho} \textcolor{comment}{// points/features.} arma::mat input, labelsTemp; GenerateNoisySines(input, labelsTemp, rho, 6); arma::mat labels = arma::zeros(rho, labelsTemp.n\_cols); \textcolor{keywordflow}{for} (\textcolor{keywordtype}{size\_t} i = 0; i < labelsTemp.n\_cols; ++i) \{ \textcolor{keyword}{const} \textcolor{keywordtype}{int} value = arma::as\_scalar(arma::find( arma::max(labelsTemp.col(i)) == labelsTemp.col(i), 1)) + 1; labels.col(i).fill(value); \} Add<> add(4); Linear<> lookup(1, 4); SigmoidLayer<> sigmoidLayer; Linear<> linear(4, 4); Recurrent<> recurrent(add, lookup, linear, sigmoidLayer, rho); RNN<> model(rho); model.Add >(); model.Add(recurrent); model.Add >(4, 10); model.Add >(); StandardSGD opt(0.1, 1, input.n\_cols \textcolor{comment}{/* 1 epoch */}, -100); model.Train(input, labels, opt); \} \end{DoxyCode} For further examples on the usage of the ann classes, see {\tt mlpack models}.\section{Layer A\+PI}\label{anntutorial_layer_api_anntut} In order to facilitate consistent implementations, we have defined a Layer\+Type A\+PI that describes all the methods that a {\ttfamily layer} may implement. mlpack offers a few variations of this A\+PI, each designed to cover some of the model characteristics mentioned in the previous section. Any {\ttfamily layer} requires the implementation of a {\ttfamily Forward()} method. The interface looks like\+: \begin{DoxyCode} \textcolor{keyword}{template}<\textcolor{keyword}{typename} eT> \textcolor{keywordtype}{void} Forward(\textcolor{keyword}{const} arma::Mat& input, arma::Mat& output); \end{DoxyCode} The method should calculate the output of the layer given the input matrix and store the result in the given output matrix. Next, any {\ttfamily layer} must implement the Backward() method, which uses certain computations obtained during the forward pass and should calculate the function f(x) by propagating x backward through f\+: \begin{DoxyCode} \textcolor{keyword}{template}<\textcolor{keyword}{typename} eT> \textcolor{keywordtype}{void} Backward(\textcolor{keyword}{const} arma::Mat& input, \textcolor{keyword}{const} arma::Mat& gy, arma::Mat& g); \end{DoxyCode} Finally, if the layer is differentiable, the layer must also implement a Gradient() method\+: \begin{DoxyCode} \textcolor{keyword}{template}<\textcolor{keyword}{typename} eT> \textcolor{keywordtype}{void} Gradient(\textcolor{keyword}{const} arma::Mat& input, \textcolor{keyword}{const} arma::Mat& error, arma::Mat& gradient); \end{DoxyCode} The Gradient function should calculate the gradient with respect to the input activations {\ttfamily input} and calculated errors {\ttfamily error} and place the results into the gradient matrix object {\ttfamily gradient} that is passed as an argument. \begin{DoxyNote}{Note} Note that each method accepts a template parameter Input\+Type, Output\+Type or Gradient\+Type, which may be arma\+::mat (dense Armadillo matrix) or arma\+::sp\+\_\+mat (sparse Armadillo matrix). This allows support for both sparse-\/supporting and non-\/sparse-\/supporting {\ttfamily layer} without explicitly passing the type. \end{DoxyNote} In addition, each layer must implement the Parameters(), Input\+Parameter(), Output\+Parameter(), Delta() methods, differentiable layer should also provide access to the gradient by implementing the Gradient(), Parameters() member function. Note each function is a single line that looks like\+: \begin{DoxyCode} OutputDataType \textcolor{keyword}{const}& Parameters()\textcolor{keyword}{ const }\{ \textcolor{keywordflow}{return} weights; \} \end{DoxyCode} Below is an example that shows each function with some additional boilerplate code. \begin{DoxyNote}{Note} Note this is not an actual layer but instead an example that exists to show and document all the functions that mlpack layer must implement. For a better overview of the various layers, see \doxyref{mlpack\+::ann}{p.}{namespacemlpack_1_1ann}. Also be aware that the implementations of each of the methods in this example are entirely fake and do not work; this example exists for its A\+PI, not its implementation. \end{DoxyNote} Note that layer sometimes have different properties. These properties are known at compile-\/time through the \doxyref{mlpack\+::ann\+::\+Layer\+Traits}{p.}{classmlpack_1_1ann_1_1LayerTraits} class, and some properties may imply the existence (or non-\/existence) of certain functions. Refer to the Layer\+Traits \doxyref{layer\+\_\+traits.\+hpp}{p.}{layer__traits_8hpp} for more documentation on that. The two template parameters below must be template parameters to the layer, in the order given below. More template parameters are fine, but they must come after the first two. \begin{DoxyItemize} \item {\ttfamily Input\+Data\+Type\+:} this defines the internally used input type for example to store the parameter matrix. Note, a layer could be built on a dense matrix or a sparse matrix. All mlpack trees should be able to support any Armadillo-\/ compatible matrix type. When the layer is written it should be assumed that Mat\+Type has the same functionality as arma\+::mat. Note that \item {\ttfamily Output\+Data\+Type\+:} this defines the internally used input type for example to store the parameter matrix. Note, a layer could be built on a dense matrix or a sparse matrix. All mlpack trees should be able to support any Armadillo-\/ compatible matrix type. When the layer is written it should be assumed that Mat\+Type has the same functionality as arma\+::mat. \end{DoxyItemize} \begin{DoxyCode} \textcolor{keyword}{template}<\textcolor{keyword}{typename} InputDataType = arma::mat, \textcolor{keyword}{typename} OutputDataType = arma::mat> \textcolor{keyword}{class }ExampleLayer \{ \textcolor{keyword}{public}: ExampleLayer(\textcolor{keyword}{const} \textcolor{keywordtype}{size\_t} inSize, \textcolor{keyword}{const} \textcolor{keywordtype}{size\_t} outSize) : inputSize(inSize), outputSize(outSize) \{ \textcolor{comment}{/* Nothing to do here */} \} \} \end{DoxyCode} The constructor for {\ttfamily Example\+Layer} will build the layer given the input and output size. Note that, if the input or output size information isn\textquotesingle{}t used internally it\textquotesingle{}s not necessary to provide a specific constructor. Also, one could add additional or other information that are necessary for the layer construction. One example could be\+: \begin{DoxyCode} ExampleLayer(\textcolor{keyword}{const} \textcolor{keywordtype}{double} ratio = 0.5) : ratio(ratio) \{\textcolor{comment}{/* Nothing to do here*/}\} \end{DoxyCode} When this constructor is finished, the entire layer will be built and is ready to be used. Next, as pointed out above, each layer has to follow the Layer\+Type A\+PI, so we must implement some additional functions. \begin{DoxyCode} \textcolor{keyword}{template}<\textcolor{keyword}{typename} InputType, \textcolor{keyword}{typename} OutputType> \textcolor{keywordtype}{void} Forward(\textcolor{keyword}{const} InputType& input, OutputType& output) \{ output = arma::ones(input.n\_rows, input.n\_cols); \} \textcolor{keyword}{template}<\textcolor{keyword}{typename} InputType, \textcolor{keyword}{typename} ErrorType, \textcolor{keyword}{typename} GradientType> \textcolor{keywordtype}{void} Backward(\textcolor{keyword}{const} InputType& input, \textcolor{keyword}{const} ErrorType& gy, GradientType& g) \{ g = arma::zeros(gy.n\_rows, gy.n\_cols) + gy; \} \textcolor{keyword}{template}<\textcolor{keyword}{typename} InputType, \textcolor{keyword}{typename} ErrorType, \textcolor{keyword}{typename} GradientType> \textcolor{keywordtype}{void} Gradient(\textcolor{keyword}{const} InputType& input, ErrorType& error, GradientType& gradient) \{ gradient = arma::zeros(input.n\_rows, input.n\_cols) * error; \} \end{DoxyCode} The three functions {\ttfamily Forward()}, {\ttfamily Backward()} and {\ttfamily Gradient()} (which is needed for a differentiable layer) contain the main logic of the layer. The following functions are just to access and manipulate the different layer parameters. \begin{DoxyCode} OutputDataType& Parameters() \{ \textcolor{keywordflow}{return} weights; \} InputDataType& InputParameter() \{ \textcolor{keywordflow}{return} inputParameter; \} OutputDataType& OutputParameter() \{ \textcolor{keywordflow}{return} outputParameter; \} OutputDataType& Delta() \{ \textcolor{keywordflow}{return} delta; \} OutputDataType& Gradient() \{ \textcolor{keywordflow}{return} gradient; \} \end{DoxyCode} Since some of this methods return internal class members we have to define them. \begin{DoxyCode} \textcolor{keyword}{private}: \textcolor{keywordtype}{size\_t} inSize, outSize; OutputDataType weights, delta, gradient, outputParameter; InputDataType inputParameter; \end{DoxyCode} Note some members are just here so {\ttfamily Example\+Layer} compiles without warning. For instance, {\ttfamily input\+Size} is not required to be a member of every type of layer. There is one last method that is especially interesting for a layer that shares parameter. Since the layer weights are set once the complete model is defined, it\textquotesingle{}s not possible to split the weights during the construction time. To solve this issue, a layer can implement the {\ttfamily Reset()} method which is called once the layer parameter is set.\section{Model Setup \& Training}\label{anntutorial_model_setup_training_anntut} Once the base container is selected ({\ttfamily F\+NN} or {\ttfamily R\+NN}), the {\ttfamily Add} method can be used to add layers to the model. The code below adds two linear layers to the model---the first takes 512 units as input and gives 256 output units, and the second takes 256 units as input and gives 128 output units. \begin{DoxyCode} FFN<> model; model.Add >(512, 256); model.Add >(256, 128); \end{DoxyCode} The model is trained on Armadillo matrices. For training a model, you will typically use the {\ttfamily Train()} function\+: \begin{DoxyCode} arma::mat trainingSet, trainingLabels; model.Train(trainingSet, trainingLabels); \end{DoxyCode} You can use mlpack\textquotesingle{}s {\ttfamily \doxyref{Load()}{p.}{namespacemlpack_1_1data_a19805d6585ac8b0be7c4e4b7f081977c}} function to load a dataset like this\+: \begin{DoxyCode} arma::mat trainingSet; data::Load(\textcolor{stringliteral}{"dataset.csv"}, dataset, \textcolor{keyword}{true}); \end{DoxyCode} \begin{DoxyCode} $ cat dataset.csv 0, 1, 4 1, 0, 5 1, 1, 1 2, 0, 2 \end{DoxyCode} The type does not necessarily need to be a C\+SV; it can be any supported storage format, assuming that it is a coordinate-\/format file in the format specified above. For more information on mlpack file formats, see the documentation for \doxyref{mlpack\+::data\+::\+Load()}{p.}{namespacemlpack_1_1data_a19805d6585ac8b0be7c4e4b7f081977c}. \begin{DoxyNote}{Note} It’s often a good idea to normalize or standardize your data, for example using\+: \end{DoxyNote} \begin{DoxyCode} \textcolor{keywordflow}{for} (\textcolor{keywordtype}{size\_t} i = 0; i < dataset.n\_cols; ++i) dataset.col(i) /= norm(dataset.col(i), 2); \end{DoxyCode} Also, it is possible to retrain a model with new parameters or with a new reference set. This is functionally equivalent to creating a new model.\section{Saving \& Loading}\label{anntutorial_model_saving_loading_anntut} Using {\ttfamily \doxyref{boost\+::serialization}{p.}{namespaceboost_1_1serialization}} (for more information about the internals see {\tt Serialization -\/ Boost C++ Libraries}), mlpack is able to load and save machine learning models with ease. To save a trained neural network to disk. The example below builds a model on the {\ttfamily thyroid} dataset and then saves the model to the file {\ttfamily model.\+xml} for later use. \begin{DoxyCode} \textcolor{comment}{// Load the training set.} arma::mat dataset; data::Load(\textcolor{stringliteral}{"thyroid\_train.csv"}, dataset, \textcolor{keyword}{true}); \textcolor{comment}{// Split the labels from the training set.} arma::mat trainData = dataset.submat(0, 0, dataset.n\_rows - 4, dataset.n\_cols - 1); \textcolor{comment}{// Split the data from the training set.} arma::mat trainLabelsTemp = dataset.submat(dataset.n\_rows - 3, 0, dataset.n\_rows - 1, dataset.n\_cols - 1); \textcolor{comment}{// Initialize the network.} FFN<> model; model.Add >(trainData.n\_rows, 3); model.Add >(); model.Add >(); \textcolor{comment}{// Train the model.} model.Train(trainData, trainLabels); \textcolor{comment}{// Use the Predict method to get the assignments.} arma::mat assignments; model.Predict(trainData, assignments); data::Save(\textcolor{stringliteral}{"model.xml"}, \textcolor{stringliteral}{"model"}, model, \textcolor{keyword}{false}); \end{DoxyCode} After this, the file model.\+xml will be available in the current working directory. Now, we can look at the output model file, {\ttfamily model.\+xml\+:} \begin{DoxyCode} $ cat model.xml 66 1 66 0 -7.55971528334903642e+00 -9.95435955058058930e+00 9.31133928948225353e+00 -5.36784434861701953e+00 ... 0 0 0 0 0 0 3 0 18 <\textcolor{keyword}{value class}\_id=\textcolor{stringliteral}{"4"} tracking\_level=\textcolor{stringliteral}{"1"} version=\textcolor{stringliteral}{"0"} object\_id=\textcolor{stringliteral}{"\_2"}> 21 3 2 <\textcolor{keyword}{value class}\_id=\textcolor{stringliteral}{"5"} tracking\_level=\textcolor{stringliteral}{"1"} version=\textcolor{stringliteral}{"0"} object\_id=\textcolor{stringliteral}{"\_3"}> 20 <\textcolor{keyword}{value class}\_id=\textcolor{stringliteral}{"6"} tracking\_level=\textcolor{stringliteral}{"1"} version=\textcolor{stringliteral}{"0"} object\_id=\textcolor{stringliteral}{"\_4"}> \end{DoxyCode} As you can see, the {\ttfamily $<$parameter$>$} section of {\ttfamily model.\+xml} contains the trained network weights. We can see that this section also contains the network input size, which is 66 rows and 1 column. Note that in this example, we used three different layers, as can be seen by looking at the {\ttfamily $<$network$>$} section. Each node has a unique id that is used to reconstruct the model when loading. The models can also be saved as {\ttfamily }.bin or {\ttfamily }.txt; the {\ttfamily }.xml format provides a human-\/inspectable format (though the models tend to be quite complex and may be difficult to read). These models can then be re-\/used to be used for classification or other tasks. So, instead of saving or training a network, mlpack can also load a pre-\/trained model. For instance, the example below will load the model from {\ttfamily model.\+xml} and then generate the class predictions for the {\ttfamily thyroid} test dataset. \begin{DoxyCode} data::Load(\textcolor{stringliteral}{"thyroid\_test.csv"}, dataset, \textcolor{keyword}{true}); arma::mat testData = dataset.submat(0, 0, dataset.n\_rows - 4, dataset.n\_cols - 1); data::Load(\textcolor{stringliteral}{"model.xml"}, \textcolor{stringliteral}{"model"}, model); arma::mat predictions; model.Predict(testData, predictions); \end{DoxyCode} This enables the possibility to distribute a model without having to train it first or simply to save a model for later use. Note that loading will also work on different machines.\section{Extracting Parameters}\label{anntutorial_extracting_parameters_anntut} To access the weights from the neural network layers, you can call the following function on any initialized network\+: \begin{DoxyCode} model.Parameters(); \end{DoxyCode} which will return the complete model parameters as an armadillo matrix object; however often it is useful to not only have the parameters for the complete network, but the parameters of a specific layer. Another method, {\ttfamily Model()}, makes this easily possible\+: \begin{DoxyCode} model.Model()[1].Parameters(); \end{DoxyCode} In the example above, we get the weights of the second layer.\section{Further documentation}\label{anntutorial_further_anntut} For further documentation on the ann classes, consult the \doxyref{complete A\+PI documentation}{p.}{namespacemlpack_1_1ann}.