Developer Guide for Intel® Data Analytics Acceleration Library 2019 Update 4
Neural network training in the distributed processing mode has the following parameters:
Parameter |
Default Value |
Description |
|
---|---|---|---|
computeStep |
Not applicable |
The parameter required to initialize the algorithm. Can be:
|
|
algorithmFPType |
float |
The floating-point type that the algorithm uses for intermediate computations. Can be float or double. |
|
method |
defaultDense |
Performance-oriented computation method. |
|
batchSize |
128 |
The number of samples, simultaneously used for training on each node. The values of this parameter must be the same on all nodes for synchronous computations and may be different for asynchronous computations. This parameter is used by algorithms running on local nodes. |
|
optimizationSolver |
SharedPtr<optimization_solver:: sgd::Batch<float, defaultDense > >() |
The optimization procedure used at the training stage. This parameter is used by the algorithm running on the master node. |
Initialize
batch sizes depending on whether the computation is synchronous or
asynchronous:
Use the two-step computation schema for neural network training in the distributed processing mode, as illustrated below.
In this step, the neural network training algorithm accepts the following input. Pass the Input ID as a parameter to the methods that provide input for your algorithm. For more details, see Algorithms.
Input ID |
Input |
|
---|---|---|
data |
Pointer to the tensor of size batchSize x n2 x ... x np that represents the i-th data block on the local node. This input can be an object of any class derived from Tensor. |
|
groundTruth |
Pointer to the tensor of size batchSize that stores i-th stated results associated with the input data. This input can be an object of any class derived from Tensor. |
|
inputModel |
The neural network model updated on the master node. This input can only be an object of the Model class. ImportantUpdate the model parameters using the setWeightsAndBiases() method of inputModel after the master node delivers them to local nodes. |
In this step, the neural network training algorithm calculates partial results described below. Pass the Partial Result ID as a parameter to the methods that access the results of your algorithm. For more details, see Algorithms.
PartialResultID |
Partial Result |
|
---|---|---|
derivatives |
Pointer to the numeric table of size weightsAndBiasesSize x 1, where weightsAndBiasesSize is the number of model parameters. By default this result is an object of the HomogenNumericTable class, but you can define this result as an object of any class derived from NumericTable except PackedTriangularMatrix, PackedSymmetricMatrix, and CSRNumericTable. |
|
batchSize |
Pointer to the numeric table of size 1 x 1 that contains the number of samples simultaneously used for training on each node. By default this partial result is an object of the HomogenNumericTable class, but you can define this numeric table as an object of any class derived from NumericTable except PackedTriangularMatrix, PackedSymmetricMatrix, and CSRNumericTable. |
In this step, the neural network training algorithm accepts the following input. Pass the Input ID as a parameter to the methods that provide input for your algorithm. For more details, see Algorithms.
Input ID |
Input |
|
---|---|---|
partialResults |
Results computed on local nodes in Step 1 (derivatives and batchSize). This input contains objects of the PartialResult class. NoteYou can update the model on the master node incrementally, upon availability of partial results from local nodes, but be aware that in synchronous computations, the master node sends out the updated model only after it processes partial results from all local nodes. |
In this step, the neural network training algorithm calculates the results described below. Pass the Result ID as a parameter to the methods that access the results of your algorithm. For more details, see Algorithms.
Result ID |
Result |
|
---|---|---|
model |
Trained model with a set of weights and biases. This result can only be an object of the Model class. |