Developer Guide for Intel® Data Analytics Acceleration Library 2018
The stochastic gradient descent (SGD) algorithm is a special case of an iterative solver. For more details, see Iterative Solver.
The following computation methods are available in Intel DAAL for the stochastic gradient descent algorithm:
The mini-batch method (miniBatch) of the stochastic gradient descent algorithm [Mu2014] follows the algorithmic framework of an iterative solver with an empty set of intrinsic parameters of the algorithm S t and the algorithm-specific transformation T defined for the learning rate sequence {η t } t=1,...,nIterations , conservative sequence {γ t } t=1,...,nIterations , and the number of iterations in the internal loop L as follows:
For l from 1 until L:
Update the function argument
Compute the gradient
The default method (defaultDense) is a particular
case of the mini-batch method with the batch size
b=1,
L=1, and conservative sequence
The momentum method (momentum) of the stochastic gradient descent algorithm [Rumelhart86] follows the algorithmic framework of an iterative solver with the set of intrinsic parameters S t and algorithm-specific transformation T defined for the learning rate sequence {η t } t=1,...,nIterations and momentum parameter μ ∈ [0,1] as follows:
For the momentum method of the SGD algorithm, the set of intrinsic parameters S t only contains the last update vector v t .