# Optimizer¶

Warning

Documentation for the module is under development.

Info

This module is intended primarily for developers who want to better understand the structure of the library, as well as those who are going to implement their own modules.

## Description¶

It is a base class for the optimizer family. Optimizers are a set of mathematical methods for solving the optimization problem for the error function.

## Initializing¶

def __init__(self, nodeinfo=None):


Parameters

Parameter Allowed types Description Default
nodeinfo NodeInfo Object containing information about the computational node None

Explanations

nodeinfo - special object containing information about the machine on which the calculations are performed; necessary for parallelization of calculations.

## Methods¶

### setAttr¶

def setAttr(self, name, attr):


Functionality

Sets the value of an attribute, for example, a learning rate. It is used as a helper method in derived classes.

Parameters

Parameter Allowed types Description Default
name str Parameter name -
attr float Parameter name -

Explanations

-

### getAttrDict¶

def getAttrDict(self):


Functionality

Returns the optimizer attributes as a dictionary.

Parameters

-

Explanations

-

### addHook¶

def addHook(self, hook):


Functionality

Adds the hook function to the optimizer.

Parameters

Parameter Allowed types Description Default
hook Hook Hook -

Explanations

-

### setupOn¶

def setupOn(self, mod, useGlobalState=False):


Functionality

Installs the optimizer to a trainable neural network.

Parameters

Parameter Allowed types Description Default
mod Module Trainable neural network -
useGlobalState bool Condition for optimizing the placement of tensors in the GPU False

Explanations

useGlobalState - this flag is responsible for two main points: if it is set, then, firstly, tensors of the neural network variables will be redistributed in the GPU so that weights can be optimized with a minimum number of queries to the GPU, and secondly, the preparation of the optimizer to the parallel mode of calculations will be implemented, if it is initiated.

### setupGlobalState¶

def setupGlobalState(self, vartable):


Functionality

Performs the redistribution of tensors of neural network variables in the GPU in such a way so that weights can be optimized with a minimum number of queries to the GPU, and prepares the optimizer to the parallel mode of calculations, if it is initiated.

Parameters

Parameter Allowed types Description Default
vartable dict Dictionary of neural network variables -

Explanations

vartable - information presented in the form of a dictionary on the variables of the neural network on which the optimizer is installed. Variables mean, for example, the weights and offsets of layers that will be updated during training.

### setupLocalStates¶

def setupGlobalState(self, vartable):


Functionality

Prepares the attributes of internal use for calculations.

Parameters

Parameter Allowed types Description Default
vartable dict Dictionary of neural network variables -

Explanations

vartable - information on the variables of the neural network on which the optimizer is installed, presented as a dictionary. Variables mean, for example, the weights and offsets of layers that will be updated during training.

### zeroGradParams¶

def zeroGradParams(self):


Functionality

A wrapper function around the operation query to reset the accumulated gradients.

Parameters

-

Explanations

-

### zeroGradGlobalParams¶

def zeroGradParams(self):


Functionality

Performs the operation of zeroing the accumulated gradients when working in global mode (the useGlobalState flag is set when the setupOn method is called).

Parameters

-

Explanations

-

### zeroGradLocalParams¶

def zeroGradParams(self):


Functionality

Performs the operation of zeroing the accumulated gradients when working in local mode (the useGlobalState flag is not set when the setupOn method is called).

Parameters

-

Explanations

-

### setupState¶

def setupState(self, var):


Functionality

An abstract helper method implemented in derived classes. It generates information about tensors placed in the GPU.

Parameters

Parameter Allowed types Description Default
var Variable Object of the Variable library class, which is a wrapper around the GPU tensors -

Explanations

-

### update¶

def update(self, useStreams=False, sync=True):


Functionality

A wrapper around the method for updating the variables of a trainable neural network.

Parameters

Parameter Allowed types Description Default
useStreams bool Flag responsible for the use of multistreaming when working in local mode False
sync bool Flag responsible for stream synchronization True

Explanations

-

### updateGlobalState¶

def update(self):


Functionality

Performs the operation of updating variables of the trainable neural network when working in local mode (the useGlobalState flag is not set when the setupOn method is called).

Parameters

-

Explanations

-

### updateLocalStates¶

def update(self, useStreams, sync):


Functionality

Performs the operation of updating variables of the trainable neural network when working in local mode (the useGlobalState flag is not set when the setupOn method is called).

Parameters

Parameter Allowed types Description Default
useStreams bool Flag responsible for use of multithreading -
sync bool Flag responsible for stream synchronization -

Explanations

-

### updateVar¶

def updateVar(self, var, state, stream=None):


Functionality

An abstract method that must be implemented in derived classes. It is this method that has to define the logic according to which the transferred variable tensor will be updated.

Parameters

Parameter Allowed types Description Default
var Variable Object of the Variable library class, which is a wrapper around the GPU tensors -
state str - -
stream - - None

Explanations

-

### save¶

def save(self, hdf, name=None):


Functionality

Saves the state of the gradients to an hdf-file.

Parameters

Parameter Allowed types Description Default
hdf Union[str, h5py.File] File path or an hdf-file object -
name str Name of the current state None

Explanations

-

### load¶

def load(self, hdf, name=None):


Functionality

Parameters

Parameter Allowed types Description Default
hdf Union[str, h5py.File] File path or an hdf-file object -
name str Name of the current state None

Explanations

-

### ensureHdf¶

def ensureHdf(file, mode):


Functionality

A wrapper method that opens and returns either an hdf-file in case the file path is specified in the file parameter, or trivially returns the file parameter if it is already a file object.

Parameters

Parameter Allowed types Description Default
file Union[str, h5py.File] File path or an hdf file object -
mode str File open mode ('w','r','a', and etc.) None

Explanations

-