Public Member Functions
def	__init__
def	backward
def	bind_exec
def	decide_slices
def	forward
def	get_input_grads
def	get_outputs
def	get_params
def	get_states
def	install_monitor
def	reshape
def	set_params
def	set_states
def	update_metric
Public Attributes
	arg_names
	aux_arrays
	aux_names
	batch_size
	contexts
	data_arrays
	data_layouts
	data_shapes
	execs
	fixed_param_names
	for_training
	grad_arrays
	grad_req
	input_grad_arrays
	inputs_need_grad
	label_arrays
	label_layouts
	label_shapes
	logger
	output_layouts
	param_arrays
	param_names
	shared_data_arrays
	slices
	state_arrays
	state_names
	symbol
	workload
Private Member Functions
def	_bind_ith_exec
def	_collect_arrays
def	_sliced_shape
Private Attributes
	_default_execs

Detailed Description

DataParallelExecutorGroup is a group of executors that lives on a group of devices.
This is a helper class used to implement data parallelization. Each mini-batch will
be split and run on the devices.

Parameters
----------
symbol : Symbol
The common symbolic computation graph for all executors.
contexts : list
A list of contexts.
workload : list
If not `None`, could be a list of numbers that specify the workload to be assigned
to different context. Larger number indicate heavier workload.
data_shapes : list
Should be a list of (name, shape) tuples, for the shapes of data. Note the order is
important and should be the same as the order that the `DataIter` provide the data.
label_shapes : list
Should be a list of (name, shape) tuples, for the shapes of label. Note the order is
important and should be the same as the order that the `DataIter` provide the label.
param_names : list
A list of strings, indicating the names of parameters (e.g. weights, filters, etc.)
in the computation graph.
for_training : bool
Indicate whether the executors should be bind for training. When not doing training,
the memory for gradients will not be allocated.
inputs_need_grad : bool
Indicate whether the gradients for the input data should be computed. This is currently
not used. It will be useful for implementing composition of modules.
shared_group : DataParallelExecutorGroup
Default is `None`. This is used in bucketing. When not `None`, it should be a executor
group corresponding to a different bucket. In other words, it will correspond to a different
symbol but with the same set of parameters (e.g. unrolled RNNs with different lengths).
In this case, many memory will be shared.
logger : Logger
Default is `logging`.
fixed_param_names: list of str
Indicate parameters to be fixed during training. Parameters in this list will not allocate
space for gradient, nor do gradient calculation.
grad_req : str, list of str, dict of str to str
Requirement for gradient accumulation. Can be 'write', 'add', or 'null'
(default to 'write').
Can be specified globally (str) or for each argument (list, dict).

Definition at line 58 of file DataParallelExecutorGroup.py.

Constructor & Destructor Documentation

def core.DataParallelExecutorGroup.DataParallelExecutorGroup.__init__	(	self,
		symbol,
		contexts,
		workload,
		data_shapes,
		label_shapes,
		param_names,
		for_training,
		inputs_need_grad,
		shared_group = `None`,
		logger = `logging`,
		fixed_param_names = `None`,
		grad_req = `'write'`,
		state_names = `None`
	)

Definition at line 102 of file DataParallelExecutorGroup.py.

Member Function Documentation

def core.DataParallelExecutorGroup.DataParallelExecutorGroup._bind_ith_exec	(	self,
		i,
		data_shapes,
		label_shapes,
		shared_group
	)	`[private]`

Internal utility function to bind the i-th executor.

Definition at line 477 of file DataParallelExecutorGroup.py.

def core.DataParallelExecutorGroup.DataParallelExecutorGroup._collect_arrays ( self ) [private]

Collect internal arrays from executors.

Definition at line 213 of file DataParallelExecutorGroup.py.

def core.DataParallelExecutorGroup.DataParallelExecutorGroup._sliced_shape	(	self,
		shapes,
		i,
		major_axis
	)	`[private]`

Get the sliced shapes for the i-th executor.

Parameters
----------
shapes : list of (str, tuple)
    The original (name, shape) pairs.
i : int
    Which executor we are dealing with.

Definition at line 569 of file DataParallelExecutorGroup.py.

def core.DataParallelExecutorGroup.DataParallelExecutorGroup.backward	(	self,
		out_grads = `None`
	)

Run backward on all devices. A backward should be called after
a call to the forward function. Backward cannot be called unless
`self.for_training` is `True`.

Parameters
----------
out_grads : NDArray or list of NDArray, optional
    Gradient on the outputs to be propagated back.
    This parameter is only needed when bind is called
    on outputs that are not a loss function.

Definition at line 444 of file DataParallelExecutorGroup.py.

def core.DataParallelExecutorGroup.DataParallelExecutorGroup.bind_exec	(	self,
		data_shapes,
		label_shapes,
		shared_group = `None`,
		reshape = `False`
	)

Bind executors on their respective devices.

Parameters
----------
data_shapes : list
label_shapes : list
shared_group : DataParallelExecutorGroup
reshape : bool

Definition at line 247 of file DataParallelExecutorGroup.py.

def core.DataParallelExecutorGroup.DataParallelExecutorGroup.decide_slices	(	self,
		data_shapes
	)

Decide the slices for each context according to the workload.

Parameters
----------
data_shapes : list
    list of (name, shape) specifying the shapes for the input data or label.

Definition at line 187 of file DataParallelExecutorGroup.py.

def core.DataParallelExecutorGroup.DataParallelExecutorGroup.forward	(	self,
		data_batch,
		is_train = `None`
	)

Split `data_batch` according to workload and run forward on each devices.

Parameters
----------
data_batch : DataBatch
    Or could be any object implementing similar interface.
is_train : bool
    The hint for the backend, indicating whether we are during training phase.
    Default is `None`, then the value `self.for_training` will be used.
Returns
-------

Definition at line 330 of file DataParallelExecutorGroup.py.

def core.DataParallelExecutorGroup.DataParallelExecutorGroup.get_input_grads	(	self,
		merge_multi_context = `True`
	)

Get the gradients with respect to the inputs of the module.

Parameters
----------
merge_multi_context : bool
    Default is `True`. In the case when data-parallelism is used, the outputs
    will be collected from multiple devices. A `True` value indicate that we
    should merge the collected results so that they look like from a single
    executor.

Returns
-------
If `merge_multi_context` is `True`, it is like `[grad1, grad2]`. Otherwise, it
is like `[[grad1_dev1, grad1_dev2], [grad2_dev1, grad2_dev2]]`. All the output
elements are `NDArray`.

Definition at line 422 of file DataParallelExecutorGroup.py.

def core.DataParallelExecutorGroup.DataParallelExecutorGroup.get_outputs	(	self,
		merge_multi_context = `True`
	)

Get outputs of the previous forward computation.

Parameters
----------
merge_multi_context : bool
    Default is `True`. In the case when data-parallelism is used, the outputs
    will be collected from multiple devices. A `True` value indicate that we
    should merge the collected results so that they look like from a single
    executor.

Returns
-------
If `merge_multi_context` is `True`, it is like `[out1, out2]`. Otherwise, it
is like `[[out1_dev1, out1_dev2], [out2_dev1, out2_dev2]]`. All the output
elements are `NDArray`.

Definition at line 357 of file DataParallelExecutorGroup.py.

def core.DataParallelExecutorGroup.DataParallelExecutorGroup.get_params	(	self,
		arg_params,
		aux_params
	)

Copy data from each executor to `arg_params` and `aux_params`.

Parameters
----------
arg_params : list of NDArray
    target parameter arrays
aux_params : list of NDArray
    target aux arrays

Notes
-----
- This function will inplace update the NDArrays in arg_params and aux_params.

Definition at line 309 of file DataParallelExecutorGroup.py.

def core.DataParallelExecutorGroup.DataParallelExecutorGroup.get_states	(	self,
		merge_multi_context = `True`
	)

Get states from all devices

Parameters
----------
merge_multi_context : bool
    Default is `True`. In the case when data-parallelism is used, the states
    will be collected from multiple devices. A `True` value indicate that we
    should merge the collected results so that they look like from a single
    executor.

Returns
-------
If `merge_multi_context` is `True`, it is like `[out1, out2]`. Otherwise, it
is like `[[out1_dev1, out1_dev2], [out2_dev1, out2_dev2]]`. All the output
elements are `NDArray`.

Definition at line 380 of file DataParallelExecutorGroup.py.

def core.DataParallelExecutorGroup.DataParallelExecutorGroup.install_monitor	(	self,
		mon
	)

Install monitor on all executors

Definition at line 587 of file DataParallelExecutorGroup.py.

def core.DataParallelExecutorGroup.DataParallelExecutorGroup.reshape	(	self,
		data_shapes,
		label_shapes
	)

Reshape executors.

Parameters
----------
data_shapes : list
label_shapes : list

Definition at line 277 of file DataParallelExecutorGroup.py.

def core.DataParallelExecutorGroup.DataParallelExecutorGroup.set_params	(	self,
		arg_params,
		aux_params
	)

Assign, i.e. copy parameters to all the executors.

Parameters
----------
arg_params : dict
    A dictionary of name to `NDArray` parameter mapping.
aux_params : dict
    A dictionary of name to `NDArray` auxiliary variable mapping.

Definition at line 296 of file DataParallelExecutorGroup.py.

def core.DataParallelExecutorGroup.DataParallelExecutorGroup.set_states	(	self,
		states = `None`,
		value = `None`
	)

Set value for states. Only one of states & value can be specified.

Parameters
----------
states : list of list of NDArrays
    source states arrays formatted like [[state1_dev1, state1_dev2],
    [state2_dev1, state2_dev2]].
value : number
    a single scalar value for all state arrays.

Definition at line 401 of file DataParallelExecutorGroup.py.

def core.DataParallelExecutorGroup.DataParallelExecutorGroup.update_metric	(	self,
		eval_metric,
		labels
	)

Accumulate the performance according to `eval_metric` on all devices.

Parameters
----------
eval_metric : EvalMetric
    The metric used for evaluation.
labels : list of NDArray
    Typically comes from `label` of a `DataBatch`.

Definition at line 464 of file DataParallelExecutorGroup.py.