megengine.module.init.calculate_gain

calculate_gain(nonlinearity, param=None)

Returns a recommended gain value (see the table below) for the given nonlinearity function.

nonlinearity	gain
Linear / Identity	\(1\)
Conv{1,2,3}D	\(1\)
Sigmoid	\(1\)
Tanh	\(\frac{5}{3}\)
ReLU	\(\sqrt{2}\)
Leaky Relu	\(\sqrt{\frac{2}{1 + {\text{negative}_\text{slope}}^2}}\)

Parameters:

• nonlinearity (str) – name of the non-linear function.

• param (Union[int, float, None]) – optional parameter for leaky_relu. Only effective when nonlinearity is “leaky_relu”.

Return type:

float