Function

class Function

Defines a block of operations with customizable differentiation.

The computation should be defined in forward method, with gradient computation defined in backward method.

Each instance of Function should be used only once during forwardding.

Examples

class Sigmoid(Function):
    def forward(self, x):
        y = 1 / (1 + F.exp(-x))
        self.y = y
        return y

    def backward(self, dy):
        y = self.y

backward(*output_grads)

Compute the gradient of the forward function. It must be overriden by all subclasses.

Parameters:

output_grads – gradients of outputs that are returned by forward.

Note

• In case when some tensors of outputs are not related to loss function, the corresponding values in output_grads would be None.

• This method should return a tuple which containing the gradients of all inputs, in the same order as the inputs argument of forward . A Tensor could be returned instead if there is only one input. If users want to stop the propagation of some gradients, the corresponding returned values should be set None .

forward(*args, **kwargs)

Applies operations to inputs and returns results. It must be overriden by all subclasses.

Parameters:

input – input tensors.

Returns:

a tuple of Tensor or a single Tensor.

Note

• This method should return a tuple of Tensor or a single Tensor representing the output of the function.

• positional arguments should all be Tensor