Activation Functions

Input X

x:

Leaky slope α:

Leaky ReLU negative slope

Outputs

Function	Output

Notes:

Heaviside: returns 0 for x<0, 1 for x>0.
Sigmoid: 1 / (1 + e^−x)
tanh: (e^x − e^−x) / (e^x + e^−x)
ReLU: max(0, x)
Leaky ReLU: x if x≥0 else slope*x