Activation Functions

What are Activation Functions?

Activation functions are used in 3rd step in the image below.

Types of Activation Functions

There are 5 types of activation functions:

Threshold function

The threshold function returns 0 if the value is less than 0, and returns 1 if the value is greater than 0. It’s a type of yes/no function. To express it in a formula:

$\phi(x)\ =\ \begin{cases} 1\ if\ x\geq0\\ 0\ if\ x < 0 \end{cases}$

And the graph will be:

Sigmoid function

The sigmoid function is useful in the final layer (output layer), especially when predicting probabilities. To express it in a formula:

$\phi(x)\ =\ \frac{1}{1\ +\ e^{-x}}$

And the graph will be:

Rectifier function (ReLU)

The rectifier function is one of the most popular functions for ANN. It returns 0 if the value is less than 0, and returns the value if the value is more than 0. To express it in a formula:

$\phi(x)\ =\ max(x,\ 0)$

And the graph will be:

Hyperbolic Tangent(tanh) function

It is similar to the sigmoid function, but it goes below 0. To express it in a formula:

$\phi(x)\ =\ \frac{1\ -\ e^{-2x}}{1\ +\ e^{-2x}}$

And the graph will be:

Mish Activation

Mish avoids saturation due to capping because the graph tends towards positive infinity. Also, it can decrease overfitting, and strong regularization may appear because it is bounded below.

$\phi(x)\ =\ xtanh(softplus(x))\ =\ xtanh(ln(1+e^{x}))$

SiLU Activation

The SiLU activation function avoids saturation due to capping because the graph tends towards positive infinity. It can also decrease overfitting, and strong regularization may appear because it is bounded below.

$\phi(x)\ =\ x \times \theta(x)$ where $\theta(x)$ is the logistic sigmoid.