Why ReLU function is not differentiable at 0?

Because it has a sharp corner at $0$, so it doesn't have a well defined tangent line; think about it, you can imagine many lines going through $(0, 0)$ that are tangent to the graph, so there are many possible tangent lines.

More formally, we have to investigate the limit

$$\lim_{h \to 0} \dfrac{f(0+h) - f(0)}{h}$$

This limit does not exist for the function, because if you let $h$ approach $0$ from the right, you get

$$\lim_{h \to 0^+} \dfrac {h-0}{h}=1.$$

While if you let $h$ approach $0$ from the left, that limit

$$\lim_{h \to 0^-} \dfrac {0-0}{h}=0.$$

Therefore the limit does not exist, so the function is not differentiable at $0$.


If you look at $x > 0$, or the righthand derivative, $$\frac{df}{dx} = \frac{d}{dx} x = 1$$ for all $x$.

If you look at $x \le 0$, or the lefthand derivative, $$\frac{df}{dx} = \frac{d}{dx} 0 = 0$$ for all $x$.

Since $x = 0$ is the "break" point, the lefthand and righthand derivatives are not the same, and thus, the derivative is not defined at $x = 0$.

Tags:

Calculus