$\langle Tu\;|\;u\rangle=0,\;\forall u\in E \Longrightarrow T=0$?

Suppose $\newcommand{\ip}[2]{\langle #1\mid #2\rangle}\ip{Tu}{u}=0$ for every $u$. Then, given $u$ and $v$ and any scalar $a$ (assuming semilinearity in the first variable, with the opposite convention the proof is essentially the same): $$ 0=\ip{T(u+av)}{u+av}= a\ip{Tu}{v}+\bar{a}\ip{Tv}{u} $$ In particular, for $a=i$, we get $\ip{Tu}{v}=\ip{Tv}{u}$ and, for $a=1$, $\ip{Tu}{v}=-\ip{Tv}{u}$.

Therefore $\ip{Tu}{v}=0$, for every $u$ and $v$, in particular for $v=Tu$. Hence $Tu=0$.

Note that this can fail on real Hilbert spaces; the easiest example is $$ T\colon\mathbb{R}^2\to\mathbb{R}^2 \qquad T\begin{bmatrix} x\\y\end{bmatrix}=\begin{bmatrix}-y\\x\end{bmatrix} $$


A sesquilinear form $s(x,y)$ on a complex vector space can be recovered from the associated quadratic form $q(x)=s(x,x)$ through the polarization identity $$ s(x,y) = \frac{1}{4}\sum_{n=0}^{3}i^n q(x+i^n y). $$ So, if the quadratic form is $0$, then so is the sesquilinear form.

In your case, $s(x,y)=\langle Tx,y\rangle$ and $q(x)=\langle Tx,x\rangle$. So it is in fact the case that $T=0$ if $\langle Tx,x\rangle=0$ for all $x$, but this relies on having a complex space. Real spaces do not behave in the same way.


On a complex Hilbert space $H$ the following proposition holds:

Let $T : H \to H$ be a bounded linear map such that $\langle Tx, x\rangle \in \mathbb{R}$ for all $x \in H$.

Then $T^* = T$, i.e. $T$ is self-adjoint.

Proof:

We have $\langle Tx, x\rangle = \overline{\langle x, Tx\rangle} = \langle x, Tx\rangle$ for all $x\in H.$

\begin{align} 4\langle Tx, y\rangle &= \langle T(x+y), x+y\rangle - \langle T(x-y), x-y\rangle + i\langle T(x+iy), x+iy\rangle - i\langle T(x-iy), x-iy\rangle\\ &= \langle x+y, T(x+y)\rangle - \langle x-y, T(x-y)\rangle + i\langle x+iy, T(x+iy)\rangle - i\langle x-iy, T(x-iy)\rangle\\ &= 4\langle x, Ty\rangle \end{align}

Hence $\langle Tx, y\rangle = \langle x, Ty\rangle$ for all $x, y \in H$ so $T^* = T$.

Now in your case we have $\langle Tx, x\rangle = 0 \in \mathbb{R}$ so in particular $T^* = T$.

Using the formula you gave for the norm of a self-adjoint operator we obtain:

$$\|T\| = \sup_{\|x\| = 1} |\langle Tx, x\rangle| = 0 \implies T = 0$$