If $f \in C[a,b]$ has $\int_{a}^{b}f(x)x^{n}dx=0$ for all $n\in \mathbb{N}$, then $f=0$.

Use the fact that polynomials are dense in $C[a,b]$ with the supremum norm (see the Stone–Weierstrass theorem).

Due to the linearity of integration, your assumption implies that

$$\int_a^b f(x)P(x) \mathrm{d}x = 0 \hspace{10px}(\star)$$

for any polynomial $P(x)$.

By the Stone-Weierstrass theorem, you can find a sequence of polynomials $\{P_n\}_{n=1}^{\infty}$ such that $\lim_{n\to\infty}P_n(x) = f(x)$ uniformly. Since the convergence is uniform, we can exchange limit and integration operators. Therefore, we have

$$\lim_{n\to\infty}\underbrace{\int_a^b f(x)P_n(x)\mathrm{d}x}_{\text{this is zero by }(\star)} = \int_a^bf(x)\lim_{n\to\infty}P_n(x)\mathrm{d}x= \int_a^b f(x)^2 \mathrm{d}x = 0$$

Since $f^2(x) \geq 0$, we conclude that $f=0$ on $[a,b]$.


Based on the context set in the text of the question, I assume we can take

$0 \in \Bbb N; \tag 1$

I also take it that

$a \le b. \tag 2$

We have the following

Lemma: Suppose

$g_n \in C[a, b] \tag 3$

is a sequence such that

$g_n \to f \; \text{as} \; n \to \infty \tag 4$

then

$\displaystyle \lim_{n \to \infty} \int_a^b fg_n \; dx = \int_a^b f^2 \; dx. \tag 5$

Proof of Lemma:

Observe that

$\displaystyle \int_a^b f^2 \; dx - \int_a^b f g_n \; dx = \int_a^b (f^2 - fg_n) \; dx = \int_a^b f(f - g_n) \; dx; \tag 7$

thus

$\left \vert \displaystyle \int_a^b f^2 \; dx - \int_a^b f g_n \; dx \right \vert = \left \vert \displaystyle \int_a^b f(f - g_n) \; dx \right \vert$ $\le \displaystyle \int_a^b \vert f (f - g_n) \vert \; dx = \displaystyle \int_a^b \vert f \vert \vert f - g_n \vert \; dx \le \int_a^b \vert f \vert \Vert f - g_n \Vert \; dx = \Vert f - g_n \Vert \int_a^b \vert f \vert \; dx; \tag 8$

passing to the limit,

$\lim_{n \to \infty} \left \vert \displaystyle \int_a^b f^2 \; dx - \int_a^b f g_n \; dx \right \vert \le \lim_{n \to \infty} \Vert f - g_n \Vert \displaystyle \int_a^b \vert f \vert \; dx = 0; \tag 9$

thus (5) binds. End: Proof of Lemma.

With this lemma in hand we proceed with the observation that the function

$x:[a, b] \to \Bbb R \tag{10}$

separates points in $[a, b]$; that is, for any

$c_1, c_2 \in [a, b], \; c_1 \ne c_2, \tag{11}$

we have

$x(c_1) = c_1 \ne c_2 = x(c_2); \tag{12}$

it then follows from the Stone-Weierstrass theoremthat the sub-algebra of $C[a, b]$ generated by $x$ and the constant functions is dense; since this sub-algebra is the set of real polynomials in $x$, it follows that there exists a sequence of polynomials $p_n(x)$ such that

$p_n(x) \to f(x) \; \text{in} \; C[a, b]; \tag{13}$

that is,

$\displaystyle \lim_{n \to \infty} \Vert p_n(x) - f(x) \Vert = 0, \tag{14}$

and from the hypothesis that

$\displaystyle \int_a^b x^m f(x) \; dx = 0, \; \forall m \in \Bbb N, \tag{15}$

the linearity of the integral allows us to conclude that

$\displaystyle \int_a^b p_n(x) f(x) \; dx = 0, \forall n \in \Bbb N; \tag{16}$

now applying the lemma we find

$\left \vert \displaystyle \int_a^b f^2(x) \; dx \right \vert = \lim_{n \to \infty} \left \vert \displaystyle \int_a^b f^2 \; dx - \underbrace{\int_a^b p_n f \; dx}_{0} \right \vert \le \lim_{n \to \infty} \Vert f - p_n \Vert \displaystyle \int_a^b \vert f \vert \; dx = 0, \tag{17}$

which forces

$\displaystyle \int_a^b f^2(x) \; dx = 0; \tag{18}$

now since $f^2(x)$ is a non-negative continuous function it follows that

$f(x) = 0, \forall x \in [a, b]. \tag{19}$

$OE\Delta$.