Assume that $\int_{a}^{ab} f(x) dx$ is independent of $a$. Prove $f(x)=\frac{c}{x}$

Most probably you mean that $f$ is continuous.

Let $F$ be an antiderivative of $f$.

Hence, for any $t>0$ we have

$$\int_a^{at}f(x)\;dx = F(at)-F(a)$$

Since the integral is independent of $a$ you have

$$\partial_a(F(at)-F(a))=tf(at) - f(a) = 0 \Rightarrow f(at)=\frac{f(a)}{t}$$

Now, seting $x=at$ you get

$$f(x) = \frac{af(a)}{x} =\frac{c}{x}$$


Hint. Let $F$ be the antiderivative of $f$, then the condition is $F(ab)-F(a)=g(b)$ for some $g$. Differentiating with respect to $a$, we get $bf(ab)-f(a)=0$, hence $f(ab)/f(a)=1/b$.


Define $$g(t)=\int_t^{tb}f(x)\operatorname dx$$

Then $g(t)=c$, so $g'(t)=0$.

By FTC, $$g'(t)=bf(tb)-f(t)$$ $$\implies f(tb)=\frac1{bf(t)}$$ Let $t=1$. Then $f(b)={f(1)\over b}$. $$\implies \boxed{f(x)=\frac cx}$$ with $c=f(1)$.