Find the derivative using the definition of derivative (limit).

You can expand $f(x)$ as $\frac{5x+1}{2\sqrt x}=\frac52\sqrt x+\frac1{2\sqrt x}$, then differentiate the two.

$$\begin{align} f'(x)&=\left(\frac52\sqrt x\right)'+\left(\frac1{2\sqrt x}\right)'\\[1ex] &=\lim_{h\to0}\frac{\frac52\sqrt{x+h}-\frac52\sqrt x}h+\lim_{h\to0}\frac{\frac1{2\sqrt{x+h}}-\frac1{2\sqrt x}}h \end{align}$$

But if you don't already know or cannot directly use the fact that differentiation distributes over sums, you can instead expand the limand in your second line and regroup the terms in the numerator, then separate the limit into two. You would end up arriving at the same point either way:

$$\begin{align} f'(x)&=\lim_{h\to0}\frac{\frac{5(x+h)+1}{2\sqrt{x+h}}-\frac{5x+1}{2\sqrt x}}h\\[1ex] &=\lim_{h\to0}\frac{\frac52\sqrt{x+h}+\frac1{2\sqrt{x+h}}-\frac52\sqrt x+\frac1{2\sqrt x}}h\\[1ex] &=\lim_{h\to0}\frac{\frac52\sqrt{x+h}-\frac52\sqrt x}h+\lim_{h\to0}\frac{\frac1{2\sqrt{x+h}}-\frac1{2\sqrt x}}h \end{align}$$


First, note that we don't need to worry about the derivative of $f$ when $x=0$, because your function isn't defined when $x=0$.

Using the Binomial expansion, as $h \to 0,$ which garuntees that $|\frac{h}{x}| < 1,$

$ \sqrt{x+h} = \sqrt{x} \sqrt{1 + \frac{h}{x}} = \sqrt{x} \left(1 + \frac{h}{x}\right)^\frac12 = \sqrt{x}\left(1 + \frac{\left(\frac12\right)}{1!\ } \left(\frac{h}{x}\right) + \frac{\left(\frac12\right)\left(\frac{-1}{2}\right) }{2!\ } \left(\frac{h}{x}\right)^2 + ...\right),$

i.e., $\ \sqrt{x+h} = \sqrt{x} + \frac{h}{2\ \sqrt{x} } + O(h^2).$

Substitute this into the last line of working in the question (but not the denominator), we get:

\begin{align*} f'(x)&= \lim\limits_{h\to 0} \left(\dfrac{\dfrac{5x\sqrt{x}+5h\sqrt{x}+\sqrt{x}-(5x+1)\sqrt{x+h}}{2\sqrt{x+h}\sqrt{x}}}{h}\right)\\ &= \lim\limits_{h\to 0} \left(\dfrac{5x\sqrt{x}+5h\sqrt{x}+\sqrt{x}-(5x+1)\left(\sqrt{x}+ \frac{h}{2\ \sqrt{x} } + O(h^2)\right)}{2h\sqrt{x+h}\sqrt{x}}\right)\\ &= \lim\limits_{h\to 0} \left(\dfrac{5x\sqrt{x}+5h\sqrt{x}+\sqrt{x}-5x\sqrt{x}-\frac52 h \sqrt{x} - \sqrt{x} - \frac{h}{2\ \sqrt{x} } }{2h\sqrt{x+h}\sqrt{x}} + O(\sqrt{h})\right).\\ \end{align*}

... and after some cancellation you will arrive at the answer.