Lie algebra of $\left(\begin{smallmatrix}a & b\\ & a^2\end{smallmatrix}\right )$ in $GL_2(\mathbb{R})$

Here is another answer which may be slightly more in line with first principles and/or what is learned at introductory level.

Consider the path in $G$ given by $$\gamma(t) = \left(\begin{array}{cc} a(t) & b(t) \\ 0 & a(t)^2 \end{array} \right),$$ which is differentiable, with $\gamma(0) = I$ (which implies $a(0)=1$ and $b(0)=0$). Since $\gamma$ is in $GL_2(\mathbb R)$, we also have $a(t)\neq 0$ for all $t$. We see that $$\gamma'(t) = \left(\begin{array}{cc} a'(t) & b'(t) \\ 0 & 2a(t)a'(t) \end{array} \right),$$ which implies that $$\gamma'(0) = \left(\begin{array}{cc} a'(0) & b'(0) \\ 0 & 2a'(0) \end{array} \right),$$ since $a(0)=1$. This shows that the Lie algebra $L$ consists of matrices of the form $\left(\begin{array}{cc} a & b \\ 0 & 2a \end{array} \right)$, where $a, b \in \mathbb R$. In fact, the above computation shows one inclusion, namely $\mathfrak g(G)\subset L$. Proving the other inclusion means specifically constructing a path in $G$ in the direction of any $A \in L$. So let $A = $$\left(\begin{array}{cc} a & b \\ 0 & 2a \end{array} \right),$ and consider the exponential mapping. We have

\begin{equation} \begin{split} e^A &= \left(\begin{array}{cc} 1 & 0 \\ 0 & 1 \end{array} \right) + \left(\begin{array}{cc} a & b \\ 0 & 2a \end{array} \right) + \frac{1}{2!}\left(\begin{array}{cc} a^2 & 3ab \\ 0 & 4a^2 \end{array} \right) + \frac{1}{3!}\left(\begin{array}{cc} a^3 & 7a^2b \\ 0 & 8a^3 \end{array} \right) + ... \\ &= \left(\begin{array}{cc} 1+a+\frac{a^2}{2!}+\frac{a^3}{3!}+... & b(1+\frac{3a}{2!}+\frac{7a^2}{3!}+...) \\ 0 & 1+2a+\frac{4a^2}{2!}+\frac{8a^3}{3!}+... \end{array} \right) \\ &= \left(\begin{array}{cc} e^a & b(1+\frac{3a}{2!}+\frac{7a^2}{3!}+...) \\ 0 & (e^a)^2 \end{array} \right). \end{split} \end{equation}

After convincing yourself that the series in the top-right matrix entry converges (it is not difficult), we see in fact that $e^A \in G$. Since $A \in L$, then $\alpha(t)= e^{tA}$ is a differentiable path in $G$ with $\alpha'(0)=A$. This shows that $L \subset \mathfrak g(G).$

At the identity $a=1$ and $b=0$. Near the identity, the group is parameterised as $$\pmatrix{1+t&u\\0&(1+t)^2}=I+t\pmatrix{1&0\\0&2}+u\pmatrix{0&1\\0&0}+O(\text{higher powers of $t$ and $u$}).$$ The tangent space at $I$ is spanned by $$A=\pmatrix{1&0\\0&2}\qquad\text{and}\qquad B=\pmatrix{0&1\\0&0}$$ and these matrices span the Lie algebra. To find the bracket, all you need is to compute $AB-BA$.