Understanding Hartshorne's definition of subvariety
I'll answer the first question . . .
Since $Y$ is locally closed, we can write $Y=A\cap B$, where $A$ is open, and $B$ is closed.
Let $\overline{Y}$ denote the closure of $Y$.
Since $Y$ is irreducible, so is $\overline{Y}$, hence $\overline{Y}$ is an affine variety.
From $Y\subseteq B$ and $Y\subseteq \overline{Y}$, we get $Y\subseteq B\cap\overline{Y}\subseteq \overline{Y}$, hence $B\cap\overline{Y}=\overline{Y}$.
Then from $Y=Y\cap\overline{Y}=(A\cap B)\cap\overline{Y}=A\cap(B\cap\overline{Y})=A\cap \overline{Y}$, we get that $Y$ is an open subset of $\overline{Y}$.
Therefore $Y$ is a quasi-affine variety.
Since $B\subseteq V$ is closed and $V$ has been given the subspace topology inherited from $\mathbf{A}^{n},$ we know that $B=V\cap W$ for some closed set $W\subseteq\mathbf{A}^{n}.$ Hence $B\subseteq\mathbf{A}^{n}$ is the intersection of two closed sets, i.e., $B$ is closed in $\mathbf{A}^{n},$ i.e., $B$ itself is an affine variety. Now the topology on $B$ can be seen as being induced from the topology on $V,$ so the fact that $Y=A\cap B$ where $A\subseteq V$ is open means precisely that $Y$ is an open subset of $B.$ Hence $Y$ is quasi-affine.
The main problem with your proposed definition is that you haven't defined what you mean by a "variety". However, Hartshorne is doing essentially doing as you suggest: a quasi-affine variety is an irreducible locally closed subset of an affine variety (in the trivial case that the closed set is the whole ambient variety), and a subvariety of a quasi-affine variety $X$ is a subset $Y$ which is irreducible and locally closed. The answer to your first question is the proof of the fact that this is equivalent to saying "a subvariety of a quasi-affine variety $X$ is a subset $Y$ which is also a quasi-affine variety".
Essentially, a quasi-affine variety $X$ is the kind of thing you're left with when you take an affine variety $V$ and, in some order, (i) restrict attention to those points also contained in a possibly strictly smaller affine variety $B$ of your choosing (because $Y=A\cap B$ is a subset of $B$) and (ii) delete points which also belong to some other affine varieties of your choosing (because $Y$ is an open subset of $B,$ and Zariski open sets are precisely the complements of affine varieties). Now if you have a quasi-affine variety $X$ and you continue (i) restricting attention to points also contained in certain affine varieties and (ii) deleting points which also belong to some other affine varieties, then of course you end up with a quasi-affine variety $Y$ at the end, because you could have just gone straight from $V$ to $Y$ to begin with.