Why study simplicial homotopy groups?

To compute the homotopy groups of a simplicial set $X$, you need to be able to construct a weak equivalence $X \to Y$ where $Y$ is a Kan complex, and then compute the homotopy groups of $Y$ using the definitions you were discussing.

This might seem circular - you need to detect if $X \to Y$ is an equivalence. However, you can construct $Y$ directly using certain more elementary equivalences. Specifically, for a map of a horn $\Lambda \to X$ we can form the pushout of the diagram $\Delta \leftarrow \Lambda \rightarrow X$, called $X'$; on geometric realizations this is homotopy equivalent because you can construct an explicit retraction. The class of maps $X \to Y$ generated by such pushouts is called the family of anodyne extensions of $X$, and you can always construct an anodyne extension which is a Kan complex by the small object argument (you keep gluing in solutions to every possible horn-filling problem).

If you want a more canonical answer, there is also Kan's $\operatorname{Ex}^\infty$ construction.

If you want another reference, there are Kan's older papers, and my recollection is that Joyal and Tierney has quite a number of details as well.

If $X$ is a simplicial set which is not Kan, you can compute the homotopy groups of $X$ by choosing a weak homotopy equivalence $f: X \rightarrow Y$ where $Y$ is Kan and then applying the construction you are familiar with to $Y$. There are many ways of characterizing the relationship between $X$ and $Y$ without even mentioning topology or model categories (though I'm not sure it is so helpful to avoid these). For example, you can take any map $f: X \rightarrow Y$ with the following property: for every Kan complex $Z$, composition with $f$ induces a bijection from $[Y,Z]$ to $[X,Z]$, where $[K,Z]$ denotes the set of maps from $K$ into $Z$ up to (simplicial) homotopy. There are several purely combinatorial constructions of $Y$ from $X$: for example, Kan's $\operatorname{Ex}^{\infty}$ functor.

So this entire discussion is in Goerss and Jardine's "Simplicial homotopy theory" and also in May's "Simplicial objects in algebraic topology". Also Curtis' papers and monographs are very nice and classical. One aesthetic reason that one may want simplicial homotopy groups is to show that we can calculate homotopy groups within the category of simplicial sets. Thus one sets up this machinery. I think that Milnor proved the comparison between simplicial and let's say topological (even though this is not quite accurate, I think something like CGHaus homotopy groups) groups.

Here is how you speak about homotopy groups in the context of simplicial sets: First you need the notion of a horn. A horn is the boundary of an n-simplex with a lost face. Now you must remember that simplicial sets have arrows on their edges, so we have a couple of different horns in each dimension. In dimension 2 for example, we have three different horns (and will need all three of these horns for our to define the fundamental group, in addition to a three dimensional horn to give associativity). So we now define a Kan complex as a simplicial set in which each horn may be filled out to any n-simplex that it is contained in (this is written as a lifting property).

So we will use this to define the fundamental groups, as the higher homotopy groups are analogous. Pick a basepoint, and two loops based at that point (if we do not want to talk about basepoints this discussion works for fundamental groupoids). This can be realized as a map of a two horn into the simplicial set. Now pick a horn filling (it doesn't matter which, they all differ by homotopy equivalence, namely an even higher horn fills this choice of two horns. This is similar to higher category theory.). The group operation of the two loops is the new loop created on the boundary of the two simplex. Considering the other two horn fillings will give left and right inverses in the group, which must be shown to be homotopy equivalent (by more horn fillings)

All the properties that you would expect of such composite can be shown to be true by more (generalized) horn fillings. These are called anodyne extensions. It turns out if you can fill all of your horns, you can fill all of your anodynes. This will show that composition is independent of choice of representatives.