Prove that $a * b = a + b - ab$ defines a group operation on $\Bbb R \setminus \{1\}$
The statement of the associativity condition is wrong; it should be $$(a \ast b) \ast c = a \ast (b \ast c) .$$
Expanding the l.h.s. gives \begin{align}(a \ast b) \ast c &= (a + b - ab) \ast c \\ &= (a + b - ab) + c - (a + b - ab) c \\ &= a + b + c - bc - ca - ab + abc .\end{align}
Now, do the same thing for the r.h.s. and verify that the expressions agree.
We cannot simply assume the existence of an identity and inverse. There are plenty of associative binary operations which lack one or both!
We can pick out the identity element using the definition: We must have, for example, that $$a \ast e = a$$ for all $a$, and expanding gives $$a + e - ae = a.$$ Can you find which $e$ makes this true for all $a$?
Likewise, to show that the operation admits inverses, it's enough to produce a formula for the inverse $a^{-1}$ of an arbitrary element $a$, that is, an element that satisfies $a \ast a^{-1} = e = a^{-1} \ast a$. As with the identity, use the definition of $\ast$ and solve for $a^{-1}$ in terms of $a$.
Remark If we glance at the triple product, which by dint of associativity we may as well write $a \ast b \ast c$, we can see the occurrence of the elementary symmetric polynomials in $a, b, c$, which motivates writing that product as $(a - 1) (b - 1) (c - 1) + 1$. Then, glancing back we can see that we can similarly write $a \ast b = (a - 1) (b - 1) + 1,$ which is just the conjugation of the usual multiplication (on $\Bbb R - \{0\}$) by the shift $s : x \mapsto x + 1$, that is, $a \ast b := s(s^{-1}(a) s^{-1}(b))$. Since multiplication defines a group structure on $\Bbb R - \{0\}$, that $\ast$ defines a group structure follows by unwinding definitions. For example, to show associativity, we have $$(a \ast b) \ast c = s(s^{-1}(s(s^{-1}(a) s^{-1}(b))) s^{-1}(c)) = s(s^{-1}(a) s^{-1}(b) s^{-1}(c)) = s(s^{-1}(a)s^{-1}(s(s^{-1}(b) s^{-1}(c))) = a \ast (b \ast c) .$$ (Note that in writing the third expression without parentheses we have implicitly used the associativity of the usual multiplication.) You can just as well use this characterization to determine the group identity $e$ and the formula for the inverse of an element under $\ast$.
Indeed, this together with analogous arguments for the other group axioms shows that conjugating any group operation by a set bijection defines an operation on the other set.
An important thing to prove here too is that for every pair $a,b\in\mathbb{R}\backslash\{1\}$, we need $a*b\in\mathbb{R}\backslash\{1\}$.
This is true, since for $a\neq1\neq b$ to have $a+b-ab=1$, we'd need $a(1-b)=1-b$ meaning either $a=1$ or $b=1$.
We have an identity, since (let's denote the identity with $I$, so $a*I=a$) for $a+I-aI=a$ we only need $I(1-a)=0$ so $I=0$. So $0$ is the identity, since, $a*0=a=0*a$. Note that, even in formal proofs, it's not necessary to include how you've found it; simply presenting it and showing it is the identity is sufficient.
This operation is associative, since: \begin{align} a*(b*c)&=a+(b+c-bc)-a(b+c-bc)\\ &=a+b+c-bc-ab-ac+abc\\ &=(a+b-ab)+c-(a+b-ab)c\\ &=(a*b)*c \end{align}
A group also needs inverse elements; elements $a^{-1}$ for every $a$ that satisfy $a*a^{-1}=0$ (because $0$ was the identity). For $a*b=0$, we need $a+b-ab=0$, so $b=\frac{a}{a-1}$ (and we won't be dividing by $0$ since $a\neq 1$). And so now: $a*\frac{a}{1-a}=0$, which is what we wanted.
Now we've checked every property a group must have, and it has it all, thus, it's a group. An important, useful fact to notice in this group is the fact that it is commutative, meaning $a*b=b*a$. This property often makes a lot of calculations and/or proofs a lot easier, so you should always look out for it.
Associativity is satisfied:
$$a*(b*c)=\\a*(b+c-bc)=\\a+b+c-bc-a(b+c-bc)=\\a+b+c-ab-bc-ac+abc$$
and $$(a*b)*c=\\(a+b-ab)*c=\\a+b-ab+c-(a+b-ab)c=\\a+b+c-ab-bc-ac+abc$$