Does a set $G$ need to be closed under a binary operation $\star$ for $(G, \star)$ to be a group?

Closure is most definitely part of the definition of a group.

It might be that in some texts, closure is part of the definition of the binary operation, e.g., implicit when writing $\star : G \times G \to G$. (Rather than reiterating it as a separate axiom.)


A binary operation $\ast$ on $G$ is a mapping $\ast:G\times G\to G$ and so by definition $G$ is closed under $\ast$. Perhaps you are thinking of verifying the axioms for subgroups, in which case you do have to verify closure. That is, to verify that $H$ is a subgroup, you have to check that the restriction of $\ast$ to $H\times H$ is a mapping $H\times H\to H$.


part of the definition of what a binary operation is ensures closure.

Recall that the first requirement is that the group has a binary opertation, in other words a function $G\times G \rightarrow G$,