In what sense is the renormalization group equation a group?
There are really several questions here: (a) What is the renormalization group? Specifically the law of composition, etc. (b) How does the equation the OP gave relate to this?
Short Answer
It's a semigroup (see references below). The equation you wrote, $$\tag{1} \left[\mu \frac{\partial}{\partial \mu} + \beta \frac{\partial}{\partial g} + m \gamma_{m^2} \frac{\partial}{\partial m} - n \gamma_d \right] \Gamma^{(n)}(\{p_i\};g,m,\mu) = 0$$ is not "the renormalization group equation". It's the Callan-Symanzik equation, which is derived from the renormalization conditions (I cannot draw Feynman diagrams, so I cannot show you the rules per se).
For a derivation of the Callan-Symanzik equation from the renormalization group, see:
- Kleinert's Path Integrals in Quantum Mechanics, Statistics, Polymer Physics, and Financial Markets, specifically chapter 10 "Renormalization Group"; eprint
- John Cardy's lecture notes on QFT chapter 6
- Jan Louis' lecture notes on QFT, section 9
- Peskin and Schroeder's Introduction to Quantum Field Theory, ch. 12.
Handwavy Derivation of Renormalization group
We will consider the $\phi^4$ model. We want a "momentum cutoff" $\Lambda$. So we basically consider the Fourier transformed field $\phi(k)$ with nonzero components for $|k|<\Lambda$. We write
$$\tag{2a}Z_{\Lambda} =\int\exp\left(-\int\left[\frac{1}{2}(\partial_{\mu}\phi)^{2}+\frac{1}{2}m^{2}\phi^{2}+\frac{\lambda}{4!}\phi^4\right]\,\mathrm{d}^{n}x\right)[D\phi]_{\lambda}$$
where
$$\tag{2b} [D\phi]_{\Lambda} = \prod_{|k|<\Lambda}\mathrm{d}\phi(k).$$
We now factor $Z$ into two components, the "high frequency" and "low frequency" parts. We want to integrate out the "high frequency" parts, so $Z$ only has an integral over "low frequency" parts. We take $0<b\leq1$.
Now the "high-frequency components" of $\phi(k)$ correspond to those with $k$ satisfying $b\Lambda\leq|k|\leq\Lambda$. We will transform $Z$ to depend only on frequencies $|k|\leq b\Lambda$. This is our transformation (or "law of composition", if you will). Since $0\lt b\leq1$, this transformation has no inverse...but it has an identity transformation when $b=1$. Hence to answer your question
Is the "renormalization group" a group?
The answer is "no". Now, lets see how to carry out this transformation (albeit slightly handwavy, just to show the milestones alont the way).
Label these components that we will integrate out as
$$\tag{3}\hat{\phi}(k) = \begin{cases}\phi(k) & \mbox{for $b\Lambda\leq|k|\leq\Lambda$}\\ 0 & \mbox{otherwise}\end{cases}$$
Lets write $\tilde{\phi}(k)=\phi(k)-\hat{\phi}(k)$. Then observe the partition function becomes
$$\tag{4} Z=\int D\tilde{\phi}\int D\hat{\phi}\exp\left(-\int\left[\frac{1}{2}(\partial_{\mu}\tilde{\phi}+\partial_{\mu}\hat{\phi})^{2}+\frac{1}{2}m^{2}(\tilde{\phi}+\hat{\phi})^{2}+\frac{\lambda}{4!}(\tilde{\phi}+\hat{\phi})^{4}\right]\,\mathrm{d}^{n}x\right)$$
The argument is that terms involving $\tilde{\phi}\hat{\phi}$ don't matter because components of different Fourier modes are orthogonal. So we integrate $\hat{\phi}$ over $b\Lambda\leq|k|\leq\Lambda$ and our partition function changes from
$$\tag{5a} Z=\int D\tilde{\phi}\exp(-S[\tilde{\phi}])\int D\hat{\phi}\exp(-S[\phi])$$
arguing
$$\tag{5b}\exp(-\int\delta\mathcal{L}_{eff}(\phi)\,\mathrm{d}^{n}x) = \int D\hat{\phi}\exp(-S[\phi])$$
we get
$$\tag{5c} Z=\int D\tilde{\phi}\exp(-\int[\mathcal{L}(\tilde{\phi})+\delta\mathcal{L}_{eff}(\phi)]\,\mathrm{d}^{n}x)$$
This transformation is parametrized by $b$, and cannot be undone. But we could have $b=1$, which gives us our original partition function, and it is associative, hence it's a monoid (or a semigroup, depending on your preference of words).
Remark. Observe that $\delta\mathcal{L}(\phi)\sim\mathcal{O}(\lambda)$ which are corrections compensating for removal of large-$k$ components of $\phi$. (End of Remark)
How to get the Callan-Symanzik equation
This is incredibly handwavy. Like, summarizing the story of a three-day relationship between a 13-year old and a 17-year old resulting in 6 deaths (i.e., Romeo and Juliet) level of handwaviness.
So don't follow what I say as the gospel, it's just meant to give some intuition as to what's going on.
Basically, consider the same model, and derive the $n$-point correlation function from the perturbation series. It stands to reason under these circumstances it should be independent of the choice scale. When we impose these symmetry conditions, then consider an "infinitesimal change" in scale, we get a differential equation.
But we demanded no change! So this differential equation (describing the infinitesimal change) should vanish.
Remark. An adequate derivation would have required many pages of manipulation. (That's what I tried doing last night.) So be forewarned, this is just one intuition of what's going on with the renormalization group. There are others; see, e.g., Ticciati's Quantum Field Theory for Mathematicians. For a more detailed explanation/derivation, see Peskin and Schroeder's Introduction to Quantum Field Theory. (End of Remark)
References
Manfred Salmhofer's Renormalization (1999), pg 63 et seq.
Giuseppe Benfatto's Renormalization Group (1995) pg 95 et seq.
Leo P. Kadanoff's University of Chicago course Lecture Slides
N Singh's "Thermodynamical Phase transitions, the mean-field theories, and the renormalization (semi)group: A pedagogical introduction". arXiv:1402.6837
Janos Polonyi's "Lectures on the functional renormalization group method". CEJP 1 (2003) pp 1–71; eprint
Giuseppe Iurato's "On Wilson’s Renormalization Group Structure". International Journal of Algebra 6 no. 23 (2012) 1121 - 1125; eprint
Lubos Motl's answer to the post "Noether theorem with semigroup of symmetry instead of group"
Alex Nelson's answer is much better that mine, but it doesn't address your question at all.
(a) Does it form a group?
No, it doesn't. See bellow, to find out what does 'look like' (but isn't) a group.
(b) What are the elements of the group?
The group-like structure is the following. Being sloppy, effective action satisfies the folowing semigroup property:
$$S_\mathrm{eff}^{\lambda_1+\lambda_2}=S_{\mathrm{eff}}^{\lambda_1}\circ S_\mathrm{eff}^{\lambda_2}$$
(this says the same as $(AB)_{ij}=A_{ik}B_{kj}$ –the usual matrix product– adapted to an additive structure or $1$-parametric flows $\varphi_{s+t}=\varphi_s\circ \varphi_t)$.)
Consider certain scale, $M$, and the regularized propagator for your theory $G_{0M}$, that is you consider only momenta $0<p<M$. For another scales $\Lambda< \Lambda'$ lying between $0$ and $M$, then the claim is that the (Fourier transform of the) propagators satisfy
$$\hat G_{\Lambda M}=\hat G_{\Lambda \Lambda'}+\hat G_{\Lambda' M}.$$
The importance in particle physics relies on the following. Denote the effective action for the field $\phi$, with momenta in $[\Lambda,\Lambda']$ and interaction (potential) $V^\Lambda$ by $S_\mathrm{eff}^{\Lambda,\Lambda'}(\phi,V^\Lambda)$. Now, notice that $S_\mathrm{eff}^{\Lambda \Lambda'}(\,\cdot\,,\,V^\Lambda)$ is a potential $V'$ as well. The claim is
$$S_\mathrm{eff}^{0,M}(\phi;V^M )= S_\mathrm{eff}^{0 \Lambda'}(\,\phi, V'),$$
that is
$$S_\mathrm{eff}^{0,M}(\phi;V^M )= S_\mathrm{eff}^{0 \Lambda'}(\,\phi\,,\,S_\mathrm{eff}^{\Lambda \Lambda'}(\,\cdot\,,\,V^\Lambda)).$$
So, you'd be able to relate the Green functions for the interaction $V^M$ (for the theory on the range $[\Lambda,M]$) with the theory in $[\Lambda,\Lambda']$ with the new potential.