Transverse Magnetic (TM) and Transverse Electric (TE) modes
Bare with me, I don't remember every little step, but I hope this derivation helps you.
First remember how a wave travels through a waveguide (dielectric).
$$ E(x,y,z) = E^{0}(x,y)e^{-\gamma z}$$ $$ H(x,y,z) = H^{0}(x,y)e^{-\gamma z}$$
Then consider Ampere's and Faraday's Laws for a source-free region. $$ \triangledown \times H = j\omega\epsilon E $$ $$ \triangledown \times E = -j\omega\mu H $$
This produces 3 equations each (for the x, y and z directions):
$$ 1) \frac{\partial E_{z}}{\partial y} + \gamma E_{y} = -j\omega\mu H_{x}$$ $$ 2) \frac{\partial E_{z}}{\partial x} + \gamma E_{x} = j\omega\mu H_{y}$$ $$ 3) \frac{\partial E_{y}}{\partial x} - \frac{\partial E_{x}}{\partial y} = -j\omega\mu H_{z}$$ $$ 4) \frac{\partial H_{z}}{\partial y} + \gamma H_{y} = j\omega\epsilon E_{x}$$ $$ 5) \frac{\partial H_{z}}{\partial x} + \gamma H_{x} = -j\omega\epsilon E_{y}$$ $$ 6) \frac{\partial H_{y}}{\partial x} - \frac{\partial H_{x}}{\partial y} = j\omega\epsilon E_{z}$$
We can combine (1) and (5) and combine (2) and (4) due to like terms to generate equations for $H_{x}$ and $E_{x}$ which become (7) and (9). We rearrange equations (3) and (6) for $H_{y}$ and $E_{y}$, respectively.
$$ 7) H_{x} = \frac{-\gamma}{h^2} \frac{\partial H_{z}}{\partial x} + \frac{j\omega\epsilon}{h^2} \frac{\partial E_{z}}{\partial y}$$
$$ 8) H_{y} = \frac{-\gamma}{h^2} \frac{\partial H_{z}}{\partial x} - \frac{j\omega\epsilon}{h^2} \frac{\partial E_{z}}{\partial x}$$
$$ 9) E_{x} = \frac{-\gamma}{h^2} \frac{\partial E_{z}}{\partial x} - \frac{j\omega\mu}{h^2} \frac{\partial H_{z}}{\partial y}$$
$$ 10) E_{x} = \frac{-\gamma}{h^2} \frac{\partial E_{z}}{\partial y} + \frac{j\omega\mu}{h^2} \frac{\partial H_{z}}{\partial x}$$
And remember $ h^{2} = \gamma^{2} + \beta^{2}$, where $\beta = \omega \sqrt{\mu\epsilon}$
Transverse components $E_{x}, E_{y}, H_{x}, H_{y}$ are expressed in terms of the longitudinal components $E_{z}, H_{z}$. And we are given three cases:
1) Transverse Electric (TE): $$ E_{z} = 0, H_{z} \neq 0$$
2) Transverse Magnetic (TM): $$ E_{z} \neq 0, H_{z} = 0 $$
3) Transverse Electromagnetic (TEM): $$ E_{z} = H_{z} = 0 $$
Where in the case of TEM modes equations (7) through (10) break down unless $ h = 0 $ meaning: $$ \gamma^{2} + \beta^{2} = 0 $$ $$ \gamma^{2} = -\beta^{2} $$ $$ \gamma = j\beta = j\omega\sqrt{\mu\epsilon} $$
Now here we need to bring in the Helmholtz equation to solve the partial differential for TE and TM modes: $$ \triangledown^{2} A + k^{2}A = 0 $$
In TE modes, we need $H_{z}$, which is our $A$ in the Helmholtz equation, and our factor $k^{2}$ is $\beta^{2}$.
Substituting: $$ \triangledown^{2} H_{z} + \beta^{2} H_{z} = 0 $$
Expand: $$ \frac{\partial^{2} H_{z}}{\partial x^{2}} + \frac{\partial^{2} H_{z}}{\partial y^{2}} + \frac{\partial^{2} H_{z}}{\partial z^{2}} + \beta^{2} H_{z} = 0 $$
$$ \frac{\partial^{2} H_{z}}{\partial z^{2}} = -\gamma^{2}H_{z}^{0}(x,y)e^{-\gamma z} $$
$$ \frac{\partial^{2} H_{z}^{0}}{\partial x^{2}} + \frac{\partial^{2} H_{z}^{0}}{\partial y^{2}} + (\gamma^{2} + \beta^{2})H_{z}$$ Since $h^{2}$ = $\gamma^{2} + \beta^{2}$ we conclude: $$ \frac{\partial^{2} H_{z}^{0}}{\partial x^{2}} + \frac{\partial^{2} H_{z}^{0}}{\partial y^{2}} + h^{2}H_{z} = 0$$
Repeat the same steps above for TM modes, where we need $E_{z}$
$$ \triangledown^{2} E_{z} + \beta^{2} E_{z} = 0 $$
And therefore: $$ \frac{\partial^{2} E_{z}^{0}}{\partial x^{2}} + \frac{\partial^{2} E_{z}^{0}}{\partial y^{2}} + h^{2}E_{z} = 0$$
Now the reason for there being only two sets of components is due to the fact that the wave propagates along a single direction given the waveguide. The key factor being that Electric and Magnetic Fields are ALWAYS perpendicular to eachother. This is a primary principle that Maxwell discovered. The two always travel together in electromagnetic waves.
For example:
Where in TM modes the electric field is in the direction perpendicular to that of propagation, so ONLY the magnetic field propagates within the waveguide, and vice versa for TE modes. This is why the electric or magnetic components are considered 0 (given that we're assuming z to be the direction of propagation).
So you have two instances for TM and TE waves, where the electric field is zero or the magnetic field is zero - why you have two sets of equations.
This differs in TEM modes where neither propagate in the direction of the waveguide, however at least two conductors are required for any TEM modes to exist.
Hints:
We are evidently only supposed to solve for $z$-dependence (as opposed to $x$- and $y$-dependence).
Note that the two variables $E_z$ and $H_z$ can be eliminated.
In the reduced coupled ODE system of four first-order ODEs and four variables $(E_x,E_y,H_x,H_y)$, note that the variables couple two and two together. Which pairs?
Within one such pair, it is possible to eliminate one of the variables to form a second-order ODE in one variable. What well-known second-order ODE would that be?
Assume that the second-order ODE is of oscillatory type. What does this mean? It has two integration constants (e.g. amplitude and phase), which we collectively will refer to as a single mode.
Repeating 4 and 5, it is possible to get four second-order ODEs in one variable each.
However, remember that the four variables are not independent of each other, but coupled two and two.
Finally, deduce how many independent modes exist?
Notice that there are three equations for the quantities $(E_y,H_x,H_z)$:
$$ \frac{\partial E_y}{\partial z} = -i\omega\mu_0 H_x $$
$$ i\beta E_y = i\omega\mu_0 H_z $$ $$ \frac{\partial H_x}{\partial z} -i\beta H_z = -i\omega\varepsilon_0\varepsilon E_y $$
These are totally independent of the equations for the three other components, $(H_y,E_x,E_z)$. If you knew exactly the solutions for $(E_y,H_x,H_z)$, you would still have precisely no additional information about the other solutions, $(H_y,E_x,E_z)$. To solve for these quantities, you'd still have to solve the differential equations that govern them. This means that the TE modes, consisting of the three components $(E_y,H_x,H_z)$, are independent of the TM modes, consisting of $(H_y,E_x,E_z)$.
If the two sets of equations are independent of each other, you only have to consider three equations and three variables at a time. You are free to work on solving the TE mode quantities without regard for the values of the TM quantities. The same goes for the TM quantities, you can solve for them without regard for the TE values. If you're only working on three field components at a time and don't care about the others, you might as well set the other ones to zero. Then the full set of equations would reduce to just the mode you're considering. This holds for both modes. It's not that there are two sets of solutions exactly, but that the solutions decouple into two groups that have nothing to do with each other, so you're free to consider these groups independently of each other. Setting the other group to zero is a way to mathematically enforce this choice of ingoring the other group.
The fields in general will have all six components, but when you go to do the calculations, you only have to solve for three components at a time. This is what your text means about two sets of self-consistent equations.