Spontaneous symmetry breaking and conservation laws revisited
This issue is resolved the same way almost all disputes over semantics are resolved in physics: there are multiple definitions of the same words at play here, each of which are perfectly legitimate, and the seemingly contradictory statements are just based on different definitions.
Whenever you set up a calculation in physics, whether in Newtonian mechanics or in quantum field theory, you pick a subset of the universe to count as your "system". Everything else is treated as external, a "background" that influences the system but whose detailed state is not kept track of.
For example, consider a ball dropped near the surface of the Earth. The Earth and ball together have three-dimensional translational symmetry, so $\mathbf{p}_{\text{ball}} + \mathbf{P}_{\text{Earth}}$ is conserved. But in practice, you might not want to consider the motion of the Earth. Instead, you can treat the ball as your system, and account for its interaction with the Earth by adding a potential term $U(\mathbf{x}) = m g z$. But this term is not invariant under translations in the $z$-direction (i.e. from this perspective, the Earth has spontaneously broken this symmetry), so $p_{\text{ball}, z}$ is not conserved, though $p_{\text{ball}, x}$ and $p_{\text{ball}, y}$ still are.
So is momentum "really" conserved in this situation? Is there "really" translational symmetry? It's not a sharp question: there are just two separate translational symmetries one might want to consider, which correspond to different momenta. One is conserved, another isn't.
Can someone explain in what sense the momentum (and angular momentum) conservation is violated in crystalline solids but not in liquids?
When you have a sample of solid or liquid sitting in your lab, there's always a translational symmetry that corresponds to moving the sample around in your lab (neglecting the effects of gravity). This is a perfectly legitimate and important symmetry, because it tells us that we can do the experiment anywhere we want in the lab, and it tells us that the ordinary momentum of the sample is conserved.
But once you've fixed where the sample goes, and want to analyze the dynamics within the sample, this symmetry isn't useful anymore. Instead, when condensed matter physicists talk about translation, they mean a symmetry that translates the excitations of the sample within it, without translating the whole sample itself. For instance, in a solid you might translate the electrons without moving the atomic lattice, or in a liquid you might translate a sound wave within the liquid without moving the bulk liquid itself. The corresponding momentum-like quantity is called crystal momentum for solids (or more generally, quasimomentum), and for solids it isn't conserved because the interaction with the lattice isn't translationally invariant.
But isn't regular momentum still conserved? Absolutely. If you want, you can artificially separate, say, the ordinary momentum of a phonon from the regular momentum of the rest of the crystal lattice. When the phonon's ordinary momentum changes, the rest of the crystal's ordinary momentum changes in the opposite way -- it serves as a "reservoir" for ordinary momentum, just like the Earth serves as a "reservoir" that allowed $p_{\text{ball}, z}$ to change.
The situation isn't any different in particle physics. For example, the universe as a whole is still $U(1)_Y$ symmetric, and accordingly the hypercharge of the entire universe is conserved. But this fact is not particularly useful in constraining reactions that we can see. The reason is that $U(1)_Y$ is spontaneously broken by the Higgs field, and hence serves as a background reservoir of hypercharge, allowing the total hypercharge of excitations to change. We're so used to living inside this situation that we often summarize it as "$U(1)_Y$ is broken". Similarly, condensed matter physicists are so used to living, e.g. in a crystal lattice that they might just say "translational symmetry is broken".
I think it's easiest to see what going on by considering particle number conservation in superfluids. For a finite system (e.g. a system defined on a finite lattice of points at which particles may or may not exist), the ground state is always an eigenstate of the total particle number operator $\hat{N} = \hat{\varphi}^\dagger \hat{\varphi}$. Therefore, there are no particle-number fluctuations and the ground-state expected number of particles in the system $\langle \hat{N} \rangle$ is quantized to be an integer.
If we introduce the conjugate thermodynamic variable - in this case the chemical potential $\mu$ - then in general there will be a term in the Hamiltonian of the form $-\mu \hat{N}$. If we plot the ground-state number of particles $\langle \hat{N} \rangle$ as a function of the chemical potential, then since the latter quantity is quantized to an integer, the resulting curve will be a series of flat segments (with a finite number of fine-tuned values of the chemical potential at which two eigenstates of $\hat{N}$ with eigenvalues that differ by 1 become exactly degenerate ground states, so the curve jumps like a step function). So the system has definite particle number at almost all values of $\mu$, except for a measure-zero set of values of $\mu$ where the ground state is degenerate and there is an ambiguity of 1 in the number of particles.
As the system gets larger and larger, these jumps occur at more and more different values of $\mu$ - but always at just a finite number of values, so still "almost nowhere".
In the infinite system limit, the absolute number of particles stops being a useful quantity to talk about because it depends on the lattice size. The relevant quantity is the particle density - that is, the fraction of lattice sites that are filled - since this is the quantity that converges to a finite value in the infinite-system limit: $\langle \hat{\rho} \rangle := \langle\hat{N}\rangle/V_\text{lattice}$. While $\hat{N}$ always jumps in integer steps, $\hat{\rho}$ jumps in steps of $1/V_\text{lattice}$, which becomes arbitrarily small in the large-system limit. So the dependence of $\hat{\rho}$ on $\mu$ is a series of piecewise flat curves that actually approaches a continuous curve, somewhat like the Cantor function (although the limiting function is usually "nicer" than the Cantor function, e.g. has nonzero derivative).
There can be two different regimes of $\mu$ in the infinite-system limit: some values of $\mu$ are "stable" against flucuations, i.e. the fractional occupation $\langle \hat{\rho}\rangle$ is unchanged by small changes in $\mu$. Such values of $\mu$ (which occur over finite-length intervals) are not accumulation points of the "jump" values of $\mu$ in the finite systems, and the ground state of the finite system is typically non-degenerate. In this sense, the conservation of $\hat{\rho}$ "survives" the thermodynamic limit and the symmetry $\hat{\varphi} \to e^{i \theta} \hat{\varphi}$ remains unbroken.
At other (intervals of) values of $\mu$, the "jump" points accumulate as the system size gets larger in such a way that the density of jump points scales like $V$. This means that over any small range of values of $\mu$, there are huge number of nearly degenerate low-lying states with different particle numbers, and the "right" ground state in that nearly degenerate manifold becomes a coherent state superposition of states with different particle number. ("Right" in the sense that it respects the cluster decomposition property and has a better-behaved infinite-system limit.)
So from an operational perspective, the fact that the symmetry is sponaneously broken in the infinite-system limit operationally means that there are such a large number of nearly-degenerate ground states in a large system that if the conjugate thermodynamic quantity changes by an infinitesimal amount $\delta \mu$, then the expectation value $\langle \hat{\rho} \rangle$ changes by an amount proportional to $\delta \mu$, i.e. $\frac{d\langle \hat{\rho} \rangle}{d\mu} \neq 0$. So the ground states become so highly degenerate in the large-system that even infinitesimal fluctuations in $\mu$ (which can never be completely eliminated in practice) will change the ground state. While conservation of $\rho$ may strictly speaking still hold for an exact choice of $\mu$, in practice microscopic fluctuations in $\mu$ will be enough to result in experimental measurements of $\hat{N}$ yielding non-deterministic answers. That's why people say that the conservation law is "violated" in the infinite-system limit.
(knzhou's equally correct answer focuses on a slightly different aspect, which in this case corresponds to the actual microscopic physics behind what's going on when you set the chemical potential. As he says, that involves particles crossing between the interface between what you're considering to be "the system" and "the environment".)