What is so special about spontaneous symmetry breaking? (time reversal example)
Indeed, one of the definitions of spontaneous symmetry breaking is in terms of its susceptibility:
Suppose we add a symmetry breaking perturbation $h \; \delta H$ to our Hamiltonian (as you do), if $$ \lim_{h \to 0} \lim_{N \to \infty} \langle m \rangle \neq 0 $$ then we say our system has spontaneous symmetry breaking.
(Note: $N$ is the number of spins in our system. Indeed, on a mathematical level, non-analyticities can only arise in the thermodynamic limit.)
What is special is that any arbitrarily small perturbation will do. Imagine you have a million spins. If the state is originally in a symmetric state (i.e. not symmetry broken yet), then even if I just apply an arbitrarily small magnetic field on a single spin, the whole system will choose that orientation.
You suggest that the fact one in principle needs the environment to 'make the choice' that this is not really spontaneous. It is true that in that philosophical sense of the word, the direction of magnetization is not 'spontaneous'. But what can be called spontaneous in the universe? If I perfectly balance an egg, then the direction it will eventually roll when it loses its balance is spontaneous (or not spontaneous) in exactly the same sense. And note that once the egg has rolled down (and stopped), the tiny perturbations in the air which influenced its original direction are now no longer sufficient to change its position. I.e.: after the `spontaneous' process, the system is now stable.
The same thing happens in the above magnet: once it has chosen a direction of magnetization, then changing the applied magnetic field on that single spin I mentioned before will not change the total magnetization. So in that sense it is not true that it is so susceptible! One needs to apply an extensive magnetic field (i.e. a field that acts on most of the spins) to change the direction of the magnetization.
That is what is so funny about these systems:
An arbitrarily small perturbation can create a magnetization, but it cannot change it!
On a more quantum-mechanical note, if one has a Hamiltonian whose ground state should display spontaneous symmetry breaking, then if one takes the ground state to be in a symmetric superposition (which one can always do), then this state has ridiculously long entanglement. These are called cat states (in reference to Schrodinger's cat). This is a natural consequence of the above: an interaction with a single spin has to influence all spins at once, which is only possible if every single spin is entangled with every other spin. An example is the state $|\uparrow \uparrow \uparrow \cdots \rangle + |\downarrow \downarrow \downarrow \cdots \rangle$. (Indeed: an interaction with a single spin will collapse this 'cat state' to a product state, and then it is clear that any subsequent single-spin interaction cannot flip the state to the other product state.) Indeed, the way symmetry breaking phases are classified in one spatial dimension is in terms of these entanglement properties [Schuch et al., 2010].
You've already mentioned the exact definition of spontaneous symmmetry breaking:
Spontaneous symmetry breaking for a system that is described by a hamiltonian $H$ with ground state $\left| g \right\rangle$ happens where there is a symmetry transformation of $H$ that doesn't leave the ground state invariant $$[T,H] = 0 \text{ but } T \left| g \right\rangle \neq 0. $$ Much like a stick that's standing on its tip can be rotated around itself but will eventually fall back to a "ground state" that doesn't have this rotational symmetry anymore.
The example you are quoting is a statistical theory, so the fluctuations that drive the system into a state of non-vanishing magnetisation below the critical temperature are already built in from the start.
Spontaneous symmetry breaking is a key ingredient in the Standard Model of particle physics, it is used to explain why the particles that mediate the weak force are massive (W and Z bosons, this is remarkable because you can't achieve that by just putting a mass term in the Lagrangian!).
But it also works the other way around, in that it explains why sometimes there are massless modes in a system (see Goldstone bosons).