What is the mathematical understanding behind what physicists call a gauge fixing?

Given a gauge potential, $A_\mu$, which can be thought of as a connection form on a vector bundle associated to some $G$-principal bundle, we know that under a gauge transformation, which is a diffeomorphism of the vector bundle fixing each fibre that is induced by the representation $\rho$ of the gauge group on the vector bundle, $A_\mu$ transforms as $$ A_\mu^\prime = \rho(g) A_\mu \rho(g)^{-1} + (\partial_\mu \rho(g))\rho(g)^{-1} $$ for $g \in G$. This is often abbreviated, without explicitly mentioning the representation, as $$ A_\mu^\prime = g A_\mu g^{-1} + (\partial_\mu g)g^{-1}. $$ So we have a whole series of gauge potentials, one for each $g$, that we know are gauge-equivalent. Now, we know that physics should be gauge invariant, meaning any physical results we derive don't depend on which specific gauge potential we choose in a given gauge class. So we might as well choose a convenient gauge (convenience depends on the nature of the problem we are trying to solve). Gauge fixing is the process of choosing a convenient gauge (i.e. connection form) for the problem in hand.

In short: if $\pi: P \rightarrow M$ is a principal $G$-bundle, then fixing a gauge corresponds to making a choice of section $\sigma_{U}: U \rightarrow \pi^{-1}(U)$, for some open $U \subset M$. A different choice of section, say $\sigma_{V}: V \rightarrow \pi^{-1}(V)$ will then be related to $\sigma_{U}$ by the action of $G$, so for example $\sigma_{V}(x) = g_{UV}(x)\sigma_{U}(x)$ for some $x \in U \cap V$ with $g_{UV}(x) \in G$. The ambiguity in choosing a section corresponds to gauge freedom, and the group action relating two different choices of section is a gauge transformation, and $G$ is said to be the gauge group.

The simplest example demonstrating this concept actually isn't strictly a principal $G$-bundle, but rather an associated vector bundle of a principal $G$-bundle, since a wavefunction actually takes values in a some copy of $\mathbb{C}^{n}$ rather than in a group. To avoid obfuscating the point however I won't go into the particulars of this.

Let $P \rightarrow M$ be a principal $U(1)$-bundle over some base manifold $M$, then a wavefunction is a section of the vector bundle associated to $P$, denoted in the literature commonly as $P \times_{U(1)} \mathbb{C}^{n}$; all that we need to now of this is that a fibre of $P$ is isomorphic to the vector space $\mathbb{C}^{n}$, which represents the value of a wavefunction. For concreteness, say that $n = 1$ so that the wavefunction $\psi(x) \in \mathbb{C}$ represents a complex scalar field, or a spin-$0$ particle. As the probability of the particle being at the point $x \in M$ is $|\psi(x)|^{2}$, choosing a different section $e^{i\alpha}\psi$ for $e^{i\alpha}\in U(1)$ needs to give the same physical result. Choosing either section $\psi$ or $e^{i\alpha}\psi$ is the same as fixing a gauge, and they are related to one another by the action of an element of $U(1)$, in this case just multiplication. So here a gauge transformation is just multiplication/division by $e^{i \alpha}$, and the gauge group is $U(1)$.

Depending on the topology of the base manifold $M$, it could be that $\alpha$ is actually a function $\alpha:M \rightarrow \mathbb{R}$ (if it is just a constant then the gauge transformation is called a global one, since it holds globally on $M$). If it is a smooth function, then this will give rise to the notion of connection 1-forms and curvature 2-forms, which are called gauge potentials and gauge forces respectively in the physics literature. These are required since both $\psi$ and $e^{i\alpha}\psi$ need to physically represent the same state, but with the introduction of derivatives in the Hamiltonian or Lagrangian, terms like $\partial_{x}\alpha$ start popping up that need to be dealt with by introducing a covariant derivative.