The geometry of the action of the semidirect product
This is a bit of a special example in a couple ways, but maybe it might be useful for building intuition.
Suppose $1 \longrightarrow H \longrightarrow G \longrightarrow \mathbb{Z} \longrightarrow 1$ is a short exact sequence of groups. The algebraic situation you describe at the end of your question is sometimes described as saying “the sequence splits.” It’s a fact that’s not hard to prove that any sequence of the above form splits.
Algebraically, there is an element $\Phi\in\operatorname{Aut}(H)$ such that if $t$ is a generator for $\mathbb{Z}$ and $g \in H \le G$, we have $tgt^{-1} = \Phi(g)$.
Topologically, such a short exact sequence of groups corresponds to a fiber bundle over the circle, in the sense that if $E \longrightarrow S^1$ is a bundle with fiber $F$, there is such a short exact sequence with $\pi_1(F)$ playing the role of $H$ and $\pi_1(E)$ playing the role of $G$.
In good situations, (e.g. if $F$ is a $K(H,1)$), we can realize the algebraic picture topologically: if there is some map $f\colon F \to F$ such that the action of $f$ on the fundamental group yields $\Phi$, then we can build the bundle $E$ in the following way: $$ E = (F \times [0,1])/_{(x,1) \sim (f(x),0)}.$$ In other words, $E$ is built by taking the product of $F$ with an interval (think of the interval as the “vertical” direction), and then gluing the “top” of the product to the “bottom” via $f$. We call $E$ the “mapping torus” of $f$.
Thinking geometrically may not always be clear from here. E.g. if $F$ is a manifold admitting a Riemannian metric of non-positive curvature and $f\colon F \to F$ is a diffeomorphism, the bundle $E$ may or may not also admit a Riemannian metric of non-positive curvature. The product $E = F\times S^1$ always does in this situation, of course, but for instance when $F$ is a $2$-torus, the only other examples of bundles $E$ admitting a non-positively curved metric are finitely covered by the product.
What’s going on here is that especially in low dimensions, the geometry of $E$ is really intimately tied up with the dynamics of the action of $f$ (or $\Phi$) on $F$ (or $H$).
Suppose $K\circlearrowright X$ is an action of $K$ on some set. We have the structure homomorphism $\varphi:K\rightarrow \text{Sym}(X).$
Let $H$ act on $K$ by automorphisms, and suppose these automorphisms can be realized as inner automorphisms within $\text{Sym}(X)$. That is, the action is given by some $\theta:H\rightarrow\text{Sym}(X)$ such that conjugations by $\theta(H)$ leave $K$ invariant. Equivalently, the $H$-action takes stabilizer subgroups of $K\circlearrowright X$ to stabilizers (it may permute them nontrivially).
Then we can construct an action of $K\rtimes H$ on $X$ by $(k,h).x = \varphi(k)\cdot\theta(h).x$, where the product $\varphi(k)\theta(h)$ is just taken in $\text{Sym}(X)$. Here, the multiplication rule for $K\rtimes H$ is $$(k_1, h_1)\cdot (k_2,h_2) = (k_1 \cdot (h_1.k_2), h_1 h_2). $$
Any action arises in this way, since $H$ acts on $K$ by conjugation in the semidirect product $K\rtimes H$ and therefore it also acts by conjugation in the image under the structure morphism $K\rtimes H \rightarrow \text{Sym}(X)$ of an action of the semidirect product on $X$. This explains when an action of $K$ can be extended to an action of $K \rtimes H$.
It is not clear how to visualize the above. So let's pass to a nice special case.
Given another action of $H$ on a set $Y$, we can extend $H\circlearrowright Y$ to an action $K\rtimes H \circlearrowright Y$ (let the latter act via the quotient $K\rtimes H\twoheadrightarrow H$). Then we can produce an action of $K\rtimes H \circlearrowright Y \times X$. This action descends to the quotient $H\circlearrowright Y$ so that $K$ fixes each fiber $\{y\}\times X$, and $H$ acts by permuting fibers and "twisting" $X$.
If the $H$-action is faithful, the action of any element $(k,h)$ can be nicely separated into an $h$-part and a $k$-part, the $h$-part being uniquely identified by the action on $Y$. Thus given a permutation $\sigma$ of $Y\times X$ we can write that it is of the form $(k,h)$ for a known $h$, and then compute the permutation $(y,x)\mapsto \sigma\left((e,h^{-1}).(y,x)\right)$, which acts the same as $(k,h)\cdot(e,h^{-1}) = (k,e)$.
One gets some examples which appear different, but are isomorphic to these, by choosing different identifications between fibers than $\text{id}_X : \{y_1\}\times X \rightarrow \{y_2\}\times X$. These identifications may be analogous to the connection described in the question.