Is there a way to describe all finite groups $G$ such that $\operatorname{Aut}(G) \cong S_3$?

Partial answer: look at $G/Z(G)$, the subgroup of inner automorphisms of $Aut(G)$. This group has order $1,2,3$ or $6$. The first three cases all lead to $G$ being abelian (we use the well-known: if $G/Z(G)$ is cyclic, then $G$ is abelian). The only abelian case is $G \cong V_4$ as you mentioned.

Hence we can focus on $G/Z(G) \cong S_3$. Indeed, as you mentioned $G=S_3$ is an example. So we are now talking about central extensions of $S_3$. These are "measured" by the so-called Schur Multiplier (and really is a cohomology group, see here.) We can infer that $G' \cong A_3$. This last fact requires some more sophistication: there is a theorem that says: if $G$ has an abelian Sylow $p$-subgroup, then $p$ does not divide $|G' \cap Z(G)|$. The proof of this requires transfer (which is connected to cohomology theory) or character theory.

$\newcommand{\Span}[1]{\left\langle #1 \right\rangle}$$\DeclareMathOperator{\Aut}{Aut}$$\DeclareMathOperator{\Inn}{Inn}$Let me continue the work of Nicky Hekster, so let $G/Z(G) \cong S_{3}$, so that all automorphism are inner.

Let $a, b \in G$ be elements such that $a Z(G)$ has order $3$ and $b Z(G)$ has order $2$ in $G/Z(G)$, so that $[a Z(G), bZ(G)] = aZ(G)$, and thus $[a, b] = a z$ for some $z \in Z(G)$. Thus $[a z, b] = [a, b] = a z$. Redefining $a$ as $a z$, we may thus assume $[a, b] = a$. In particular, $a$ commutes with $[a, b]$, so that for every $n$ we have $[a^n, b] = a^n$. Since $a^3 \in Z(G)$, we have $a^3 = [a^3, b] = 1$, so that $a$ has order $3$.

Note that this shows that $$\tag{comm}G' = [G, G] = [G, \Inn(G)] = [G, \Aut(G)] = \Span{a}$$ has order $3$.

We have $a^b = a [a, b] = a^2 = a^{-1}$. If the order of $G$ is $2^h \cdot s$, with $s$ odd, we see that $a^{b^s} = a^{-1}$, so that replacing $b$ with $b^s$ we may assume that $b$ has order $2^l$, for some $l \ge 1$.

If $l > 1$, one sees that the map on $G$, that is the identity on $a$ and $Z(G)$, and maps $b$ to $b^{1 + 2^{l-1}}$, extends to an automorphism $\gamma$ of $G$ such that $[b, \gamma] = b^{2^{l-1}} \notin \Span{a}$, contradicting (comm). Therefore $b$ has order $2$.

We are nearly there. We have now $G = \Span{a, b} \times Z(G)$, where $\Span{a, b} \cong S_{3}$. We want to show that $Z(G) = 1$. If $Z(G) \ne 1$, we see that there are extra automorphisms coming from $Z(G)$, unless $Z(G) = \Span{w}$ has order $2$. But then the map that fixes $a$ and $w$, and maps $b$ to $b w$, extends to an automorphism $\delta$ of $G$ such that $[b, \delta] = w \notin \Span{a}$, once more contradicting (comm).