Confusion on symmetry and basis transformation

I suppose I'll formally write this up since there seems to still be some confusion. Let's firmly establish that our $U$ is a transformation from $a$ to $b$, that has it's representation in the $a$ basis as

$$\langle a_i |U|a_j\rangle = \langle a_i|b_j\rangle$$

Let's look at how the representation of $|\psi\rangle$ in the $a$ basis transforms when we go to the $b$ basis:

$$|\psi\rangle = \sum_{j}|b_j\rangle\langle b_j|\psi\rangle=\sum_{j}\sum_{i}|b_j\rangle\langle b_j|a_i\rangle\langle a_i|\psi\rangle$$

Now pick out the $k$'th component of $b$

$$\langle b_k|\psi\rangle = \sum_{j}\sum_{i}\delta_{kj}\langle b_j|a_i\rangle\langle a_i|\psi\rangle = \sum_{i}\langle b_k|a_i\rangle\langle a_i|\psi\rangle \\ = \sum_{i}(\langle a_i|b_k\rangle)^{\dagger}\langle a_i|\psi\rangle = \sum_{i}(\langle a_i|U|a_k\rangle)^{\dagger}\langle a_i|\psi\rangle.$$

Letting subscripts denote the basis (i.e. $|\psi\rangle_a \equiv \langle\vec{a}|\psi\rangle$, and likewise for $b$), we see that this is telling us $|\psi\rangle_b = U^{\dagger}|\psi\rangle_a$. Now, from $U|a_i\rangle=|b_i\rangle$ we know that $\Omega_b = U^{\dagger}\Omega_a U$, so lets check that everything is consistent with your $|\psi\rangle = \Omega|\phi\rangle$ when we hit it with $U^{\dagger}$. Keeping subscripts to denote the basis for absolute clarity:

$$|\psi\rangle_a = \Omega_a|\phi\rangle_a \to U^{\dagger}|\psi\rangle_a = U^{\dagger}\Omega_a|\phi\rangle_a$$

Looking at each side individually, we have

$$U^{\dagger}|\psi\rangle_a = |\psi\rangle_b \\ U^{\dagger}\Omega_a|\phi\rangle_a =U^{\dagger}\Omega_a U U^{\dagger}|\phi\rangle_a = \Omega_b |\phi\rangle_b,$$

which shows us that everything is nice and consistent:

$$ |\psi\rangle_a = \Omega_a|\phi\rangle_a \xrightarrow{U^{\dagger}} |\psi\rangle_b = \Omega_b|\phi\rangle_b $$


In the first example, you're taking $|b_i\rangle$ into $|a_i\rangle$, while in the second, you're doing the opposite. Explicitly, if $$|\psi\rangle = \sum_n b_n|b_n\rangle$$ then $$\hat U|\psi\rangle = \sum_n b_n\hat U|b_n\rangle.$$

But we don't know anything about $U|b_n\rangle$! To go from $|b_i\rangle$ into $|a_i\rangle$, we what we need is $\hat U^\dagger$. Then the calculation will come out as $\Omega \mapsto \hat U^\dagger \Omega \hat U$.