Post-tenure move: Reference letter from institution to which I'm applying

A letter written by someone the reader knows is likely to be more credible than a letter written by someone unknown. So it makes sense to get a letter from someone at X when you know the reader will be at X.

they might be seen as having a vested interest

Most letter writers have an interest in the success of the person they are recommending. A writer who works at X will have additional knowledge of the interests of the whole university, which could result in a more useful letter.

would it be weird to draw on a reference from this collaborator when applying to the University of X?

No.

Sorry, I don't see an issue for you. If you ask for a letter they might say no, of course, and they might want to game a new position (of course), but that is out of your hands. Letters from collaborators are a good thing, especially if they are also well respected academics.

Ask, and see what happens. If anyone thinks there are ethical or other issues, they aren't on you.

Or at least ask them for advice on whether a letter from them would be good or not.