Ignore Greek letters using `\mathcal`
Since you said XeLaTeX, just load unicode-math:
\documentclass{article}
\usepackage{amsmath}
\usepackage{fontspec}
\usepackage{unicode-math}
\setmainfont{Latin Modern Roman}
\setmathfont{Latin Modern Math}
\newcommand{\set}[1]{\mathcal{#1}}
\begin{document}
\begin{itemize}
\item $\set{A} = ...$
\item $\set{\Omega} = ...$
\end{itemize}
\end{document}
It is possible to test the slot number of each character in your argument and work with that, but this is a really simple fix.
To bypass math alphabet commands like \mathcal you can simply make all uppercase greek letters mathord instead of mathalpha. If conflict arises revert the letters with \mathchardef.
\documentclass{article}
\DeclareMathSymbol{\Gamma}{\mathord}{operators}{"00}
\DeclareMathSymbol{\Delta}{\mathord}{operators}{"01}
\DeclareMathSymbol{\Theta}{\mathord}{operators}{"02}
\DeclareMathSymbol{\Lambda}{\mathord}{operators}{"03}
\DeclareMathSymbol{\Xi}{\mathord}{operators}{"04}
\DeclareMathSymbol{\Pi}{\mathord}{operators}{"05}
\DeclareMathSymbol{\Sigma}{\mathord}{operators}{"06}
\DeclareMathSymbol{\Upsilon}{\mathord}{operators}{"07}
\DeclareMathSymbol{\Phi}{\mathord}{operators}{"08}
\DeclareMathSymbol{\Psi}{\mathord}{operators}{"09}
\DeclareMathSymbol{\Omega}{\mathord}{operators}{"0A}
\begin{document}
\[\mathcal{\Gamma\Delta\Theta\Lambda\Xi\Pi\Sigma\Upsilon\Phi\Psi\Omega}\]
\mathchardef\Gamma="7000
\mathchardef\Delta="7001
\mathchardef\Theta="7002
\mathchardef\Lambda="7003
\mathchardef\Xi="7004
\mathchardef\Pi="7005
\mathchardef\Sigma="7006
\mathchardef\Upsilon="7007
\mathchardef\Phi="7008
\mathchardef\Psi="7009
\mathchardef\Omega="700A
\[\mathcal{\Gamma\Delta\Theta\Lambda\Xi\Pi\Sigma\Upsilon\Phi\Psi\Omega}\]
\mathchardef\Gamma="00
\mathchardef\Delta="01
\mathchardef\Theta="02
\mathchardef\Lambda="03
\mathchardef\Xi="04
\mathchardef\Pi="05
\mathchardef\Sigma="06
\mathchardef\Upsilon="07
\mathchardef\Phi="08
\mathchardef\Psi="09
\mathchardef\Omega="0A
\[\mathcal{\Gamma\Delta\Theta\Lambda\Xi\Pi\Sigma\Upsilon\Phi\Psi\Omega}\]
\end{document}
