xparse creates infinite loop

It seems the \hlineX should be expandable, since a simplified example will cause the same problem.

\documentclass{article}

\usepackage{booktabs,xparse}
\protected\def\xline{\specialrule{1pt}{0pt}{0pt}}

\begin{document}
    \begin{tabular}{|l|l|l|}
        \hline
        1    & 2    & 3 \\
        \xline
        4    & 5    & 6 \\
        \hline
    \end{tabular}
\end{document}

The expandable variant of \NewDocumentCommand is \NewExpandableDocumentCommand, and since it defines expandable new command, the argument specification should not end with optional arguments. Therefore we have the following working example, with the syntax of \hlinex modified:

\documentclass{article}

\usepackage{booktabs,xparse}
    \NewExpandableDocumentCommand\hlineX{ O{0pt} O{#1} m }{%
        \specialrule{#3}{#1}{#3}%
    }

\begin{document}
    \begin{tabular}{|l|l|l|}
        \hline
        1    & 2    & 3 \\
        \hlineX[1pt][2pt]{1pt}
        4    & 5    & 6 \\
        \hline
    \end{tabular}
\end{document}

The answer of muzimuzhi Z shows what you should do to avoid the error (and you should probably accept that one) but your actual question asks why it loops rather than how to avoid the error.

The looping isn't really related to xparse other than looking for an optional argument is not allowed before \specialrule (or \hrule or \multicolumn or \rowcolor or similar special table cell commands) If any non-expandable material is seen the cell template starts and it is too late to insert the non-aligned material for the rule.

So a simpler example of the loop is

\documentclass{article}

\usepackage{booktabs}


\begin{document}


\tracingall
    \begin{tabular}{|l|l|l|}
        \hline
        1    & 2    & 3 \\
        \relax\specialrule{0pt}{0pt}{1pt}
        4    & 5    & 6 \\
        \hline
    \end{tabular}
\end{document}

As in your example this stops with several essentially spurious error messages but if you scroll past them you eventually hit a tight loop and will have to kill the program, \tracingall will show:

.
.
.
\par ->
{\hrule}

\par ->
{\hrule}

\par ->
{\hrule}
.
.
.

Which shows that essentially you have hit this well known loop condition:

\documentclass{article}

\begin{document}

{\let\par\relax x\hrule}

\end{document}

Which makes TeX run forever with no error.

\hrule, which makes a horizontal line across the page is only allowed between paragraphs, in vertical mode. If it is encountered in horizontal mode TeX does not raise an error it inserts \par to end the current paragraph before the \hrule token then proceeds as normal.

The thing to note is that unlike the similar situation of a vertical box ending in horizontal mode, TeX does not execute the primitive end-paragraph code, it literally inserts the token \par into the input stream.

So if as here (and for technical reasons in latex tabular) \par is locally defined to do nothing then \hrule triggers insertion of \par which does nothing, then the \hrule is seen again, which inserts \par....