Using \@ifstar to define a star variant

The reason is that with \foo*{test} * is the argument for \foo and {test} is just the word text in a group. \foo must be a macro without argument, which calls another macro with argument

\documentclass{article}

\makeatletter
\newcommand{\foo}{\@ifstar{\@foob}{\@fooi}}
\newcommand{\@foob}[1]{\textbf{#1}}
\newcommand{\@fooi}[1]{\textit{#1}}
\makeatother

\begin{document}

\foo{test}
\foo*{test}

\end{document}

Macro \foo is defined with one argument. If it is called as \foo*{test}, then the star becomes the argument (#1); then \foo is expanded and \@ifstar is called. But the following token { is not a star. The argument, the star, is then set as \textit{*}. The processing of \foo is finished and {test} is interpreted as group with the word test inside.

The example can be fixed the following way:

\documentclass{article}

\makeatletter
\newcommand*{\foo}{\@ifstar\textbf\textit}
\makeatother

\begin{document}

\foo{test}
\foo*{test}

\end{document}

Macro \foo is defined without argument, thus that \@ifstar can look for a following star. Then the code in the arguments for "with star" and "without star" of \@ifstar read the following argument.

If the code in the argument of \@ifstar is more complicate, then the more general answer of Mike helps.

In your definition the argument has already been read before the \@ifstar comes into the picture. So you would have to put the star behind the argument (remaek that it eats a space::

\documentclass{article}

\makeatletter
\newcommand{\foo}[1]{\@ifstar{\textbf{#1}}{\textit{#1}}}
\makeatother

\begin{document}

\foo{test}
\foo{test}*

\end{document}

But what you want is probably this:

\documentclass{article}

\makeatletter
\newcommand{\foo}{\@ifstar\textbf\textit}
\makeatother

\begin{document}

\foo{test}
\foo*{test}

\end{document}