New command with two definitions

Yes, this is possible, using \NewDocumentCommand from xparse for example, making the second argument with {} behave like an optional argument then, i.e. using the g argument modifier.

Please note, that there is already a \set{} macro in the braket package, which provides for typesetting of sets, so I called the macro \myset instead and use \set{...} inside.

In principle, this could be done with an trailing optional argument with [] too, which is perhaps even better, because this enforces you to distinguish between the list version and the conditional version of the set notation.

Edit I've added the \mybetterset{}[] command as a variant to \myset, using the [] as 2nd optional argument.

\documentclass{article}

\usepackage{braket}
\usepackage{mathtools}
\usepackage{xparse}

\NewDocumentCommand{\myset}{mg}{%
  \IfValueTF{#2}{%
    \set{#1\;\vert\;#2}
  }{%
    \set{#1}%
  }%
}

%% The better command with [] as optional argument

   \NewDocumentCommand{\mybetterset}{mo}{%
  \IfValueTF{#2}{%
    \set{#1\;\vert\;#2}
  }{%
    \set{#1}%
  }%
}

\usepackage{hyperref}

\begin{document}
$\myset{1,2,3}$

$\myset{x}{x > 0}$

$\mybetterset{1,2,3}$

$\mybetterset{x}[x>0]$


\end{document}

You can do it, but I discourage you to. Better defining a command that distinguishes between \set{1,2,3} and \set{x|<condition>}.

You can find perhaps better methods in the documentation of mathtools.

\documentclass{article}

\usepackage{xparse}

\NewDocumentCommand{\set}{>{\SplitArgument{1}{|}}m}{\printset#1}

\NewDocumentCommand{\printset}{mm}{%
  \IfNoValueTF{#2}
   {% no |
    \{#1\}%
   }
   {% |
    \{\,#1\mid#2\,\}%
   }%
}

\begin{document}

$\set{1,2,3}=\set{x | 1\le x\le 3}$

\end{document}

Here is another soultion, with optional argument

\documentclass{article}
\newcommand{\set}[2][]{\left\lbrace\if\relax\detokenize{#1}\relax\else#1\mid\fi#2\right\rbrace}

\begin{document}
bla bla
\[\set{a,b,c},\set[x]{x>0}\]
$\set{a,b,c},\set[x]{x>0}$
\end{document}

Classical method

\documentclass{article}

\makeatletter
\newcommand{\set}{\@ifnextchar[{\@@set}{\@set}}
\def\@set#1{\left\lbrace#1\right\rbrace}
\def\@@set[#1]#2{\@set{#1\mid#2}}
\makeatother

\begin{document}
bla bla
\[\set{a,b,c},\set[x]{x>0}\]
$\set{a,b,c},\set[x]{x>0}$
\end{document}