How to handle syntax errors in a macro?

Perhaps not the easiest or quickest way, but the \@ifnextchar way could be exploited to test whether there's a ( after the optional argument [] (it does not check for ) however.

Use \GenericInfo{...}{...} to write some information to the Log File

\documentclass{article} 

\usepackage{tikz}


\begin{document}    

\makeatletter
\def\macro{\@ifnextchar[{\@macroopt}{\@macronoopt}}

\gdef\@temp@{}%
\def\@macroopt[#1]{\gdef\@temp@a{#1}\@macroparen}
\def\@macronoopt{\@macroparen}

\def\@macroparen{\@ifnextchar({%
    \GenericInfo{\macro}{Parenthesis used}
  }{%
    \@macronoparen%
  }%
}

\def\@macronoparen#1{%
  \begingroup
  \foreach \ptx/\pty/\name in {#1}{%
    \expandafter\node\expandafter[\@temp@a] at (\ptx,\pty){\name};}%
  \endgroup
}


\makeatother


\begin{tikzpicture}
  \macro[blue]{2/3/a,4/4/b,5/0/c}
  \macro[red,circle](2/3/a,4/4/b,5/0/c) % Error/Warning message to log
\end{tikzpicture}    


\end{document}

Some content from \jobname.log

(/usr/local/texlive/2015/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg

File: epstopdf-sys.cfg 2010/07/13 v1.3 Configuration of (r)epstopdf

for TeX Live ))

Parenthesis used on input line 35.

If ( is used, it will be absorbed as #2.

\documentclass{article} 
\usepackage{tikz}

\makeatletter
\def\macro{\pgfutil@ifnextchar[{\macrob}{\macrob[]}}
\def\macrob[#1]#2{%
  \begingroup
  \def\parenthesis@for@err{(}%
  \def\maybe@parenthesis{#2}%
  \ifx\maybe@parenthesis\parenthesis@for@err
    \PackageError{mynicepackage}{You donkey!}{I told you to use {}, not ()}%
    \expandafter\@firstoftwo
  \else
    \expandafter\@secondoftwo
  \fi
  {\absorb@bad@arg}% try and recover somehow
  {\foreach \ptx/\pty/\name in {#2}{%
   \node[#1] at (\ptx,\pty){\name};}%
   \endgroup}%
}
\def\absorb@bad@arg#1){\endgroup}
\makeatother

\begin{document}    

\begin{tikzpicture}
  \macro[red,circle]{2/3/a,4/4/b,5/0/c}
  \macro[red,circle](2/3/a,4/4/b,5/0/c)
\end{tikzpicture}    

\end{document}

Here's the terminal output:

! Package mynicepackage Error: You donkey!.

See the mynicepackage package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                              

l.28   \macro[red,circle](
                          2/3/a,4/4/b,5/0/c)
? h
I told you to use {}, not ()