How to declare a "bracket-style" math operator

I suggest you load the mathtools package and use its \DeclarePairedDelimiter macro to define a macro called, say, \inprod. This method automatically declares a companion macro called \inprod*, which auto-sizes the angle brackets. The size of the "fences" can be controlled directly as well, by writing \inprod[\big]{...}, \inprod[\Big]{...}, etc.

\documentclass{article} 
\usepackage{mathtools}
\DeclarePairedDelimiter\inprod\langle\rangle
\begin{document}

$\inprod{a,b}$ 
$\displaystyle \inprod*{\frac{a}{b},\frac{c}{d}}$

\end{document}

A variant of the mathtools solution, which takes care of empty arguments, with the xparse and etoolbox packages. It uses two arguments, separated by a comma; if some argument is empty, it is replaced with a dot:

\documentclass{article}
\usepackage{mathtools, nccmath}
\usepackage{xparse, etoolbox}

\DeclarePairedDelimiterX{\innerp}[1]{\langle}{\rangle}{\innpargs{#1}}
\NewDocumentCommand{\innpargs}{>{\SplitArgument{1}{,}}m}
{\innpargsaux#1}
\NewDocumentCommand{\innpargsaux}{mm}
{\ifblank{#1}%
{\ifblank{#2}{\,\cdot\,{,}\,\cdot\,}{{\,\cdot \,}{,}{\mkern2mu#2}}}%
{{#1\,}{,}\ifblank{#2}{\,\cdot\,}{\mkern2mu#2}}%
}%

%}%
\begin{document}

\[ \innerp{x, y}, \quad \innerp[\big]{\tfrac{1}{2}x, y} \quad \innerp*{\frac{x}{2}, y}\quad\innerp{x , }\quad \innerp{ , y}\quad \innerp{ , }\]

\end{document} 

In fact I remember a conversation about precisely this, but it may take me a while to find it. In the meantime:

\documentclass{article}
\def\iprelay#1,#2{\langle#1,#2\rangle}
\def\ip#1{\iprelay#1}
\begin{document}
\[
\ip{a,b}
\]
\end{document}