How to write a macro that takes a variable number of arguments

This is quite easy with xparse and \NewDocumentCommand, having two optional arguments, however, you need to check for their existence.

Please note that oo could be replaced by gg to allow for optional {}{} arguments, but that's not recommended, but see below in this answer as an alternative.

\documentclass{article}

\usepackage{xparse}

\NewDocumentCommand{\com}{oo}{%
  \lambda%
  %Check if the first arg is given
  \IfValueT{#1}{%
    % Now check if has a 2nd argument as well. 
    \IfValueTF{#2}{% Yes, the 2nd one is present, use {f}(x) style
      \{#1\}(#2)%
    }{% No, use [f] style
      [#1]% 
    }%
  }%
}

\begin{document}

$\com$

$\com[f]$

$\com[f][x]$

\end{document}

Update with gg type (use it with care!) and another optional argument to use that instead of \lambda

\documentclass{article}

\usepackage{xparse}

\NewDocumentCommand{\com}{oo}{%
  \lambda%
  %Check if the first arg is given
  \IfValueT{#1}{%
    % Now check if has a 2nd argument as well. 
    \IfValueTF{#2}{% Yes, the 2nd one is present, use {f}(x) style
      \{#1\}(#2)%
    }{% No, use [f] style
      [#1]% 
    }%
  }%
}

\NewDocumentCommand{\comother}{O{\lambda}gg}{%
  #1%
  %Check if the first arg is given
  \IfValueT{#2}{%
    % Now check if has a 2nd argument as well. 
    \IfValueTF{#3}{% Yes, the 2nd one is present, use {f}(x) style
      \{#2\}(#3)%
    }{% No, use [f] style
      [#2]% 
    }%
  }%
}


\begin{document}

$\com$

$\com[f]$

$\com[f][x]$


$\comother$

$\comother{f}$

$\comother{f}{x}$

% Now with \beta instead of \lambda

$\comother[\beta]$

$\comother[\beta]{h}$

$\comother[\beta]{h}{y}$
\end{document}

Of course, the syntax uses the standard LaTeX convention of optional arguments in square brackets, in this case for both optional arguments.

Also, the OP implied that the command should be used not in math mode, so I have accommodated that.

\documentclass{article}
\newcommand\com[1][\relax]{\ifx\relax#1$\lambda$\else\def\firstarg{#1}\comhelp\fi}
\newcommand\comhelp[1][\relax]{\ifx\relax#1$\lambda [\firstarg]$%
  \else$\lambda \{\firstarg\}(#1)$\fi}
\begin{document}
\com\quad\com[f]\quad\com[f][x]
\end{document}

For a version that instead oeprates in math mode, here it is:

\documentclass{article}
\newcommand\com[1][\relax]{\ifx\relax#1\lambda\else\def\firstarg{#1}\comhelp\fi}
\newcommand\comhelp[1][\relax]{\ifx\relax#1\lambda [\firstarg]%
  \else\lambda \{\firstarg\}(#1)\fi}
\begin{document}
$\com\quad\com[f]\quad\com[f][x]$
\end{document}

Here's a LuaLaTeX-based solution. It works as a preprocessor: The Lua function comscan runs at a very early stage of processing, before TeX gets to do its usual work. The Lua function find all instances of \com and replaces with a different macro, depending on whether \com is found to have 0, 1, or 2 arguments.

% !TeX program = lualatex
\documentclass{article}
\usepackage{luacode}
\begin{luacode}
function comscan ( s )
  s = string.gsub ( s , "\\com(%b{})(%b{})" , "\\lambda\\{%1\\}(%2)" )
  s = string.gsub ( s , "\\com(%b{})", "\\lambda[%1]" )
  s = string.gsub ( s , "\\com(%A)" , "\\lambda%1" )
  return ( s )
end
luatexbase.add_to_callback( "process_input_buffer", comscan, "comscan" )
\end{luacode}
\providecommand{\command}{aaa} % dummy command
\begin{document}
$\com{f}{x}$, $\com{g}$, $ \com $, \command
\end{document}