Why won't this compile? Argument of \h has an extra {

use

\e[{\h[n-1]}] 

Put the inner \h[..] into braces

If you define commands with optional argument through xparse, then you can nest them.

\documentclass{article}
\usepackage{amsmath}
\usepackage{xparse}

\newcommand{\grplus}{\dfrac{1+\sqrt{5}}{2}}
\newcommand{\grminus}{\dfrac{1-\sqrt{5}}{2}}

\newcommand{\eopt}{%
  \left(1+\grplus\right)-\f\left(1+\grminus\right)%
}

\NewDocumentCommand{\x}{O{1}}{a_{n-#1}}
\NewDocumentCommand{\e}{O{\eopt}}{\\&=\dfrac{\n #1}{\sqrt{5}}}

\NewDocumentCommand{\g}{O{n-1}}{\left(\grminus\right)^{#1}}
\NewDocumentCommand{\h}{O{n-1}}{\left(\grplus\right)^{#1}}

\newcommand{\n}{\g[n-2]}
\newcommand{\f}{\h[n-2]}

\begin{document}

$\begin{aligned}[t]
a_n&=\x+\x[2]\e[-\f+\h\g]\e\e[\h[2]]
\end{aligned}$

I also used aligned in order to get a better alignment.

There's no reason for plaguing you code with useless \ensuremath.