Vector calculus in classical fields
Please read the edits/comments before continuing.
When in doubt, write out the indices.
So we start with
$$ {\cal L} = \left(\frac{\partial h}{\partial t}\right)^2 + (\nu \partial_i \partial^i h)^2 = \left(\frac{\partial h}{\partial t}\right)^2 + (\nu \partial_i \partial^i h)(\nu \partial_k \partial^k h) $$
and
\begin{align*} &\frac{\partial \cal L}{\partial( \partial_j h)} = \frac{\partial }{\partial( \partial_j h)} \nu^2 ( \partial_i \partial^i h)( \partial_k \partial^k h)\\ &= \nu^2 (\partial_i \delta_{ij})( \partial_k \partial^k h) + \nu^2(\partial_i \delta_{kj})( \partial_i \partial^i h)\\ &=\nu^2 \partial_j (\partial_i\partial^i h)\\ &=2 \nu^2 \partial_j \nabla^2h \end{align*}
This is true for each component $j$ so we may adopt the short hand
$$\frac{\partial \cal L}{\partial( \nabla h)} = 2 \nu^2 \nabla \nabla^2h $$
As for the negative sign, I have no idea where that comes from.
Edit: As pointed out by @Michael Seifert, the Euler Lagrange equations change with higher order derivatives! Therefore this answer is incorrect and I will type out the right solution as soon as possible.
The solution plays fast and loose with the calculus of variations, and uses a trick (or a rule of thumb) that you can usually get away with, but is not obvious to the beginner. Here's how it works:
The action is $$ S = \int d^2\textbf{x}\,dt \left[\left(\frac{\partial h}{\partial t}\right)^2 + (\nu \,\nabla^2h)^2\right]. $$ If we integrate the last term by parts (this is what is meant by "freely integrating by parts"), the action becomes to $$ S = \int d^2\textbf{x}\,dt \left[\left(\frac{\partial h}{\partial t}\right)^2 + \nu^2 [\nabla^2h] [\nabla \cdot (\nabla h)] \right] \\= \int d^2\textbf{x}\,dt \left[\left(\frac{\partial h}{\partial t}\right)^2 - \nu^2 [\nabla (\nabla^2h)] \cdot (\nabla h) \right] $$ If we take the derivative of this quantity with respect to $\nabla h$, we should obtain $$ \frac{\partial \mathcal{L}}{\partial (\nabla h)} = - \nu^2 \nabla (\nabla^2h), $$ which is pretty close to what's provided.
But where does the factor of two come from? The answer is that there are two "copies" of $h$ in the term in question; and when we take the functional derivative of $- \nu^2 [\nabla (\nabla^2h)] \cdot (\nabla h)$ with respect to $\nabla h$, we should really be differentiating with respect to both of them. This is much more obvious if you actually write out the variation of this term in terms of a variation $h + \delta h$: $$ \delta \left[\nu^2 (\nabla^2h)^2 \right] = 2 \nu^2 (\nabla^2 h) (\nabla^2 \delta h). $$ You can view the factor of two as coming from the fact that there are two "copies" of $\nabla^2 h$ in this expression. When you vary this term, you have to vary both the first "copy" and the second "copy", thereby picking up a factor of 2.