Can the quadratic formula be used with variable coefficients?

Yes! A variable just stands for a number we don't know, so anything you can do with numbers you can also do with variables. (Actually, sometimes a variable stands for a number we do know, but either way, it still is just a number.)

There is one thing to be careful of, though. The quadratic formula is only valid when you have a genuine quadratic equation, which means the coefficient of $z^2$ must be nonzero. So in this case, you can only use the quadratic formula to solve for $z$ in terms of $x$ and $y$ assuming that $xy\neq 0$ (or equivalently, that both $x\neq 0$ and $y\neq 0$).

Yes! In fact, you probably learned it with variable coefficients; e.g. in the form of solving the equation

$$ a x^2 + bx + c = 0 $$

when given the presumption that $a \neq 0$.

Write $$z^2xy - zx^2y + y = 0$$ as $$(xy)z^2 - (x^2y)z + (y) = 0$$

so that you can see $a = xy$, $b=-x^2y$ and $c=y$.

Then \begin{align} z &= \dfrac{-b \pm \sqrt{b^2-4ac}}{2a} \\ z &= \dfrac{x^2y \pm \sqrt{x^4y^2-4xy^2}}{2xy} \\ z &= \dfrac{x^2y \pm |y|\sqrt{x^4-4x}}{2xy} \\ z &= \dfrac{x^2 \pm \sqrt{x^4-4x}}{2x} \\ \end{align}

Notice that $y$ has disappeared. That is because $(xy)z^2 - (x^2y)z + (y) = 0 \iff y(xz^2 - x^2z + 1) = 0$.

So we need to notice that $y=0$ is a solution. If $y\ne 0$ then we get $z = \dfrac{x^2 \pm \sqrt{x^4-4x}}{2x}$; where we are going to need to require that $x \ne 0$ and, if we only want real-valued solutions, $x^4-4x \ge 0$.