How to find the "most upright" orthogonal vector to another vector?
Using the observation made by dtldarek in his answer, you’re looking for the unit vector in the plane generated by $w=(a,b,c)$ and $e_3=(0,0,1)$ that is perpendicular to $w$ and has a non-negative $z$-coordinate. Observe that such a vector is also perpendicular to $e_3\times w$, so the vector you seek is $w\times(e_3\times w)$, normalized.
Restating your problem (and adding a condition), you are given a vector $\vec{v}=\langle a,b,c\rangle$ and you want to find a vector $\vec{w}=\langle x,y,z\rangle$ so that
- $\vec{w}\cdot\vec{v}=0$,
- $\|\vec{w}\|=1$
- $\vec{w}\cdot\vec{e}_3$ is maximized
As you state, you want to maximize $z$ subject to \begin{align*} ax+by+cz&=0\\ x^2+y^2+z^2&=1 \end{align*}
Perhaps a Lagrange multiplier approach would be good here: \begin{align*} 0&=\lambda a+2\mu x\\ 0&=\lambda b+2\mu y\\ 1&=\lambda c+2\mu z\\ 0&=ax+by+cz\\ 1&=x^2+y^2+z^2 \end{align*}
We should really assume that $a$ and $b$ are not both zero because otherwise $\vec{v}$ points vertically and there are no orthogonal, upward pointing vectors.
If $\mu=0$, then we get that $\lambda=\frac{1}{c}$ (and $c\not=0$), but then the first two equations are $0=\frac{a}{c}$ and $0=\frac{b}{c}$, so $a=0=b$ This contradicts our assumption, so we're ok here.
If $\mu\not=0$, then we can solve for $x$, $y$, and $z$: \begin{align*} x&=-\frac{\lambda a}{2\mu}\\ y&=-\frac{\lambda b}{2\mu}\\ z&=\frac{1-\lambda c}{2\mu} \end{align*} We can substitute these into the fourth equation (and cancel the $2\mu$'s) to get $$ -\lambda a^2-\lambda b^2-\lambda c^2+c=0. $$ Now, we make things simpler by adding the assumption that $\langle a,b,c\rangle $ is a unit vector, so we could rewrite this equation as $c=\lambda$. If not, we use $c=\lambda\|\vec{v}\|$ throughout the rest of this problem.
Therefore, \begin{align*} x&=-\frac{ac}{2\mu}\\ y&=-\frac{bc}{2\mu}\\ z&=\frac{1-c^2}{2\mu} \end{align*} Substituting all of this into the final equation gives $$ 4\mu^2=a^2c^2+b^2c^2+1-2c^2+c^4=(a^2+b^2+c^2)c^2+1-2c^2. $$ Therefore $$ 4\mu^2=1-c^2 $$ or $$ \mu=\pm\sqrt{\frac{1-c^2}{4}} $$ Substituting these into the formulae for $x,y,z$ gives that \begin{align*} x&=\mp\frac{ac}{\sqrt{1-c^2}}\\ y&=\mp\frac{bc}{\sqrt{1-c^2}}\\ z&=\pm\frac{1-c^2}{\sqrt{1-c^2}} \end{align*} Depending on the signs, you should get the largest and smallest that $z$ could be (provided I made no errors).
Assuming $\mathbb{R}^3$, a geometric approach would be to observe that the solution would belong to plane generated by $(a,b,c)$ and $(0,0,1)$. That forces the ratio between $x$ and $y$ to be the same as between $a$ and $b$. In other words, we have two equations:
$$\begin{cases}ax+by+cz=0\\ay-bx=0\end{cases}$$
If $c = 0$, then $(0,0,z)$ is your solution for any $z>0$. Otherwise, if $x=ka$ and $y=kb$, then $k(a^2+b^2)+cz=0$ and $z=-k\frac{a^2+b^2}{c}$. To maximize $z$ make sure $z\geq 0$, that is, $k\cdot c \leq 0$.
I hope this helps $\ddot\smile$