Point circle from pair of straight line equation

Take $ax^2+2hxy+by^2=0$. Multiply by $4a$ $(a\neq 0)$ and complete the square to obtain the equivalent equation $$(2ax+hy)^2+(4ab-h^2)y^2=0$$

If $4ab-h^2\gt 0$ then both terms on the left-hand side are non-negative, and must therefore be zero. If $4ab-h^2=0$ you get the single line $2ax+hy=0$ (the two lines coincide to give a degenerate case). And if $4ab-h^2\lt 0$ you get two lines by factoring the left-hand side as the difference of two squares.

Note: @MarkBennet results gave me an idea of alternative approach.

Lets start off by assuming that $ax^{2}+2hxy+by^{2}=0$ represent a pair of straight lines (real/coincident/imaginary).

Substituting $\frac{y}{x}=m$ we have

$\Rightarrow bm^{2}+2hm+a=0$

$m=\frac{-h \pm \sqrt{h^{2}-ab}}{b}$

Plugging back $m=\frac{y}{x}$ we have two equations,

$$by=(-h + \sqrt{h^{2}-ab})x$$ and $$by=(-h - \sqrt{h^{2}-ab})x$$

These are the two lines represented by our equation $ax^{2}+2hxy+by^{2}=0$ , and as you can see we have an under root expression so we have two cases.

$(i)$ The lines are real and distinct if $h^{2}-ab>0$.

$(ii)$The lines are imaginary if $h^{2}-ab<0$.

This is a special case of what is known as quadratic forms.

If you rewrite $ax^2+2hxy+by^2=0$ using matrices as $$\left[\matrix{x \\ y}\right]^T \left[\matrix{a & h \\ h & b}\right] \left[\matrix{x \\ y}\right] = 0 $$ you can perform a change of variables to the inner matrix's eigenvectors to get something on the form $$\left[\matrix{x' \\ y'}\right]^T \left[\matrix{d_1 & 0 \\ 0 & d_2}\right] \left[\matrix{x' \\ y'}\right] = 0 $$ which expands to $d_1{x'}^2 + d_2{y'}^2 = 0$

From this it is easy to see if it represents a single point, two intersecting lines or a single line just from the signs of $d_1$ and $d_2$ (or if one is zero).

If you instead had something non-zero on the right, you could instead get an ellipse, a hyperbola, or two parallell lines.

Quadratic forms also work in higher dimensions, but this answer is getting long as it is.
If you want, you can probably read more in a standard linear algebra textbook.