Identify the Conic Section

JavaScript (ES6), 316 323 347

p=>[1,2,4].some(x=>(d=D(Q=[[x&1,x&2,x&4,0,0,0],...p.map(([x,y])=>[x*x,x*y,y*y,x,y,1])]))?[a,b,c]=Q.map((v,i)=>D(Q.map((r,j)=>(r=[...r],r[i]=x*!j,r)))/d):0,D=m=>m[1]?m[0].reduce((r,v,i)=>r+(i&1?-v:v)*D(m.slice(1).map(r=>r.filter((a,j)=>j-i))),0):m)&&(d=b*b-4*a*c)?d<0?!b&c==a?'Circle':'Ellipse':'Hyperbola':'Parabola'

Any language better suited for handling matrix and determinant should score better (APL, J, CJAM, Jelly)

References: General form of a conic, Five points determine a conic, System of linear equations, Determinant

In the cartesian plane, the general equation of a conic is

A*x*x + B*x*y + C*y*y + D*x + E*y + F = 0 

having A or B or C not equal to 0 (otherwise it's a straight line)

A ... F are six unknowns to be found. With five pairs of (x,y) we can build a linear system with five equations, and scaling remove one dimension. That is, we can set one of A,B or C to 1 if it's not 0 (and we know that at least one is not 0).

I build and try to solve 3 systems: first trying A=1. If not solvable then B=1, then C. (There could be a better way, but that's my best at the time)

Having the values of A,B,C we can classify the conic looking at the discriminant d=B*B-4*A*C

• d == 0 -> parabola
• d > 0 -> hyperbola
• d < 0 -> ellipse, particularly (A == C and B == 0) -> circle

Less golfed

F=p=>(
  // Recursive function to find determinant of a square matrix
  D=m=>m[1]
    ?m[0].reduce((r,v,i)=>r+(i&1?-v:v)*D(m.slice(1).map(r=>r.filter((a,j)=>j-i))),0)
    :m,
  // Try 3 linear systems, coefficients in Q
  // Five equation made from the paramaters in p
  // And a first equation with coefficient like k,0,0,0,0,0,1 (example for A)  
  [1,2,4].some(
    x => (
      // matrix to calc the determinant, last coefficient is missing at this stage
      Q = [ 
        [x&1, x&2, x&4, 0,0,0] // first one is different
        // all other equations built from the params 
        ,...p.map( ([x,y]) => [x*x, x*y, y*y, x, y, 1] )
      ],
      d = D(Q), // here d is the determinant
      d && ( // if solvable  then d != 0
        // add missing last coefficient to Q
        // must be != 0 for the first row, must be 0 for the other
        Q.map( r=> (r.push(x), x=0) ),
        // solve the system (Cramer's rule), I get all values for A...F but I just care of a,b,c
        [a,b,c] = Q.map((v,i)=>D(Q.map(r=>(r=[...r],r[i]=r.pop(),r))) / d),
        d = b*b - 4*a*c, // now reuse d for discriminant
        d = d<0 ? !b&c==a ? 'Circle' : 'Ellipse' // now reuse d for end result
        : d ? 'Hyperbola' : 'Parabola'
      ) // exit .some if not 0
    ), d // .some exit with true, the result is in d
  )  
)

Test

F=p=>[1,2,4].some(x=>(d=D(Q=[[x&1,x&2,x&4,0,0,0],...p.map(([x,y])=>[x*x,x*y,y*y,x,y,1])]))?[a,b,c]=Q.map((v,i)=>D(Q.map((r,j)=>(r=[...r],r[i]=x*!j,r)))/d):0,D=m=>m[1]?m[0].reduce((r,v,i)=>r+(i&1?-v:v)*D(m.slice(1).map(r=>r.filter((a,j)=>j-i))),0):m)&&(d=b*b-4*a*c)?d<0?!b&c==a?'Circle':'Ellipse':'Hyperbola':'Parabola'

console.log=(...x)=>O.textContent+=x+'\n'

;[
 [[0, 0], [1, 5], [2, 3], [4, 8], [9, 2]]
,[[1.2, 5.3],[4.1, 5.6], [9.1, 2.5], [0, 1], [4.2, 0]]
,[[5, 0], [4, 3], [3, 4], [0, 5], [0, -5]]
,[[1, 0], [0, 1], [2, 1], [3, 4], [4, 9]]
].forEach(t=>console.log(t.join`|`+' => '+F(t)))

<pre id=O></pre>