How does the square end?

Google Sheets, 52 51 47 bytes

=ArrayFormula(Join(",",Unique(Mod(Row(A:A)^2,A1

_{^{Saved 4 bytes thanks to Taylor Scott}}

Sheets will automatically add 4 closing parentheses to the end of the formula.

It doesn't return the results in ascending order but it does return the correct results.

Jelly, 5 bytes

R²%³Q

Try it online!

Explanation

R²%³Q   Main link, argument: n

R       Range from 1 to n
 ²      Square each
  %³    Mod each by n
    Q   Deduplicate

05AB1E, 5 bytes

Lns%ê

Try it online! or as a Test Suite

L     # Range 1 .. input
 n    # Square each
  s%  # Mod by input
    ê # Uniquify (also sorts as a bonus)