Of all polygons inscribed in a given circle which one has the maximum sum of squares of side lengths?

Yes, the maximal sum is the one of the equilateral triangle, that is $9R^2$.

Since Prove that in any triangle $ABC$, $\cos^2A+\cos^2B+\cos^2C\geq\frac{3}{4}$ then $$ \sin^2 A+\sin^2 B+\sin^2 C=3-\cos^2 A-\cos^2 B-\cos^2 C\leq \frac{9}{4} $$ where $A$, $B$ and $C$ are non negative numbers such that $A+B+C=\pi$. Hence, for any inscribed triangle, the sum of the squares of the sides is $$(2R\sin A)^2+(2R\sin B)^2+(2R\sin C)^2\leq 9R^2.$$

This problem can be stated as

$$ \max_{n}\sum_{k=1}^n\left(2r\sin\left(\frac{\theta_k}{2}\right)\right)^2 $$

s. t.

$$ \sum_{k=1}^n\theta_k = 2\pi $$

but

$$ \sum_{k=1}^n\left(2r\sin\left(\frac{\theta_k}{2}\right)\right)^2\ge n\left(2^{2n}r^{2n}\prod_{k=1}^n\sin^2\left(\frac{\theta_k}{2}\right)\right)^{\frac 1n} $$

assuming $\theta_1=\cdots=\theta_n$ we have

$$ \sum_{k=1}^n\left(2r\sin\left(\frac{\theta_k}{2}\right)\right)^2\ge n\left(2^{2n}r^{2n}\sin^{2n}\left(\frac{\pi}{n}\right)\right)^{\frac 1n} = n2^2r^2\sin^{2}\left(\frac{\pi}{n}\right) $$

Now calling

$$ f(n) = n\sin^{2}\left(\frac{\pi}{n}\right) $$

we have clearly a maximum about $n = 3$ as can be depicted in the attached plot

Let $\theta_k$ be the successive angles subtended by the sides, but the last one. The sum of squares is given by

$$4\sum_{k=1}^n\sin^2\frac{\theta_k}2+4\sin^2\left(\pi-\frac12\sum_{k=1}^n\theta_k\right)$$ which has the same extrema as $$\sum_{k=1}^n\cos\theta_k-\cos\left(\sum_{k=1}^n\theta_k\right).$$

Canceling the gradient,

$$\sin\theta_k=\sin\left(\sum_{k=1}^n\theta_k\right).$$

This shows that all angles $\theta_k$ must be equal, and then

$$n\cos\theta-\cos n\theta$$ is minimized with $n\theta=2\pi$.

Finally,

$$n\cos\frac{2\pi} n-\cos\pi$$ is the smallest with $n=3$.