When you inscribe a triangle in a circle in a triangle in a circle in a triangle ...

Without loss of generality assume that the circle has radius $1$, and that the triple $(a,b,c)$ is positively oriented (i.e., they go counterclockwise around the circle). As the problem is circularly symmetric, it suffices to show that we can choose $a,b,c$ so that the limit point has any specified distance from its center.

For any $\epsilon > 0$, we can choose $a, b, c$ such that the triangle $abc$ does not intersect the circle of radius $1-\epsilon$. Since the limit point must lie inside the triangle $abc$, it follows that the limit point can come arbitrarily close to the boundary.

On the other hand, if the triangle $abc$ is equilateral, the limit point is clearly the center of the circle (as it is fixed under a rotation by $\frac{2 \pi}{3}$ about the center).

So, if we can show that the map from $(a,b,c)$ to the limit point is continuous, it will follow that the limit point can have any distance less than $1$ from the center. The set of all possible distances will be a continuous image in $\Bbb{R}$ of the connected set of all possible oriented triples $(a,b,c)$, so it must be an interval; we have showed that this interval both contains $0$ and comes arbitrarily close to $1$.


This answer has very much the same flavor as Micah's answer, since it relies on there being a continuous map from the points $(a,b,c)$ to the limit point, and on the Intermediate Value Theorem.

Diagram of the construction

Take any point $X$ (red-violet and blinking for some reason) in the interior of your circle and let $O$ (green) be the center of your circle. Draw the ray $\overrightarrow{OX}$, and let $a$ (violet) be the point where this ray intersects the circle. Draw the other two points $b$ and $c$ (both violet) such that $\angle aOb = \angle aOc$ and each are less than or equal to $2\pi/3$. By the symmetry of this construction, the limit point must lie on the segment $\overline{Oa}$.

Now the claim is that this function from $(0,2\pi/3] \to \overline{Oa}$ where $\angle aOb$ maps to the limit point is continuous. In order to rigorously show that it's continuous (without just trusting the picture) we'd have to do some calculations like in Thomas Andrews's answer. But if you trust that it is continuous, since the limit point is the center of the original circle when $\angle aOb = 2\pi/3$, and since the limit point approaches $a$ as $\angle aOb \to 0$, by the Intermediate Value Theorem the limit point must be $X$ at some point in between. The point $X$ was chosen arbitrarily in the interior of the circle, so we're good: every point in the interior of the circle is the limit point of this construction for some choice of $a$, $b$, and $c$.


Not an answer, but too long for a comment.

We can get an explicit formula for this operation if we use vector operations.

Letting $A=|\vec b-\vec c|,B=|\vec a-\vec c|, C=|\vec a-\vec b|$, define $$\alpha=\frac{1}{2}(B+C-A),\\\beta=\frac{1}{2}(A+C-B),\\ \gamma=\frac{1}{2}(A+B-C).$$

Then the new points $\vec {a'},\vec {b'},\vec {c'}$ are $$\vec {a'}=\frac{\gamma \vec b+\beta \vec c}{\gamma+\beta}\\\vec {b'}=\frac{\gamma \vec a+\alpha \vec c}{\gamma+\alpha}\\\vec {c'}=\frac{\beta \vec a+\alpha \vec b}{\beta+\alpha}$$

Worth noting that $\gamma+\beta=A, \gamma+\alpha=B,\gamma+\beta=C.$

We can thus write this as:

$$\vec {a'}=\frac{1}{2}\left(\left(1+\frac{B-C}{A}\right)\vec b+\left(1+\frac{C-B}{A}\right)\vec c\right)$$

or $$\vec {a'}=\frac{\vec b+\vec c}{2}+\frac{|\vec a-\vec c|-|\vec a-\vec b|}{2}\frac{\vec b-\vec c}{|\vec b-\vec c|}$$

So, if $$f(\vec u,\vec v,\vec w)=\frac{\vec v+\vec w}{2}+\frac{|\vec u-\vec w|-|\vec u-\vec v|}{2}\frac{\vec v-\vec w}{|\vec v-\vec w|}$$

We can define $f$ so that it is continuous by defining $f(\vec u,\vec v,\vec v)=\vec v.$

Then our transformation is:

$$M:(\vec a,\vec b,\vec c)\mapsto (f(\vec a,\vec b,\vec c),f(\vec b,\vec c,\vec a),f(\vec c.\vec a,\vec b))$$