Hyperbolic geometry: 2/3 ideal triangle in Poincare Disk

Touching circles in general

The general question of how to construct the incircle of a hyperbolic triangle in the Poincaré disk model can be seen as a version of the problem of Appolonius. With two points ideal, you have two of these circles tangent to one another, which might make some of the standard construction techniques easier. I'll not go into details on this.

Angular bisector at an ideal vertex

In Euclidean geometry, the incircle center lies at the intersection of the angular bisectors. In hyperbolic geometry, this is still true, since the angular bisectors are the locus of all points at equal distance to both your lines. Interestingly you get a well-defined angle bisector even when bisecting the zero angle at the ideal points. And it's even easier to construct. Suppose you have an ideal point $A$, and through that ideal point two hyperbolic lines $b$ and $c$ which intersect the unit circle again in ideal points $B$ and $C$. Then the (Euclidean) tangents to the circle in $B$ and $C$ will intersect in a point $E$, and the (Euclidean) line from $A$ to $E$ will intersect the circle in a point $D$ which is the endpoint of the angle bisector.

If you used the Beltrami-Klein model with its straight lines then you are done at this point. If you stick to Poincaré disk then the construction of point $D$ is still correct, since both models match in their set of ideal points. But you need a hyperbolic line through $A$ and $D$, i.e. an Euclidean circle orthogonal to the unit circle. You find its center at the point $F$ where the tangents in $A$ and $D$ intersect.

Proof for bisector construction

The theory behind this construction is that $(B,C;A,D)$ form a harmonic throw: their cross ratio in the complex plane is $\frac{(B-A)(C-D)}{(B-D)(C-A)}=-1$. In a way this started as a guess, based on my knowledge that many geometric operations can be expressed in terms of harmonic throws.

But I can also sketch a proof: find a hyperbolic isometry (Möbius transformation in the Poincaré disk, projective transformation for Beltrami-Klein) which makes $AD$ a Euclidean diameter of the unit circle. Then $AB$ and $AC$ will lie symmetric to that. Due to this symmetry, points on $AD$ have to be at equal distance to both the other lines. Also due to this symmetry, the tangents in $B$ and $C$ have to intersect in a point $E$ which lies on line $AD$ (possibly at infinity if the tangents are parallel).

Since all of this was invariant under projective transformations of the real plane, this specific position is without loss of generality. This explanation works well for Beltrami-Klein, to focus on the Poincaré disk you'd observe that the construction results in a harmonic throw in the complex plane, and that property is invariant under Möbius transformation so again w.l.o.g.

Constructing the incircle

Constructing the two angular bisectors in the two ideal vertices of your triangle will give you the hyperbolic circle center $I$. Note that this is not the same as the Euclidean circle center, even in the Poincaré disk model where hyperbolic circles are Euclidean circles. We'll get to that.

To find the point where the circle touches a triangle edge, you need a geodesic through the center orthogonal to that triangle edge. One way of getting that would be by reflecting the incenter in one of the edges. Hyperbolic reflection is Euclidean circle inversion. So invert the incenter in one of the edges (e.g. $I_a$ for edge $a$ in the diagram below). Draw a hyperbolic line connecting these. One way of drawing that hyperbolic line is by taking into account the inverse of the incenter in the unit circle (labeled $I'$ below). It lies at the other point of intersection for the two circles you intersected to find the incenter. Now you have three points defining a circle (e.g. $I, I', I_a$), which is the hyperbolic line orthogonal to the edge you used for the reflection. It will intersect the edge in the contact point of the incircle (e.g. $T_a$).

Now do the same for all three edges, and you have the three contact points. The Euclidean circle through these three points ($T_a,T_b,T_c$) will be the incircle.

Compass and straightedge

I am aware that my construction uses several primitives (like tangent line, circle through three points or inversion in a circle) that translate to a considerable number of steps when executed using compass and straightedge. I like the conceptual clarity of this approach, but chances are there will be solutions that are simpler to execute. They might be harder to understand why they work, though.

Computing the radius

The hyperbolic radius of the incircle depends on the shape of the triangle. If two points are ideal, then the shape of the triangle is determined by a single real parameter. You might use the angle at the non-ideal vertex to describe it, or the area of the ideal triangle which is equal to the angle deficit and thus equal to $\pi$ minus that angle.

To make the computation simple, I'd go with the Beltrami-Klein model since computations involving circles tend to be annoying. I'd pick a particularly simple and very symmetric choice of coordinates. Assume the angle at $C$ is $\gamma$ so that the area is $\pi-\gamma$. Define $x:=\cos\frac\gamma2$ and $y:=\sin\frac\gamma2$. Then follow the naming from my second figure, and pick points

$$A=(x,y)\qquad B=(x,-y)\qquad C=(0,0)\qquad D=(-x,-y)$$

which gives you the right angle at $C$ in a position at the center where Euclidean and hyperbolic angle match. Then the tangents in $B$ and $D$ meet in $P=(0,-\frac 1y)$ (where the point $P$ here plays the role of $E$ in the first figure above). The line $AP$ intersects the unit circle in

$$F=\frac1{3y^2+1}\left(x-xy^2,-3y-y^3\right)$$

but you don't really need that point, since $AP$ itself already is the angle bisector in this model. Intersecting that bisector with the $x$ axis, and also intersecting the line $AB$ with the $x$ axis, you get

$$I=\left(\frac{x}{y^2+1},0\right)\qquad T_c=\left(x,0\right)$$

Feed that into the distance formula, using the ideal points $X=(1,0)$ and $Y=(-1,0)$ as ideal “endpoints” of the $x$ axis:

$$\operatorname{CR}(X,Y;I,T_a)=\frac{[X,I][Y,T_a]}{[X,T_a][Y,I]} =\frac {\begin{vmatrix}1&x\\1&y^2+1\end{vmatrix}\cdot \begin{vmatrix}-1&x\\1&1\end{vmatrix}} {\begin{vmatrix}1&x\\1&1\end{vmatrix}\cdot \begin{vmatrix}-1&x\\1&y^2+1\end{vmatrix}} =\frac{(1+x)(y^2-x+1)}{(1-x)(y^2+x+1)}\\ r(\gamma)=\tfrac12\ln\operatorname{CR}(X,Y;I,T_a)= \tfrac12\ln\frac {\left(1+\cos\frac\gamma2\right) \left(\sin^2\frac\gamma2-\cos\frac\gamma2+1\right)} {\left(1-\cos\frac\gamma2\right) \left(\sin^2\frac\gamma2+\cos\frac\gamma2+1\right)}=\\ =\tfrac12\ln\frac {\left(1+\cos\frac\gamma2\right) \left(2-\cos\frac\gamma2-\cos^2\frac\gamma2\right)} {\left(1-\cos\frac\gamma2\right) \left(2+\cos\frac\gamma2-\cos^2\frac\gamma2\right)} =\tfrac12\ln\frac {\left(1+\cos\frac\gamma2\right) \left(1-\cos\frac\gamma2\right) \left(2+\cos\frac\gamma2\right)} {\left(1-\cos\frac\gamma2\right) \left(1+\cos\frac\gamma2\right) \left(2-\cos\frac\gamma2\right)}=\\ =\tfrac12\ln\frac{2+\cos\frac\gamma2}{2-\cos\frac\gamma2} $$

Probably due to my background in projective geometry, I'm strongly inclined to write the elements of the cross ratio as $2\times2$ determinants of homogeneous coordinate vectors instead of finite distances. Doing so allows me to avoid the fraction in the $x$ coordinate of $I$. I have to thank the comment by Blue for expressing the $\sin$ using $\cos$ and the subsequent simplification.

One concrete value

When I first read your question, I assumed that by “⅔ ideal triangle” you meant a triangle with ⅔ the area of an ideal triangle. In that case you'd have an interior angle of $\frac\pi3$. Plugging that into the formula above (the one with the $\sin$ because I got the other one only after the comment from Blue) you get

$$x=\cos\frac\pi6=\frac{\sqrt3}2\qquad y=\sin\frac\pi6=\frac12\\ r\left(\tfrac\pi3\right)= \tfrac12\ln\frac{(1+x)(y^2-x+1)}{(1-x)(y^2+x+1)}= \tfrac12\ln\frac{19+8\sqrt3}{13}\approx 0.4636$$

When I wrote my first version of this answer, I had constructed that situation, and then used the guess function in Cinderella to get the radius. That function attempts to turn numeric results into likely symbolic expressions (up to quadratic expressions if I remember correctly). It reported guess(exp(2*dist0)) = 19/13+8/13*sqrt(3) already providing the solution above but with some remaining uncertainty due to potential numeric errors.

Tangent half-angle formulation

Many trigonometric problems become simpler if parametrized using a rational parametrization of the unit circle, based on the tangens of half the angle. In the above case you'd get

$$t:=\tan\frac\gamma4\qquad x=\frac{1-t^2}{1+t^2}\qquad y=\frac{2t}{1+t^2}\\ r(\gamma)=\tfrac12\ln\frac{t^2+3}{3t^2+1}$$

At the time when I wrote this, this formula was a clear win over the one with $\sin$ and $\cos$, but it doesn't have much to offer over the pure $\cos$ formula which is both simpler, lower degrees and more symmetric. Well, still worth checking.