Point within the interior of a given angle

Denote the mid-point of $OM$ be $N$. Then $N$ is the center of the circle passing through the points $O,P,M,K$.

We have:

$$\angle PNK = 2\alpha,\quad \angle PMK = \pi-\alpha$$

By Cosine Theorem,

$$PK^2 = a^2+b^2-2ab\cos \angle PMK = PN^2 + KN^2 - 2PN\cdot KN \cos 2\alpha$$

Writing $PN = KN = \frac12 OM = r$, we have:

$$a^2 + b^2 +2ab\cos \alpha = 2r^2(1-\cos 2\alpha)$$

or

$$r = \sqrt\frac {a^2+b^2+2ab\cos \alpha}{2(1-\cos2\alpha)}$$

To find $OM$ we just double this value:

$$OM = 2r = 2\sqrt\frac {a^2+b^2+2ab\cos \alpha}{2(1-\cos2\alpha)} = 2\sqrt\frac {a^2+b^2+2ab\cos \alpha}{2(2 \sin^2 \alpha)} = \frac {\sqrt{a^2+b^2+2ab\cos \alpha}}{\sin \alpha}$$

Reflect $M$ across $OA$ and $OB$ and you get $A'$ resp. $B'$.

Let $t=AB$, then $A'B' = 2t$ and $OA'= OB' = OM = m$. Also $\angle A'OB' = 2\alpha$.

By the Law of cosine in triangle $ABM$ we have $$ t = \sqrt{a^2+b^2+2ab\cos \alpha}$$

and finally in isosceles triangle $A'B'O$ we have $$\boxed{m = {t\over \sin \alpha}}$$

Let $m=OM$, then $${a\over m} = \sin \alpha _1$$ and $${b\over m} = \sin \alpha _2$$

so $$\cos \alpha _1 = \sqrt{1-{a^2\over m^2}}\;\;\;\;\;\;\;\;\;\; \wedge \;\;\;\;\; \;\;\;\;\; \cos \alpha _2 = \sqrt{1-{b^2\over m^2}}$$

so $$\sin \alpha = {a\over m}\sqrt{1-{b^2\over m^2}} + {b\over m} \sqrt{1-{a^2\over m^2}}$$

Now you have to find out $m$. :)