How to determine the probabilities for a cuboid die?
I think a reasonable first approximation can be made like this: choose an arbitrary orientation for the die, and figure out, if the die were released in that orientation with its lowermost point resting on a surface, which side would it fall on? That can be easily calculated; you just draw a line going straight down from the center of mass, and whichever face it passes through, that's the one that will land down. We can then assume (this is the unjustified approximation) that the orientation of the die before it begins its final tip is uniformly random, so the problem is reduced to finding the proportion of all possible directions which pass through each face. Hopefully it's clear that that is just the solid angle subtended by the face divided by the total solid angle, $4\pi$.
For a cuboid, we can figure out the solid angle of a face by doing an integral,
$$\Omega = \iint_S\frac{\hat r\cdot\mathrm{d}A}{r^2}$$
as given on MathWorld. Take the face to lie in the plane $z = Z$ (capital letters will be constants) and to have dimensions $2X\times 2Y$. The integral then becomes
$$\begin{align}\Omega &= \int_{-Y}^{Y}\int_{-X}^{X} \frac{(x\hat{x} + y\hat{y} + Z\hat{z})\cdot\hat{z}}{(x^2 + y^2 + Z^2)^{3/2}}\mathrm{d}x\mathrm{d}y \\ &= Z\int_{-Y}^{Y}\int_{-X}^{X} \frac{1}{(x^2 + y^2 + Z^2)^{3/2}}\mathrm{d}x\mathrm{d}y\\ &= 4\tan^{-1}\biggl(\frac{XY}{Z\sqrt{X^2 + Y^2 + Z^2}}\biggr) \end{align}$$
This gets divided by $4\pi$ to produce the probability. Translating into your notation, and multiplying by 2 to take into account the two opposite sides under one probability, this becomes
$$P_{ab} = \frac{2}{\pi}\tan^{-1}\biggl(\frac{ab}{c\sqrt{a^2 + b^2 + c^2}}\biggr)$$
$P_{bc}$ and $P_{ca}$ are the same under the appropriate permutation of the labels.
Some sanity checks show that this is at least a reasonable candidate solution:
- $P_{ab}$ is symmetric in $a$ and $b$
- $P_{ab}$ is directly related to $a$ and $b$ and inversely related to $c$
- $P_{ab} \to 1$ as $a,b\to\infty$ or $c\to 0$
- $P_{ab} \to 0$ as $a,b\to 0$ or $c\to\infty$
For three equal sides,
$$P_{aa} = \frac{2}{\pi}\tan^{-1}\biggl(\frac{a}{\sqrt{3a^2}}\biggr) = \frac{2}{\pi}\tan^{-1}\frac{1}{\sqrt{3}} = \frac{1}{3}$$
For two equal sides, $b = c$, as in your simplified example:
$$\begin{align}P_{ab} &= \frac{2}{\pi}\tan^{-1}\biggl(\frac{ab}{b\sqrt{a^2 + 2b^2}}\biggr) = \frac{2}{\pi}\tan^{-1}\biggl(\frac{\rho}{\sqrt{\rho^2 + 2}}\biggr) \\ P_{bb} &= \frac{2}{\pi}\tan^{-1}\biggl(\frac{b^2}{a\sqrt{a^2 + 2b^2}}\biggr) = \frac{2}{\pi}\tan^{-1}\biggl(\frac{1}{\rho\sqrt{\rho^2 + 2}}\biggr)\end{align}$$
- $P_{ab}(0) = 0$ as expected
- $P_{ab}(1) = \frac{2}{\pi}\tan^{-1}\frac{1}{\sqrt{3}} = \frac{2}{\pi}\frac{\pi}{6} = \frac{1}{3}$ for one particular pair of opposite sides, as expected
- $P_{ab}(\infty) = \frac{2}{\pi}\tan^{-1}0 = \frac{2}{\pi}\frac{\pi}{2} = 1$, as expected
$P_{ab} + P_{bc} + P_{ca} = 1$ can be shown using the identity
$$\tan^{-1} x + \tan^{-1} y = \tan^{-1}\frac{x + y}{1 - xy}$$
from which follows
$$\begin{align}\tan^{-1} x + \tan^{-1} y + \tan^{-1} z &= \tan^{-1}\frac{(x + y)/(1 - xy) + z}{1 - z(x + y)/(1 - xy)} \\ &= \tan^{-1}\frac{(x + y + z - xyz)/(1 - xy)}{(1 - xy - zx - zy)/(1 - xy)} \\ &= \tan^{-1}\frac{x + y + z - xyz}{1 - xy - zx - zy} \end{align}$$
The relevant products are
$$\begin{align} x + y + z &= \frac{ab}{c\sqrt{a^2 + b^2 + c^2}} + \frac{bc}{a\sqrt{a^2 + b^2 + c^2}} + \frac{ca}{b\sqrt{a^2 + b^2 + c^2}} \\ &= \frac{(ab)^2 + (bc)^2 + (ca)^2}{abc\sqrt{a^2 + b^2 + c^2}} \\ xy &= \frac{ab}{c\sqrt{a^2 + b^2 + c^2}}\frac{bc}{a\sqrt{a^2 + b^2 + c^2}} \\ &= \frac{b^2}{a^2 + b^2 + c^2} \text{and cyclic permutations, so}\\ xy + yz + zx &= 1 \\ xyz &= \frac{abc}{(a^2 + b^2 + c^2)^{3/2}} \\ x + y + z - xyz &= \frac{[(ab)^2 + (bc)^2 + (ca)^2](a^2 + b^2 + c^2) - (abc)^2}{abc(a^2 + b^2 + c^2)^{3/2}} \neq 0 \end{align}$$
so you wind up with $\tan^{-1}\frac{x + y + z - xyz}{0} = \frac{\pi}{2}$, giving $\sum P = 1$.