Prove that the matrix $[\Gamma(\lambda_{i}+\mu_{j})]$ is nonsingular

By Andreieff's identity: $$ {\rm det}(A)=\frac{1}{n!}\int_{(0,\infty)^{n}} {\rm det}[e^{-\frac{t_k}{2}}t_k^{\lambda_i-\frac{1}{2}}]_{1\le i,k\le n}\ \times\ {\rm det}[e^{-\frac{t_k}{2}}t_k^{\mu_j-\frac{1}{2}}]_{1\le k,j\le n}\ \ dt_1\cdots dt_n $$ $$ =\frac{1}{n!}\int_{(0,\infty)^{n}} \left(\prod_{l=1}^{n} \frac{e^{-t_l}}{t_l}\right)\times {\rm det}[t_k^{\lambda_i}]_{1\le i,k\le n}\ \times\ {\rm det}[t_k^{\mu_j}]_{1\le k,j\le n}\ \ dt_1\cdots dt_n\ . $$ By the identity, e.g., in Reference for exponential Vandermonde determinant identity , the two determinants in the integrand have the same sign (write $t_l=e^{\alpha_l}$ to see the usual expression of the Harish-Chandra-Itzykson-Zuber integral). It is then easy to conclude that ${\rm det}(A)>0$ and so $A$ is nonsingular. The same argument and the use of Sylvester's criterion shows that $A$ is in fact positive definite. It is even totally positive.


Edit: I just found this other MO question: A difficult determinant which is directly related to this one. In the case where the Vandermonde-like determinants have integer exponents then one can of course bring Schür functions into play. One can then go a long way towards the computation of ${\rm det}(A)$ (see Marcel's answer to that question).


This is merely a reformulation of Abdelmalek Abdesselam's answer, in a somewhat different language and with different references. It should be a comment to that answer, but it is unfortunately too long. Long story short: see Karlin, Total positivity, formula (2.10) in Section 1.2, with $u(t) = t$ and $\sigma(dt) = \mathbb{1}_{(0, \infty)}(t) t^{-1} e^{-t} dt$.


The kernel $K(x,y)$ is said to be totally positive on $X \times Y$, where $X, Y \subseteq \mathbb{R}$, if $$ K\pmatrix{x_1&x_2&\cdots&x_n\\y_1&y_2&\cdots&y_n} := \det \left|\matrix{K(x_1,y_1)&K(x_1,y_2)&\cdots&K(x_1,y_n)\\K(x_2,y_1)&K(x_2,y_2)&\cdots&K(x_2,y_n)\\\vdots&\vdots&&\vdots\\K(x_n,y_1)&K(x_n,y_2)&\cdots&K(x_n,y_n)\\}\right| \geqslant 0 $$ whenever $x_1 < x_2 < \ldots < x_n$ and $y_1 < y_2 < \ldots < y_n$ (and, of course, $x_1, x_2, \ldots, x_n \in X$, $y_1, y_2, \ldots, y_n \in Y$). It is strictly totally positive if strict inequality holds. A standard reference for totally positive kernels is Karlin's book Total positivity (Stanford, 1968). Our goal is thus to prove that the kernel $G(\mu,\nu) = \Gamma(\mu + \nu)$ is strictly totally positive.


It is known that the kernel $e^{x y}$ is strictly totally positive on $\mathbb{R} \times \mathbb{R}$; see, for example, Example (i) in Section 2.1 of Karlin's book. Substituting $t = e^y$, we see that $K(x, t) = t^x$ and $\check{K}(t, x) = t^x$ are strictly totally positive on $\mathbb{R} \times (0, \infty)$ and $(0, \infty) \times \mathbb{R}$, respectively.

Define $\sigma(dt) = t^{-1} e^{-t} dt$ on $(0, \infty)$. Observe that $$ G(\mu,\nu) = \Gamma(\mu + \nu) = \int_0^\infty t^{\mu + \nu - 1} e^{-t} dt = \int_0^\infty K(\mu, t) \check{K}(t, \nu) \sigma(dt) . $$ The basic composition formula (as it is called by Karlin, see (2.5) in Section 1.2 in his book; Karlin's reference for this formula is problem 68 in Pólya and Szegő, Aufgaben und Lehrsdtze aus der Analysis, vol. 1) tells us that $$ \begin{aligned} & G\pmatrix{\mu_1&\mu_2&\cdots&\mu_n\\\nu_1&\nu_2&\cdots&\nu_n} = \idotsint\limits_{0<t_1 < t_2 < \ldots < t_n} K\pmatrix{\mu_1&\mu_2&\cdots&\mu_n\\t_1&t_2&\cdots&t_n} \times \\ & \hspace{10em} \times \check{K}\pmatrix{t_1&t_2&\cdots&t_n\\\nu_1&\nu_2&\cdots&\nu_n} \sigma(dt_1) \sigma(dt_2) \ldots \sigma(dt_n) . \end{aligned} $$ The right-hand side is clearly positive, and our claim is proved.


The above argument is (essentially) contained in Karlin's book, when he proves that moment sequences generate totally positive kernels, see formula (2.10) in Section 1.2 of his book. He is only concerned with integer moments, but the argument carries over with no modifications to arbitrary moments.