How to understand an example in Krattenthaler's “Advanced Determinant Calculus”?

In summary: you have the correct matrix $A$, and the correct idea for how to get it, and it is indeed equal in every term to the matrix $B$, but you are being misled by confusing notation in the author's description of $B$, and everything you wrote down for the factors whose element-wise product is supposed to be $B$ is wrong.

When you take your starting matrix $$ \begin{bmatrix} \frac{5!}{2!3!} & \frac{5!}{3!2!} & \frac{5!}{4!1!} \\ \frac{5!}{1!4!} & \frac{5!}{2!3!} & \frac{5!}{3!2!} \\ \frac{5!}{0!5!} & \frac{5!}{1!4!} & \frac{5!}{2!3!} \end{bmatrix} $$ and factor out $\frac{5!}{4!3!}$, $\frac{5!}{3!4!}$, and $\frac{5!}{2!5!}$ from the three rows respectively, we are indeed left with the matrix you gave, but it's more helpful to write it down as: $$ \begin{bmatrix} \frac{4!3!}{2!3!} & \frac{4!3!}{3!2!} & \frac{4!3!}{4!1!} \\ \frac{3!4!}{1!4!} & \frac{3!4!}{2!3!} & \frac{3!4!}{3!2!} \\ \frac{2!5!}{0!5!} & \frac{2!5!}{1!4!} & \frac{2!5!}{2!3!} \end{bmatrix} = \begin{bmatrix} \frac{4!}{2!}\cdot\frac{3!}{3!} & \frac{4!}{3!}\cdot\frac{3!}{2!} & \frac{4!}{4!} \cdot \frac{3!}{1!} \\ \frac{3!}{1!}\cdot\frac{4!}{4!} & \frac{3!}{2!}\cdot \frac{4!}{3!} & \frac{3!}{3!} \cdot \frac{4!}{2!} \\ \frac{2!}{0!}\cdot\frac{5!}{5!} & \frac{2!}{1!}\cdot\frac{5!}{4!} & \frac{2!}{2!} \cdot \frac{5!}{3!} \end{bmatrix} $$ In general, the $(i,j)$ entry was $\frac{(a+b)!}{(a-i+j)! (b-j+i)!}$ from which we factored out $\frac{(a+b)!}{(a-i+n)! (b-1+i)!}$ leaving $\frac{(a-i+n)! (b-1+i)!}{(a-i+j)! (b-j+i)!} = \frac{(a-i+n)!}{(a-i+j)!} \cdot \frac{(b-1+i)!}{(b-j+i)!}$.

We lose some information when we write each entry of this matrix down as a product of two (in general, $n-1$) numbers. In the first column, each entry is a product of two "left factors"; in the second column, it's a product of a "left factor" and a "right factor", and in the third column, it's a product of two "right factors". By "left factors", I mean factors from $\frac{(a-i+n)!}{(a-i+j)!}$ and by "right factors" I mean factors from $\frac{(b-1+i)!}{(b-j+i)!}$.

The author's description of the new matrix $B$ comes from writing $$\frac{(a-i+n)!}{(a-i+j)!} = (a-i+n)(a-i+n-1) \dotsb (a-i+j+1)$$ and $$\frac{(b-1+i)!}{(b-j+i)!} = (b-1+i)(b-2+i) \dotsb (b-(j-1)+i) \\= (b+i-(j-1))(b+i-(j-2))\dotsb (b+i-1).$$ This is misleading because these products vary in length depending on the column, and the $\dotsb$ style of writing them is confusing. It's especially silly to write out the products as "first factor, second factor, ..., last factor" because there's many cases when there's just zero, or one, or two factors in the product! That's what's throwing you off when you write down the matrices representing the individual factors: not all of those factors appear for every entry.

So, for example, let's look at entry $(1,2)$. It consists of

• The factor $4$, corresponding to the falling product $(a-i+n)(a-i+n-1)\dotsb(a-i+j+1)$ which is in fact the product starting and ending at $4$.
• The factor $3$, corresponding to the rising product $(b+i-j+1)(b+i-j+2)\dotsb (b+i-1)$ which is in fact the product starting and ending at $3$.

For entry $(1,1)$, which is also a product $4 \cdot 3$, the breakdown is

• The product $4 \cdot 3$, corresponding to the falling product $(a-i+n)(a-i+n-1)\dotsb(a-i+j+1)$ which multiplies the numbers from $4$ all the way down to $3$.
• The empty product, corresponding to the rising product $(b+i-j+1)(b+i-j+2)\dotsb (b+i-1)$, which multiplies the numbers from $4$ "all the way up to $3$".

It would be more accurate to write the $(i,j)$ entry of the new matrix as $$ \left(\prod_{k=j+1}^n (a-i+k)\right) \left(\prod_{\ell=1}^{j-1}(b+i-\ell) \right) $$ with the understanding that the first product is $1$ (an empty product) when $j=n$, and the second product is $1$ (an empty product) when $j=1$.

Here I'd like to look at this example with a slightly different focus, which might help to get a clearer picture of it.

Multilinearity: The idea in the first step, namely factoring out so that all denominators are cleared, is based upon the multilinearity property of determinants.

In his example Krattenthaler uses multilinearity row-wise \begin{align*} \det(A)&=\det_{1\leq i,j\leq n}\left(c_ia_{i,j}\right)\\ &=\det\begin{pmatrix} c_1a_{1,1}&c_1a_{1,2}&\cdots&c_1a_{1,n}\\ c_2a_{2,1}&c_2a_{2,2}&\cdots&c_2a_{2,n}\\ \vdots&\vdots&\ddots&\vdots\\ c_na_{n,1}&c_na_{n,2}&\cdots&c_na_{n,n}\\ \end{pmatrix}\\ &=c_1c_2\cdots c_n\cdot\det\begin{pmatrix} a_{1,1}&a_{1,2}&\cdots&a_{1,n}\\ a_{2,1}&a_{2,2}&\cdots&a_{2,n}\\ \vdots&\vdots&\ddots&\vdots\\ a_{n,1}&a_{n,2}&\cdots&a_{n,n}\\ \end{pmatrix}\\ &=\prod_{i=1}^nc_i\cdot\det_{1\leq i,j\leq n}\left(a_{i,j}\right)\tag{1} \end{align*} where we factor out $c_i$ from row $i, 1\leq i\leq n$.

In the following we have the representation in mind which Krattenthaler states as first identity:

\begin{align*} &\det_{1\leq i,j\leq n}\left(\binom{a+b}{a-i+j}\right)=\prod_{i=1}^n\frac{(a+b)!}{(a-i+n)!(b+i-1)!}\\ &\quad\cdot\det_{1\leq i,j\leq n}\left((a-i+n)(a-i+n-1)\cdots(a-i+j+1)\right.\\ &\quad\qquad\qquad\quad\left.\cdot(b+i-j+1)(b+i-j+2)\cdots(b+i-1)\right)\tag{2} \end{align*}

Special case: $n=3$:

To make things easier for analysis we look at the special case $n=3$. But we keep $a,b$ general to better see what's going on. This way we can also switch back to general $n$ more easily. We consider \begin{align*} \det\left(A_3\right)&=\det_{1\leq i,j\leq 3}\left(\binom{a+b}{a-i+j}\right)\\ &=\det\begin{pmatrix} \binom{a+b}{a}&\ \binom{a+b}{a+1}&\ \binom{a+b}{a+2}\\ \\ \binom{a+b}{a-1}&\ \binom{a+b}{a}&\ \binom{a+b}{a+1}\\ \\ \binom{a+b}{a-2}&\ \binom{a+b}{a-1}&\ \binom{a+b}{a} \end{pmatrix} \end{align*}

At first we factor out $(a+b)!$ from each row. Using the notation from (1) we have \begin{align*} c_1=c_2=c_3=(a+b)!\qquad\qquad c_1c_2c_3=\prod_{i=1}^3(a+b)! \end{align*}

We obtain \begin{align*} \det\left(A_3\right)&=\prod_{i=1}^3(a+b)! \cdot\det\begin{pmatrix} \frac{1}{a!b!}&\ \frac{1}{(a+1)!(b-1)!}&\ \frac{1}{(a+2)!(b-2)!}\\ \\ \frac{1}{(a-1)!(b+1)!}&\ \frac{1}{a!b!}&\ \frac{1}{(a+1)!(b-1)!}\\ \\ \frac{1}{(a-2)!(b+2)!}&\ \frac{1}{(a-1)!(b+1)!}&\ \frac{1}{a!b!} \end{pmatrix} \end{align*}

Next we look at the factors with $a$ in the denominator. We identify the largest terms in a row-wise manner and take \begin{align*} c_1=\frac{1}{(a+2)!},c_2=\frac{1}{(a+1)!},c_3=\frac{1}{a!}\qquad\qquad c_1c_2c_3=\prod_{i=1}^3\frac{1}{(a-i+3)!} \end{align*}

We obtain \begin{align*} \det\left(A_3\right)&=\prod_{i=1}^3\frac{(a+b)!}{(a-i+3)!} \cdot\det\begin{pmatrix} \frac{(a+2)(a+1)}{b!}&\ \frac{a+2}{(b-1)!}&\ \frac{1}{(b-2)!}\\ \\ \frac{(a+1)a}{(b+1)!}&\ \frac{a+1}{b!}&\ \frac{1}{(a+1)!(b-1)!}\\ \\ \frac{a(a-1)}{(b+2)!}&\ \frac{a}{(b+1)!}&\ \frac{1}{a!b!} \end{pmatrix} \end{align*}

Finally we look at the factors with $b$ in the denominator. We identify the largest terms in a row-wise manner and take \begin{align*} c_1=\frac{1}{b!},c_2=\frac{1}{(b+1)!},c_3=\frac{1}{(b+2)!}\qquad\qquad c_1c_2c_3=\prod_{i=1}^3\frac{1}{(b+i-1)!} \end{align*}

We obtain \begin{align*} \det\left(A_3\right)&=\prod_{i=1}^3\frac{(a+b)!}{(a-i+3)!(b+i-1)!}\\ &\qquad \cdot\det\begin{pmatrix} (a+2)(a+1)&\ (a+2)b&\ (b-1)b\\ \\ (a+1)a&\ (a+1)(b+1)&\ b(b+1)\\ \\ a(a-1)&\ a(b+2)&\ (b+1)(b+2) \end{pmatrix}\tag{3} \end{align*}

Now the somewhat more technical part:

Determinant entries:

We compare the special case $n=3$ in (3) with the general case (2). It's easy to see how the product representations correlate. We can simply replace $3$ by $n$ to switch from the special case to the general case.

And what about the determinant entries? Here we turn the table and ask Krattenthaler by setting $n=3$ in his representation (2). We obtain \begin{align*} &\det_{1\leq i,j\leq 3}\left(\binom{a+b}{a-i+j}\right)=\prod_{i=1}^3\frac{(a+b)!}{(a-i+3)!(b+i-1)!}\\ &\quad\cdot\det_{1\leq i,j\leq 3}\left((a-i+3)(a-i+2)\cdots(a-i+j+1)\right.\\ &\quad\qquad\qquad\quad\left.\cdot(b+i-j+1)(b+i-j+2)\cdots(b+i-1)\right)\tag{4} \end{align*}

Looking at the product with the $a$-terms in the determinant representation (4) we see falling factors which give \begin{align*} &(a-i+3)(a-i+2)\cdots(a-i+j+1)\\ &\qquad=(a-i+3)(a-i+(3-1))\cdots(a-i+3-(2-j))\\ &\qquad=\prod_{k=0}^{2-j}(a-i+3-k)\qquad\qquad\qquad 1\leq j\leq 3 \end{align*}

Note that the upper limit $2-j$ of the product is $<0$ in case $j=3$. Here we have to keep in mind the convention that empty products are set to $1$, similarly as empty sums are set to $0$.

We do the same considerations for the product with the $b$-terms and get \begin{align*} &(b+i-j+1)(b+i-j+2)\cdots(b+i-1)\\ &\qquad=(b+i-j+1)(b+i-j+(1+1)\cdots(b+i-j+(j-2))\\ &\qquad=\prod_{l=0}^{j-2}(b+i-j+l)\qquad\qquad\qquad 1\leq j\leq 3 \end{align*} where we have again an empty product in case $j=1$.

Having done this analysis we can now roll back from (4) to get the representation (3): \begin{align*} &\det_{1\leq i,j\leq 3}\left(\binom{a+b}{a-i+j}\right)=\prod_{i=1}^3\frac{(a+b)!}{(a-i+3)!(b+i-1)!}\\ &\quad\qquad\cdot\det_{1\leq i,j\leq 3}\left((a-i+3)(a-i+2)\cdots(a-i+j+1)\right.\\ &\quad\qquad\qquad\qquad\quad\left.\cdot(b+i-j+1)(b+i-j+2)\cdots(b+i-1)\right)\\ &\quad=\prod_{i=1}^3\frac{(a+b)!}{(a-i+3)!(b+i-1)!}\\ &\quad\qquad\cdot\det_{1\leq i,j\leq 3}\left(\prod_{k=0}^{2-j}(a-i+3-k)\prod_{l=0}^{j-2}(b+i-j+l)\right)\\ &\quad=\prod_{i=1}^3\frac{(a+b)!}{(a-i+3)!(b+i-1)!}\\ &\quad\cdot\det \begin{pmatrix} \prod_{k=0}^1(a+2-k)&\prod_{k=0}^0(a+2-k)\prod_{l=1}^1(b-1+l)&\prod_{l=1}^2(b-2+l)\\ \prod_{k=0}^1(a+1-k)&\prod_{k=0}^0(a+1-k)\prod_{l=1}^1(b+l)\qquad&\prod_{l=1}^2(b-1+l)\\ \prod_{k=0}^1(a-k)\qquad&\prod_{k=0}^0(a-k)\prod_{l=1}^1(b+1+l)\qquad&\prod_{l=1}^2(b+l)\qquad\\ \end{pmatrix}\\ &\quad=\prod_{i=1}^3\frac{(a+b)!}{(a-i+3)!(b+i-1)!}\\ &\quad\qquad\cdot\det \begin{pmatrix} (a+2)(a+1)&\ (a+2)b&\ (b-1)b\\ \\ (a+1)a&\ (a+1)(b+1)&\ b(b+1)\\ \\ a(a-1)&\ a(b+2)&\ (b+1)(b+2) \end{pmatrix} \end{align*}

Based upon this analysis the derivation of the general case should now be feasible. We might conclude that Krattenthalers example is nice and interesting. But also the representation is albeit somewhat technical, quite appropriate.