Generalized Cauchy-Binet sum over a fixed subset of indices

This is probably a bit late, but the following formula is proved in this paper (Eq. S4 in the Supplementary Material):

\begin{equation} \sum_{S \in \binom{[n - j]}{m - j}} \det(A_{[m], S \cup T}) \det(B_{S\cup T, [m]}) = (-1)^{j} \det \left(\begin{array}{c c} \mathbf{0}_{j \times j} & B_{T, [m]} \\ A_{[m], T} & A_{[m], [n - j]}B_{[n - j], [m]} \end{array} \right) \rm{,} \end{equation}

where $0_{j \times j}$ is the $j \times j$ matrix of all zeroes. I'll leave the detailed proof to the paper, but the essential idea is to apply the Laplace expansion by complementary minors formula to the columns $T$ of $A$ and rows $T$ of $B$, rearrange sums to apply the ordinary Cauchy-Binet formula over $S$, and then recognize what remains as another expansion by complimentary minors of the above matrix on the r.h.s. The $j \times j$ block of zeros is to ensure that the minors you don't want to count end up vanishing, much along the same lines as Adam Przeździecki's answer. Notice that the special case for $j = 0$ is the ordinary Cauchy-Binet formula.

Let $I_j(x)$ be the diagonal $n\times n$ matrix with all 1's on the diagonal, except the last $j$ elements, where a 1 is replaced by an $x$. I believe that the identity you are looking for is $$ [x^j] \det(A I_j(x) B) = \sum_{S\in\binom{[n-j]}{m-j}} \det(A_{[m],S\cup T}) \det(B_{S\cup T,[m]}) , $$ where the notation $[x^j]$ denotes the coefficient of $x^j$ in whatever follows it, which is a polynomial in $x$. If you apply the standard Cauchy-Binet formula to $\det(A I_j(x) B)$, then the only way to get $j$ powers of $x$ is from the terms that involve the columns of $A$ or rows of $B$ that you want to keep fixed.

You can write $I_j(x) = Q_j + x P_j$, where $Q_j$ and $P_j$ are projections defined in the obvious way. So, essentially, you want the highest power of $x$ from the polynomial $\det (AQ_j B + x A P_j B)$.

You may modify one of your matrices without changing any of the relevant determinants. Lets choose $B$. Write $$ B = \begin{bmatrix} B_{[n-j],[m]} \\ \hline B_T \end{bmatrix} $$ If the rank of $B_T$ is less than $j$ then all the determinants $\det(B_{S\cup T,[m]})$ are $0$. Otherwise, assume for simplicity that the first $j$ columns of $B_T$ are linearly independent. You may apply row operations to $B_T$ so as to obtain $$ B' = \begin{bmatrix} \begin{matrix} B_{[n-j],[m]} \end{matrix} \\ \hline \begin{matrix} D & \vert &B_T' \end{matrix} \end{bmatrix} $$ where $D$ is some nonsingular diagonal matrix and the interesting determinants don't change. Then subtract from the rows of $B_{[n-j],[m]}$ suitable multiplicities of the last $j$ rows so as to obtain the matrix $$ B'' = \begin{bmatrix} 0 & \vert & * \\ \hline D & \vert & B_T' \end{bmatrix} $$ The interesting determinants are still the same. Now you apply the usual Cauchy-Binet formula to the product $AB''$ and see that all the determinants you didn't want to count are null.

Note that the algorithm described by Igor Khavkine, although slower, still has its value. Write $[n_1]=\{1,2,\ldots,n_1\}$, $[n_2]=\{n_1+1,n_1+2,\ldots,n\}$ and $m=m_1+m_2$. An obvious modification of his algorithm will count the following $$ \sum_{s_1\in\tbinom{[n_1]}{m_1}}\sum_{s_2\in\tbinom{[n_2]}{m_2}} \det(A_{[m],s_1\cup s_2})\det(B_{s_1\cup s_2,[m]}) $$