Evaluating a certain integral which generalizes the ${_3F_2}$ hypergeometric function

In the special case $\mu=\gamma$ Prudnikov-Brychkov-Marychev (Vol. III, formula 2.21.1.20) gives an evaluation in terms of Appell's $F_3$ (of four Appel's functions, this is the one with the maximal number of parameters): $$\mathcal I\left(\alpha,\beta,\gamma,z;\gamma,\nu,\rho,w\right)= \frac{B\left(\gamma,\nu-\gamma\right)}{(1-w)^{\rho}}{}F_3\left(\rho,\alpha,\nu-\gamma,\beta,\nu;\frac{w}{w-1};z\right)$$ Since all arguments are ''free'' (there is no relation between them), your generalization is yet one more step beyond Appell. A relation to generalized hypergeometric functions for generic $w,z$ would be extremely surprising.

This answer is meant to connect the ones given by @Harry Peter and @Start wearing purple, clarifying a few questions emerged in the comments.

The integral of interest can be evaluated in the way pointed out by @Harry Peter, without forgetting to set some conditions on the parameters. First of all, for $\left|z\right|<1$ we can use the power series representation of the Gauss hypergeometric fucntion ${}_2F_1$ $$\begin{align*}\mathcal{I}(\alpha,\beta,\gamma,z;\mu,\nu,\rho,w)&=\int_0^1\frac{t^{\mu-1}(1-t)^{\nu-\mu-1}}{(1-wt)^{\rho}}{}_2F_1(\alpha,\beta,\gamma,zt)\,\mathrm{d}t\\[6pt]&=\int_0^1\sum_{n=0}^{\infty}\frac{t^{\mu+n-1}(1-t)^{\nu-\mu-1}}{(1-wt)^{\rho}}\frac{(\alpha)_n(\beta)_n}{(\gamma)_n}\frac{z^n}{n!}\,\mathrm{d}t, \end{align*}$$ where $(d)_n$ is the (rising) Pochhammer symbol, defined by $$(d)_n=\begin{cases} 1 &\;n=0\\ d(d+1)\cdots(d+n-1) &\;n>0. \end{cases}$$ The integral can now be performed using Euler representation, which in our case holds for $\Re(\nu+n)>\Re(\mu+n)>0$ and $\left|\mathrm{arg}(1-w)\right|<\pi$, $$\begin{align*}\mathcal{I}(\alpha,\beta,\gamma,z;\mu,\nu,\rho,w)&=\sum_{n=0}^{\infty}\frac{(\alpha)_n(\beta)_n}{(\gamma)_n}\frac{z^n}{n!}B(n+\mu,\nu-\mu){}_2F_1(\rho,n+\mu;n+\nu;w)\\[6pt]&=\sum_{n,m=0}^{\infty}\frac{\Gamma(n+\mu)\Gamma(\nu-\mu)}{\Gamma(n+\nu)}\frac{(\alpha)_n(\beta)_n}{(\gamma)_n}\frac{(\rho)_m(n+\mu)_m}{(n+\nu)_m}\frac{z^n}{n!}\frac{w^m}{m!}, \end{align*}$$ valid for $\left|w\right|<1$. Considering that $$(d)_n=\frac{\Gamma(d+n)}{\Gamma(d)}\quad\text{for}\;\;d\neq 0,-1,-2,\dots$$ when $\mu,\nu\neq 0,-1,-2,\dots$ we can write $$\begin{align*}\mathcal{I}(\alpha,\beta,\gamma,z;\mu,\nu,\rho,w)&=\sum_{n,m=0}^{\infty}\frac{\Gamma(n+\mu+m)\Gamma(\nu-\mu)}{\Gamma(n+\nu+m)}\frac{(\alpha)_n(\beta)_n}{(\gamma)_n}(\rho)_m\frac{z^n}{n!}\frac{w^m}{m!}\\[6pt]&=\frac{\Gamma(\mu)\Gamma(\nu-\mu)}{\Gamma(\nu)}\sum_{n,m=0}^{\infty}\frac{(\mu)_{n+m}(\alpha)_n(\beta)_n(\rho)_m}{(\nu)_{n+m}(\gamma)_n}\frac{z^n}{n!}\frac{w^m}{m!}\\[6pt]&=B(\mu,\nu-\mu)\,\mathrm{F}^{1:2;1}_{1:1;0}\left(\left.\begin{matrix}\mu&:&\alpha,\beta&;&\rho&\\\nu&:&\gamma&;&-&\end{matrix}\right|z,w\right). \end{align*}$$ $\mathrm{F}^{p:q;k}_{l:m;n}$ denotes Kampé de Fériet's double hypergeometric function in the (modified) notation of Burchnall and Chaundy [see Srivastava and Panda - "An integral representation for the product of two Jacobi polynomials", Eq. (26)] $$\begin{align*}&\mathrm{F}^{p:q;k}_{l:m;n}\left(\left.\begin{matrix}(a_p)&:&(b_q)&;&(c_k)&\\(\alpha_l)&:&(\beta_m)&;&(\gamma_n)&\end{matrix}\right|x,y\right)\\[6pt]&\quad=\sum_{r,s=0}^{\infty}\frac{\prod_{j=1}^p(a_j)_{r+s}\prod_{j=1}^q(b_j)_r\prod_{j=1}^k(c_j)_s}{\prod_{j=1}^l(\alpha_j)_{r+s}\prod_{j=1}^m(\beta_j)_r\prod_{j=1}^n(\gamma_j)_s}\frac{x^r}{r!}\frac{y^s}{s!}, \end{align*}$$ where $(d_h)$ denotes the sequence of $h$ parameters $d_1,\dots,d_h$. In general, convergence of this double series is assured if one of the following conditions holds

i) $p+q<l+m+1$, $p+k<l+n+1$ and $\left|x\right|<\infty$, $\left|y\right|<\infty$

ii) $p+q=l+m+1$, $p+k=l+n+1$ and

$ \begin{align*}\;\;\begin{cases}\left|x\right|^{\frac{1}{p-l}}+\left|y\right|^{\frac{1}{p-l}}<1 &\text{if}\;\;p>l\\[5pt]\max\left\{\left|x\right|,\left|y\right|\right\}<1 &\text{if}\;\;p\le l.\end{cases}\end{align*} $

In our case we have $p=1$, $q=2$, $k=1$, $l=1$, $m=1$ and $n=0$, so we are in (ii) and the double series converges only if $\max\left\{\left|z\right|,\left|w\right|\right\}<1$. Collecting all the constraints introduced we finally have $$\mathcal{I}(\alpha,\beta,\gamma,z;\mu,\nu,\rho,w)=B(\mu,\nu-\mu)\,\mathrm{F}^{1:2;1}_{1:1;0}\left(\left.\begin{matrix}\mu&:&\alpha,\beta&;&\rho&\\\nu&:&\gamma&;&-&\end{matrix}\right|z,w\right),$$ if $\max\left\{\left|z\right|,\left|w\right|\right\}<1$, $\left|\text{arg}(1-w)\right|<\pi$ and $\Re(\nu)>\Re(\mu)>0$.

In the case $\mu=\gamma$ the general result reduces to the one given by @Start wearing purple $$\begin{align*}\mathcal{I}(\alpha,\beta,\gamma,z;\gamma,\nu,\rho,&w)=B(\gamma,\nu-\gamma)\,\mathrm{F}^{1:2;1}_{1:1;0}\left(\left.\begin{matrix}\gamma&:&\alpha,\beta&;&\rho&\\\nu&:&\gamma&;&-&\end{matrix}\right|z,w\right)\\[6pt]&=\int_0^1\frac{t^{\gamma-1}(1-t)^{\nu-\gamma-1}}{(1-wt)^{\rho}}{}_2F_1(\alpha,\beta,\gamma;zt)\,\mathrm{d}t\\[6pt]&=\frac{1}{(1-w)^{\rho}}\int_0^1t^{\gamma-1}(1-t)^{\nu-\gamma-1}\left(1-\frac{w}{w-1}+\frac{wt}{w-1}\right)^{-\rho}{}_2F_1(\alpha,\beta,\gamma;zt)\,\mathrm{d}t, \end{align*}$$ where we have multiplied and divided by $(1-w)^{\rho}$. According to the single integral representation of Appell series $F_3$, the last expression is $$\mathcal{I}(\alpha,\beta,\gamma,z;\gamma,\nu,\rho,w)=\frac{B(\gamma,\nu-\gamma)}{(1-w)^{\rho}}F_3\left(\rho,\alpha,\nu-\gamma,\beta;\nu;\frac{w}{w-1};z\right).$$

$\int_0^1\dfrac{t^{\mu-1}(1-t)^{\nu-\mu-1}}{(1-wt)^\rho}{_2F_1}(\alpha,\beta;\gamma;zt)~dt$

$=\int_0^1\sum\limits_{n=0}^\infty\dfrac{(\alpha)_n(\beta)_nz^nt^{n+\mu-1}(1-t)^{\nu-\mu-1}}{(\gamma)_nn!(1-wt)^\rho}dt$

$=\sum\limits_{n=0}^\infty\dfrac{(\alpha)_n(\beta)_nz^nB(n+\mu,\nu-\mu){_2F_1}(\rho,n+\mu;n+\nu;w)}{(\gamma)_nn!}$

$=\sum\limits_{n=0}^\infty\sum\limits_{k=0}^\infty\dfrac{\Gamma(n+\mu)\Gamma(\nu-\mu)(\alpha)_n(\beta)_n(\rho)_k(n+\mu)_kz^nw^k}{\Gamma(n+\nu)(\gamma)_n(n+\nu)_kn!k!}$

$=\sum\limits_{n=0}^\infty\sum\limits_{k=0}^\infty\dfrac{\Gamma(n+k+\mu)\Gamma(\nu-\mu)(\alpha)_n(\beta)_n(\rho)_kz^nw^k}{\Gamma(n+k+\nu)(\gamma)_nn!k!}$

$=\sum\limits_{n=0}^\infty\sum\limits_{k=0}^\infty\dfrac{\Gamma(\mu)\Gamma(\nu-\mu)(\mu)_{n+k}(\alpha)_n(\beta)_n(\rho)_kz^nw^k}{\Gamma(\nu)(\nu)_{n+k}(\gamma)_nn!k!}$

$=B(\mu,\nu-\mu)\mathrm{F}^{1:2;1}_{1:1;0}\Bigg(\begin{matrix}\mu&:&\alpha,\beta&;&\rho&\\\nu&:&\gamma&;&-&\end{matrix}\Bigg|z,w\Bigg)$