A contour integral with 3 branch points

All that's happening is that you are avoiding the branch points with your contour so you can apply Cauchy's theorem. You have branch points where the argument any non-integral power is zero, in your case, at $z=0$ and $\pm i$.

Cauchy's theorem states that

$$\oint_C dz \: f(z)=0$$

So the original integral is going to equal the integral along the line joining $i$ to $-i$ plus any contribution from the indentations, if any. Is there any? Let's take the one at $z=i$ Let $z=i + \epsilon e^{i \phi}$ and the contribution is

$$\lim_{\epsilon \rightarrow 0}\: i \epsilon^{1+b} \int_0^{-\pi/2} d \phi \: e^{i (a+b) \phi}$$

This is zero because $-1>b$. (Whew!) Same for the other branch point at $z=-i$.

At $z=0$, we let $z= \epsilon e^{i \phi}$, and we will see that this contribution goes to zero because $a>b$.

So the integral you seek is simply the real part of the integral of $f(z)$ along the straight line from $i$ to $-i$.


$\newcommand{\+}{^{\dagger}} \newcommand{\angles}[1]{\left\langle\, #1 \,\right\rangle} \newcommand{\braces}[1]{\left\lbrace\, #1 \,\right\rbrace} \newcommand{\bracks}[1]{\left\lbrack\, #1 \,\right\rbrack} \newcommand{\ceil}[1]{\,\left\lceil\, #1 \,\right\rceil\,} \newcommand{\dd}{{\rm d}} \newcommand{\down}{\downarrow} \newcommand{\ds}[1]{\displaystyle{#1}} \newcommand{\expo}[1]{\,{\rm e}^{#1}\,} \newcommand{\fermi}{\,{\rm f}} \newcommand{\floor}[1]{\,\left\lfloor #1 \right\rfloor\,} \newcommand{\half}{{1 \over 2}} \newcommand{\ic}{{\rm i}} \newcommand{\iff}{\Longleftrightarrow} \newcommand{\imp}{\Longrightarrow} \newcommand{\isdiv}{\,\left.\right\vert\,} \newcommand{\ket}[1]{\left\vert #1\right\rangle} \newcommand{\ol}[1]{\overline{#1}} \newcommand{\pars}[1]{\left(\, #1 \,\right)} \newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}} \newcommand{\pp}{{\cal P}} \newcommand{\root}[2][]{\,\sqrt[#1]{\vphantom{\large A}\,#2\,}\,} \newcommand{\sech}{\,{\rm sech}} \newcommand{\sgn}{\,{\rm sgn}} \newcommand{\totald}[3][]{\frac{{\rm d}^{#1} #2}{{\rm d} #3^{#1}}} \newcommand{\ul}[1]{\underline{#1}} \newcommand{\verts}[1]{\left\vert\, #1 \,\right\vert} \newcommand{\wt}[1]{\widetilde{#1}}$ $\ds{\int_{-\pi/2}^{\pi/2}\cos\pars{ax}\cos^{b}\pars{x}\,\dd x:\ {\large ?}\,,\qquad \pars{~a >\ b\ >\ -1~}}$.

\begin{align}&\color{#c00000}{\int_{\pi/2}^{\pi/2}\cos\pars{ax}\cos^{b}\pars{x}% \,\dd x} =\Re\int_{\pi/2}^{\pi/2}\expo{\ic ax}\cos^{b}\pars{x}\,\dd x \\[3mm]&=\Re \int_{\verts{z}\ =\ 1 \atop{\vphantom{\Huge A}\verts{{\rm Arg}\pars{z}}\ <\ \pi/2}} z^{a}\pars{z^{2} + 1 \over 2z}^{b}\,{\dd z \over \ic z} \\[3mm]&={1 \over 2^{b}}\,\Im\color{#00f}{% \int_{\verts{z}\ =\ 1 \atop{\vphantom{\Huge A}\verts{{\rm Arg}\pars{z}}\ <\ \pi/2}} z^{a - b - 1}\pars{z^{2} + 1}^{b}\,\dd z}\tag{1} \end{align}

If we 'close' the arc with the line $\ds{\braces{\pars{0,y}\ \mid \ -1 \leq y \leq 1}}$ wich include indents 'around' $\ds{z = \pm\ic}$ and $\ds{z = 0}$, the integral vanishes out. That means that the integral in question is equal to a minus the integral along the above mentioned segment, from $\ds{y = 1}$ t0 $\ds{y = -1}$, with a suitable handling of the indented points. It's summarized in the following expression: \begin{align}&\color{#00f}{% \int_{\verts{z}\ =\ 1 \atop{\vphantom{\Huge A}\verts{{\rm Arg}\pars{z}}\ <\ \pi/2}} z^{a - b - 1}\pars{z^{2} + 1}^{b}\,\dd z} =\lim_{\epsilon \to 0^{+}}\left\lbrace% -\ \overbrace{\left.\int_{0}^{-\pi/2}z^{a - b - 1}\pars{z^{2} + 1}^{b}\,\dd z\, \right\vert_{z\ \equiv\ \ic + \epsilon\expo{\ic\theta}}}^{\ds{\large\tt I}}\right. \\[3mm]&-\ \overbrace{% \int_{1 - \epsilon}^{\epsilon}y^{a - b - 1}\expo{\ic\pars{a - b - 1}\pi/2} \pars{-y^{2} + 1}^{b}\ \ic\,\dd y}^{\ds{\large\tt II}}\ -\ \overbrace{\left.\int_{\pi}^{-\pi}z^{a - b - 1}\pars{z^{2} + 1}^{b}\,\dd z\, \right\vert_{z\ \equiv\ \epsilon\expo{\ic\theta}}}^{\ds{\large\tt III}} \\[3mm]&-\ \overbrace{\int_{-\epsilon}^{-1 + \epsilon}\pars{-y}^{a - b - 1} \expo{-\ic\pars{a - b - 1}\pi/2} \pars{-y^{2} + 1}^{b}\ \ic\,\dd y}^{\ds{\large\tt IV}} \\[3mm]&\left.-\ \overbrace{% \left.\int_{\pi/2}^{0}z^{a - b - 1}\pars{z^{2} + 1}^{b}\,\dd z\, \right\vert_{z\ \equiv\ -\ic + \epsilon\expo{\ic\theta}}}^{\ds{\large\tt V}} \right\rbrace\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\pars{2} \end{align}

Now, we'll check the integral behaviours when $\ds{\epsilon \to 0^{+}}$. Lets examine the integral $\ds{\large\tt I}$: \begin{align} &\mbox{When}\ \epsilon \to 0^{+}\,,\quad \left.\int_{0}^{-\pi/2}z^{a - b - 1}\pars{z^{2} + 1}^{b}\,\dd z\, \right\vert_{z\ \equiv\ \ic + \epsilon\expo{\ic\theta}} \sim \epsilon^{b + 1} \to 0\quad\mbox{since}\quad b > -1. \\[3mm]&\mbox{Similarly, the integral}\ {\large\tt V} \to 0. \\[3mm]&\mbox{The integral}\ {\large\tt III}\ \mbox{behaves as}\ \epsilon^{a - b} \to 0\ \mbox{since}\ a > b. \\[3mm]&\mbox{Integrals}\ {\large\tt II}\ \mbox{and}\ {\large\tt IV}\ \mbox{converge when}\ \epsilon \to 0^{+}\ \mbox{since}\ a - b - 1>-1\ \mbox{and}\ b > -1. \end{align}

Then, we are left with $\ds{\pars{~\mbox{see expression}\ \pars{2}~}}$ \begin{align}&\color{#00f}{% \int_{\verts{z}\ =\ 1 \atop{\vphantom{\Huge A}\verts{{\rm Arg}\pars{z}}\ <\ \pi/2}} z^{a - b - 1}\pars{z^{2} + 1}^{b}\,\dd z} \\[3mm]&=-\expo{\ic\pars{a - b}\pi/2}\int_{1}^{0}y^{a - b - 1} \pars{1 - y^{2}}^{b}\,\dd y +\expo{-\ic\pars{a - b}\pi/2}\int_{0}^{-1}\pars{-y}^{a - b - 1} \pars{1 - y^{2}}^{b}\,\dd y \\[3mm]&=\expo{\ic\pars{a - b}\pi/2}\int_{0}^{1}y^{a - b - 1} \pars{1 - y^{2}}^{b}\,\dd y -\expo{-\ic\pars{a - b}\pi/2}\int_{0}^{1}y^{a - b - 1}\pars{1 - y^{2}}^{b}\,\dd y \\[3mm]&=\color{#00f}{2\ic\sin\pars{\bracks{a - b}\,{\pi \over 2}}\int_{0}^{1} y^{a - b - 1}\pars{1 - y^{2}}^{b}\,\dd y} \end{align}

Replacing this result in expression $\pars{1}$: \begin{align}&\color{#c00000}{\int_{-\pi/2}^{\pi/2}\cos\pars{ax}\cos^{b}\pars{x} \,\dd x} ={\sin\pars{\bracks{a - b}\pi/2} \over 2^{b - 1}}\,\int_{0}^{1} y^{a - b - 1}\pars{1 - y^{2}}^{b}\,\dd y \\[3mm]&={\sin\pars{\bracks{a - b}\pi/2} \over 2^{b - 1}}\,\int_{0}^{1} y^{\pars{a - b - 1}/2}\pars{1 - y}^{b}\,\half\,y^{-1/2}\,\dd y \\[3mm]&={\sin\pars{\bracks{a - b}\pi/2} \over 2^{b}}\,\int_{0}^{1} y^{\pars{a - b}/2 - 1}\pars{1 - y}^{b}\,\dd y \\[3mm]&={\sin\pars{\bracks{a - b}\pi/2} \over 2^{b}}\, {\Gamma\pars{\bracks{a - b}/2}\Gamma\pars{b + 1}\over \Gamma\pars{\bracks{a + b}/2 + 1}} \end{align}

Here we used the Gamma Function Euler Reflection Identity: \begin{align}&\color{#66f}{\large\int_{\pi/2}^{\pi/2}\cos\pars{ax}\cos^{b}\pars{x}\,\dd x} \\[3mm]&=\color{#66f}{\large{\pi \over 2^{b}}\, {\Gamma\pars{b + 1}\over \Gamma\pars{1 + \bracks{b + a}/2}\Gamma\pars{1 + \bracks{b - a}/2}}} \end{align}