Why have a 0.1 uF capacitor in parallel with 2500 uF in a power supply filter?

What you have there isn't two capacitors in parallel. It's more like this:

^{simulate this circuit – Schematic created using CircuitLab}

The small cap is a ceramic type, it has a low series resistance and inductance, so high frequencies can pass it easily. It doesn't need high capacitance because the current in those frequencies is usually low.

The big cap is an electrolytic type, it has a high series inductance and most times a high series resistance, too. For high frequencies, it's similar to an open circuit.

Big electrolytic capacitors have substantial parasitic inductance in series. This makes its operation at high frequencies less than perfect. The amplifier circuit probably does not stand that much of unwanted inductance and becomes unstable. Bypassing the electrolytic with smaller non-electrolytic capacitor helps.