Why does scp get stalled? How do I resolve it?

This happens because scp is trying to grab as much bandwidth as possible, and any delay (by a firewall, etc.) can stall it. Limiting the bandwidth (with -l option) will fix it.

For example, you might want to limit the bandwidth to 1 MB/s (= 8192 Kbits/s):

 scp -l 8192 <file> <destination>

Source: http://www.aixmind.com/?p=1371 - Wayback Machine

I've managed to solve it by using rsync:

rsync -avz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --progress /tmp/bigfile.txt [email protected]:/tmp/

Any chance you're behind a Cisco ASA firewall? If so, turn off "sequence number randomization" and that'll help a lot -- also disable TCP Offload (ethtool -K $INTERFACE tso off gso off gro off) if you're on a Cisco ASA with Broadcom NICs in your server.