SCP fails without error

Ok LOL, I just figured out what the problem is.

Since I like cows so much, I've put fortune | cowsay at the top of my .bashrc file which produces output like the following when starting bash:

 _______________________________________
< You will lose an important disk file. >
 ---------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

This is all fine (and sometimes funny) when running bash interactively. However, bash reads ~/.bashrc when it is interactive and not a login shell, or when it is a login shell and its parent process is rshd or sshd. When you run scp, the server starts a shell which starts a remote scp instance. The output from .bashrc confuses scp because it is sent the same way the scp protocol data is sent. This is apparently a known bug, see here for more details.

Also note that the underscores I mentioned in the question are those in the top line of the text balloon.

So the solution was simple: I put the following at the top of .bashrc on the remote (destination) machine:

# If not running interactively, don't do anything
[[ $- == *i* ]] || return

This line is present in the default .bashrc but was put way down because of my many (apparently careless) edits.

AFAIK, the right way to enable un-hindered scp is less about which conditional for stdout in your ~/.bashrc script, and more about simply restricting screen output to the ~/.bash_profile script. At least that is how it works for my distro (CentOS.)

Edit for clarity:

1. Put only lines in your ~/.bashrc file as required by "all" remote conections (i.e. setting certain ENV vars is OK, but echoing human-readable text is not.)
2. YMMV