Why (and how) did using cat on binary files mess up the terminal?

cat concatenates files given as arguments on command line to standard output, it reads bytes at a time an as default does not perform any interpretation of the bytes it reads.

In your first example you are redirecting stdout to a file, that's why you get a new file.

In your second example the bytes are written to the terminal, and it is the terminal which is interpreting sequences of characters as control sequences for the terminal, this is why you get unusual behaviour on your terminal. It has nothing to do with cat as such, cat doesn't know what you are going to do with it's output. You might be sending it through a pipe to another program to interpret/process/print or play "Singing in the rain".

So following the unix philosophy,

do one thing, do one thing only, but do it well

cat should not attempt to second guess what you are trying to do.

edit 1 reply to @kiwy's 1st comment below.

Yes and No, let me explain,

No, if you cat to a terminal, because it (the terminal software) is sending the output to your screen or interpreting control sequences (it is emulating an old piece of hardware ie. a teletype device).

but,

Yes if you cat to a pipe and the program recieving can interpret the characters as commands.

look cat this for an example, cat anyOldShellScript | bash bash will interpret what it gets as commands.

I guess this happens mostly because of non-printable characters with codes below 0x20. Those are special control/escape codes, which are used for keys like Backspace, Delete etc.