What does CTRL+4 (and CTRL+\) do in bash?

Ctrl+4 sends ^\

Terminals send characters (or more precisely bytes), not keys. When a key that represents a printable character is pressed, the terminal sends that character to the application. Most function keys are encoded as escape sequences: sequences of characters that start with the character number 27. Some keychords of the form Ctrl+character, and a few function keys, are sent as a control characters — in the ASCII character set, which all modern computers use as a basis (Unicode, ISO Latin-n, etc. are all supersets of ASCII), 33 characters are control characters: characters number 0 through 31 and 127. Control characters are not printable, but intended to have an effect in applications; for example character 10, which is Control-J (commonly written ^J), is a newline character, so when a terminal displays that character, it moves the cursor to the next line, rather than displaying a glyph. The escape character itself is a control character, ^[ (value 27).

There aren't enough control characters to cover all Ctrl+character keychords. Only letters and the characters @[\]^_? have a corresponding control character. When you press Ctrl+4 or Ctrl+$ (which I presume is Ctrl+Shift+4), the terminal has to pick something to send. Depending on the terminal and its configuration, there are several common possibilities:

• The terminal ignores the Ctrl modifier and sends the character 4 or $.
• The terminal sends an escape sequence that encodes the exact key and modifiers that were pressed.
• The terminal sends some other control character.

Many terminal send control characters for some keys in the digit row:

• Ctrl+2 → ^@
• Ctrl+3 → ^[
• Ctrl+4 → ^\
• Ctrl+5 → ^]
• Ctrl+6 → ^^
• Ctrl+7 → ^_
• Ctrl+8 → ^?

I don't know where this particular convention arose.

Ctrl+| sends the same character because it's Ctrl+Shift+\ and the terminal sends ^\ whether the shift key was pressed or not.

^\ quits

The terminal itself (more precisely, the generic terminal support in the kernel) interprets a few control characters specially. This interpretation can be configured to map different characters or turned off by applications that want to process the characters by themselves. One well-known such interpretation is that ^M, the character send by the Return key, sends the current line to the application, if the terminal is in cooked mode, in which applications receive input line by line.

A few characters send signals to the application in the foreground. ^C sends the interrupt signal (SIGINT), which conventionally tells the application to stop what it's doing and read the user's next command. Non-interactive applications usually exit. ^\ sends the quit signal (SIGQUIT), which conventionally tells the application to exit as soon as possible without saving anything; many applications don't override the default behavior, which is to kill the application immediately¹. So when you press Ctrl+4 (or anything that sends the ^\ character) in cat or bc, neither of which overrides the default behavior, the application is killed.

The terminal itself prints the ^\ part of the message: it's a visual depiction of the character that you typed, and the terminal is in cooked mode and with echo turned on (characters are displayed by the terminal as soon as you type them, as opposed to non-echo mode where the characters are only sent to the application, which may or may not choose to display them). The Quit part comes from bash: it notices that its child process died from a quit signal, and that's its way of letting you know.

Shells handle all common signals, so that if you type ^\ in a shell, you don't kill your session, you just get a new prompt, same as ^C.

You can play with terminal settings with the stty command.

¹ _{And traditionally generate a core dump, but many systems disable that by default nowadays.}

In addition to Gilles answer let me add, that you can always input non-printable characters in bash with Ctrl-v+key (Ctrl-v+Ctrl+4 in this case) and check the character code with

$ printf '^\' | od -An -tu    # input ^\ as C-v C-4
28

you get the decimal code of the character, which as you may check in man ascii corresponds to file separator (FS).