Why does exclamation mark within double quotes cause a Bash error?

! is the default history expansion character in Bash, see the section "HISTORY EXPANSION" in the Bash manpage

• History expansion doesn't take place if the ! is enclosed by single quotes, as in

  notify-send 'SYNC TIME!'
  

• History expansion doesn't take place if the ! is followed by a space, tab, newline, carriage return, or =, as in

  notify-send SYNC TIME!
  

• History expansion does take place in

  echo "SYNC TIME!"
  

  So you'll get an error if there isn't a command starting with " in your history

Because in bash, ! is a reserved word (OK, character), it has special meaning in different contexts. In this particular case, you are falling afoul of its significance in history searching. From man bash:

   History expansions introduce words from the history list into the input
   stream, making it easy to repeat commands, insert the  arguments  to  a
   previous command into the current input line, or fix errors in previous
   commands quickly.

  [...]

   History expansions are introduced by
   the appearance of the  history  expansion  character,  which  is  !  by
   default.   Only  backslash  (\) and single quotes can quote the history
   expansion character.

Basically, what this means is that bash will take the characters after the ! and search your history for the first command it finds that starts with those characters. It is easier to demonstrate than explain:

$ echo foo
foo
$ !e
echo foo
foo

The ! activated history expansion, which matched the first command starting with e which was the previously run echo foo which was then run again. So, when you wrote "SYNC TIME!", bash saw the !", searched history for a command starting with ", failed and complained about it. You can get the same error by running, for example !nocommandstartswiththis.

To print an exclamation mark, you need to escape it in one of these two ways:

echo 'Hello world!'
echo Hello world\!