What's the difference betwen " and ' - and when to use it?

There is a difference between single and double quotes, and it's not a matter of style. Single quotes will literally echo what you write, so you can be sure that what you write is what it'll be printed. On the other hand, double quotes will interpolate variables' value.

From man bash:

Enclosing characters in single quotes preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash.

Enclosing characters in double quotes preserves the literal value of all characters within the quotes, with the exception of $, backtick, \, and, when history expansion is enabled, !. The characters $ and backtick retain their special meaning within double quotes. The backslash retains its special meaning only when followed by one of the following characters: $, `, ", \, or newline. A double quote may be quoted within double quotes by preceding it with a backslash. If enabled, history expansion will be performed unless an ! appearing in double quotes is escaped using a backslash. The backslash preceding the ! is not removed.

So in your example:

#!/bin/bash
PLACE="World"
echo "Hello $PLACE !"
echo 'Hello $PLACE !'

will result in this output:

Hello World !
Hello $PLACE !

https://www.gnu.org/software/bash/manual/bashref.html#Quoting

Inside double quotes variables are expanded.

Single quotes prevent string interpolation:

Enclosing characters in single quotes preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash.

There’s also the notation $'…' which allows allows for ANSI C escape sequences.