What's the difference between $(stuff) and `stuff`?

The old-style backquotes ` ` do treat backslashes and nesting a bit different. The new-style $() interprets everything in between ( ) as a command.

echo $(uname | $(echo cat))
Linux

echo `uname | `echo cat``
bash: command substitution: line 2: syntax error: unexpected end of file
echo cat

works if the nested backquotes are escaped:

echo `uname | \`echo cat\``
Linux

backslash fun:

echo $(echo '\\')
\\

echo `echo '\\'`
\

The new-style $() applies to all POSIX-conformant shells.
As mouviciel pointed out, old-style ` ` might be necessary for older shells.

Apart from the technical point of view, the old-style ` ` has also a visual disadvantage:

• Hard to notice: I like $(program) better than `program`
• Easily confused with a single quote: '`'`''`''`'`''`'
• Not so easy to type (maybe not even on the standard layout of the keyboard)

_{(and SE uses ` ` for own purpose, it was a pain writing this answer :)}

Obvious difference I observe is that you cannot nest backticks while you can nest $(). Maybe both exist for legacy reasons. Similarly, the . and source commands are synonyms.

$() does not work with old Bourne shell. But it has been years decades since I worked with old Bourne shell.