How to use sed with round brackets?

( isn't a special character in sed regular expressions. You don't need to escape it.

sed -e 's/jQuery(function/head.ready(function/g'

[(] in a regex means a character set containing just one element, (. It's a convoluted way of writing (.

[(] in replacement text means the three characters [(]. Brackets have no special meaning in replacement text.

You can avoid repeating the function name if you like, by putting it in a group. Groups are delimited by backslash-parenthesis.

sed -e 's/jQuery(\(function\)/head.ready(\1/g'

Beware that your replacement will also affect jQuery(functionwithsuffix. The easy way to avoid this is to use a tool that understands extended Perl regular expressions, so you can use (?![^A-Za-z0-9_]) at the end of the search string, meaning “followed by something that isn't an alphanumeric character”. Note that in Perl regular expressions, ( is special, you need to prefix it with a backslash.

perl -pe 's/jQuery\(function(?![^A-Za-z0-9_])/head.ready(function/g'
perl -pe 's/jQuery\((function)(?![^A-Za-z0-9_])/head.ready($1/g'

There is no need to escape the brackets in the replacement string.

sed -e 's/jQuery[(]function/head.ready(function/g'

( is a special character in sed, if using extended regular expression (ERE).

For example, sed -e will fail on Unmatched error, when you need to escape round brackets (for whatever reason), but run sed without specifying ERE:

$ regex="\("; echo "...(..." | sed 's/'$regex'/_/g'
sed: -e expression #1, char 8: Unmatched ( or \(

$ regex="\("; echo "...(..." | sed -e 's/'$regex'/_/g'
sed: -e expression #1, char 8: Unmatched ( or \(

In such case you'll have to use sed with -E option (or -r, --regexp-extended):

$ regex="\("; echo "...(..." | sed -E 's/'$regex'/_/g'
..._...

$ regex="\("; echo "...(..." | sed -r 's/'$regex'/_/g'
..._...