Matching special characters with Regular Expression

To match a literal ] and a literal - in a Bracket Expression you'll have to use them like this:

[^]/\^:-]

or, even better, since some tools require the backslash to be escaped:

[^]/\\^:-]

that is
The right-square-bracket ( ']' ) shall lose its special meaning and represent itself in a bracket expression if it occurs first in the list (after an initial '^', if any)
and
The hyphen-minus character shall be treated as itself if it occurs first (after an initial '^', if any) or last in the list
hence
If a bracket expression specifies both '-' and ']', the ']' shall be placed first (after the '^', if any) and the '-' last within the bracket expression.
The rules for bracket expressions are the same for ERE and BRE.

Strangely perhaps, you need to have a couple of characters in specific places. You must have ] as the first character in a set, and - must be the last character in a set.

Here is an RE that meets your requirements: [^]\^/-]:

echo 'Hello[world]-[sun] ^^ 4/5 or 3\4' | grep -Eo '[^]\^/-]' | xargs
H e l l o [ w o r l d [ s u n 4 5 o r 3 4