Interlocking Brackets

CJam, 18

l7~f&_f{\/~;&}s,2/

Thanks isaacg for some golfing ideas :)
Try it online or try all examples

Explanation:

l         read a line of input
7~f&      clear the lowest 3 bits of each character
           the goal is to convert brackets of the same type to the same char
_         duplicate the resulting string, let's call it S
f{…}      for each character in S, and S (the char and S are pushed every time)
  \       swap the character with S
  /       split S around that character, resulting in 3 pieces:
           before, between, after
  ~       dump the pieces on the stack
  ;       pop the last piece
  &       intersect the first 2 pieces
          after the loop, we have an array of strings
          containing the chars interlocking to the left with each char of S
s         join all the string into one string
,         get the string length
2/        divide by 2, because S has duplicated characters

Python 2, 163 bytes

def f(b,e='([{<)]}>',q=range(4)):
 b=[b[b.index(e[j])+1:b.index(e[j+4])]for j in q]
 print sum(sum(abs(b[k].count(e[j])-b[k].count(e[j+4]))for j in q)for k in q)/2

This looks at the stuff between each pair of matching brackets and counts the number of individual left or right brackets present. The sum of these divided by two is the output.

I'm sure it could be golfed a lot more by better golfers than me.

GNU sed -r, 147

Output is in unary as per this meta-answer.

y/([{</)]}>/
s/.*/\t& & & & /
:b
y/)]}>/]}>)/
s/\S*>(\S*)>\S* /\1\t/
t
s/\S* //
:
s/(\t\S*)(\S)(\S*)\2(\S*\t)/\1\3\4/
t
s/\S *$/&/
tb
s/\s//g
s/../1/g

Note: Replace \t with actual tab characters to get the correct score. However, the program will work either way with GNU sed.

Try it online.