How can I do filtering between two matrix?

I would use a matrix language for such a task, e.g. GNU Octave.

Assuming you converted the pass/fail file into numerical values, e.g.:

sed 's/pass/1/g; s/fail/0/g' passfail > passfail.nums

You can now do the following:

marks    = dlmread('marks');
passfail = dlmread('passfail.nums');

for i = 1:size(marks)(1)
  sum(marks(i,:)(passfail(i,:) == 0))
end

Output:

ans =  100
ans =  78
ans =  53
ans =  91

While I think using awk is good for portability, other languages seem easier to write and read for this task. GNU Octave was mentioned but does not come pre-installed on most machines. On the other hand, most systems have a version of python preinstalled. Here is a python version:

for marks, decisions in zip(open('file1').readlines(), open('file2').readlines()):
    row_score = 0
    for mark, decision in zip(marks.split(), decisions.split()):
        if decision == 'fail':
            row_score += int(mark)
    print(row_score)

which returns the outputs you expected.

Here is my awk approach:

awk 'NR==FNR{for(i=1;i<=NF;i++) a[NR"-"i]=$i; next} \
            {for(j=1;j<=NF;j++) if($j=="fail") b[FNR]+=a[FNR"-"j]} \
         END{for(k in b) print b[k]}' file1 file2

Awk doesn't support two-dimensional arrays, so we cooked ones by combining two numbers (row and field) in the same array index. The output is:

100
78
53
91