Score a Game of Boggle

Husk, 21 20 19 bytes

-2 bytes thanks to Zgarb

Idea taken from A055228

ṠṀöṁ(⌈√Π-3▼8L)fε`#Σ

Try it online!

Explanation (Of older version)

            ṠṀ-oṠ-uΣ   Remove duplicated words
                   Σ   Concatenate
                  u    Remove duplicates
               oṠ-     Remove those unique elements from the list (give list of elements that appear more than once)
            ṠṀ-        Remove those words from each list in the input
m                      For each list
 ṁ(                    Map then sum
          L)           Length
        ▼8             Min(8,x)
      -3               Minus 3
     Π                 Factorial
    √                  Square root
   ⌈                   Ceiling

R, 142 126 121 117 bytes

function(L)sapply(lapply(L,setdiff,(l=unlist(L))[duplicated(l)]),function(x)sum(c(1,1,2,3,5,11)[pmin(6,nchar(x)-2)]))

Try it online!

Takes L as a list of vectors of strings; returns the values.

First, it unlists the words, finds the duplicates, then removes them from the players' word lists. Then it takes these unique word lists and computes the scores of each, using pmin to ensure that words longer than 8 get scored as 11.

JavaScript (ES6), 106 93 bytes

[Saved 13(!) bytes, thanks to Arnauld, Shaggy, and JollyJoker.]

a=>a.map(b=>b.map(c=>x+=(a+'').split`,`.filter(d=>d==c)[1]?0:+'11235'[c.length-3]||11,x=0)|x)

Test cases:

let f=

a=>a.map(b=>b.map(c=>x+=(a+'').split`,`.filter(d=>d==c)[1]?0:+'11235'[c.length-3]||11,x=0)|x)

console.log(f([["cat","dog","bird","elephant"],
               ["bird","dog","coyote"],
               ["dog","mouse"]
              ])
           )  // [12,3,2]
           
console.log(f([["abc","def","ghi"],
               ["ghi","def","abc"]
              ])
           ) // [0,0]
           
console.log(f([["programming","puzzles"],
               ["code","golf"],
               []
              ])
            ) // [16,2,0]