The Will Rogers Phenomenon

Pyth, 29 28 26 24 bytes

Thanks to @Jakube for saving me 3 bytes with .p and L.

Very simple, checks if any elements in list 2 are greater than mean of list 1 and less than mean of list 2, then repeats with list 1 and list 2 switched.

Lcsblbff&>YyhT<YyeTeT.pQ

Prints an a non-empty list for truthy, and [] for falsey.

L                    Define y(b). Pyth has no builtin for mean
 c                   Float div
  sb                 Sum of b
  lb                 Length of b
f        .pQ         Filter all permutations of input
 f     eT            Filter the last element of the filter var
  &                  Logical and
   >Y                Inner filter var greater than
    y                Call the mean function we defined earlier
     hT              First element of outer filter var
   <Y                Inner filter var less than
    y                Mean of
     eT              Last element of outer filternvar

Try it online here.

Test Suite.

Haskell, 58 57

x%y=any(\n->(\g->g x<0&&g y>0)$sum.map(n-))x
x?y=x%y||y%x

we can check if we enlarge or lower the average by checking whether the element to remove or include is bigger or smaller than the average.

we can check that by checking if the average is smaller or bigger than an element by removing that element from the array, and checking whether the new array's average is negative or positive, which in turn is just as checking whether the sum is positive or negative.

checking that is put very simply as sum.map(-n+).

Python 3, 74

lambda*L:any(sum(C)/len(C)>x>sum(D)/len(D)for(C,D)in[L,L[::-1]]for x in C)

Takes two lists as input. Checks if the first list has an element that's bigger than it's average but smaller than the other one's. Then, does the same for the two inputs swapped. Having a two-layer list comprehension was shorter than defining a separate function to try the two orders (82):

f=lambda A,B:any(sum(A)/len(A)>x>sum(B)/len(B)for x in A)
lambda A,B:f(A,B)|f(B,A)