Why do some umask values not take effect?

umask is a mask, it’s not a subtracted value. Thus:

• mode 666, mask 022: the result is 666 & ~022, i.e. 666 & 755, which is 644;
• mode 666, mask 021: the result is 666 & ~021, i.e. 666 & 756, which is 646.

Think of the bits involved. 6 in a mode means bits 1 and 2 are set, read and write. 2 in a mask masks bit 1, the write bit. 1 in a mask masks bit 0, the execute bit.

Another way to represent this is to look at the permissions in text form. 666 is rw-rw-rw-; 022 is ----w--w-; 021 is ----w---x. The mask drops its set bits from the mode, so rw-rw-rw- masked by ----w--w- becomes rw-r--r--, masked by ----w---x becomes rw-r--rw-.

You need to think in binary, not decimal. Specifically, there are three 3-bit binary numbers: one each for Owner, Group, and Other. Each with values ranging from 000 to 111 (0-7 in decimal).

e.g. rw-rw-rw (666) is 110 110 110.

The umask value is a mask specifying which bits will be on or off (1 or 0) when creating a new file or directory. e.g. 022 decimal is 000 010 010 binary, while 021 decimal is 000 010 001

The permission bits are AND-ed together with the negated umask to arrive at the final value. "negated" means that all bits are inverted, i.e. all 1s flipped to 0, and vice-versa. e.g. NOT 022 (000 010 010) = 755 (111 101 101)

Example: 666 & !022 = 644. In binary, that's:

  Owner  Group  Other     mode
110    110    110      666
&  111    101    101      755   (this is the negated 022)
---    ---    ---      ---
110    100    100      644


Also, 777 & !022 = 755:

  Owner  Group  Other     mode
111    111    111      777
&  111    101    101      755
---    ---    ---      ---
111    101    101      755


Note how the final value of each bit can only be 1 if it is 1 in both the original permission value (666 or 777) AND in the negated umask. If either of them is 0, the result is 0. That is, 1 & 1 = 1, while 1 & 0 = 0.

Strictly speaking there's a fourth 3-bit binary number for the setuid, setgid, and sticky bits. That's why you often see permissions and masks specified with a leading 0 (or some other leading number from 0-7). e.g. 0777 or 2755.