Removing and adding permission using numerical notation on the same line

Noting the title of your question:

Removing and adding permission using numerical notation on the same line

With chmod from GNU coreutils, which you probably have on a Linux system, you could use

$ chmod -020,+004 test.txt

to do that. It works in the obvious way: middle digit for the group, 2 is for write; and last digit for "others", and 4 for read.

Being able to use + or - with a numerical mode is a GNU extension, e.g. the BSD-based chmod on my Mac gives an error for +004:

$ chmod +004 test.txt
chmod: Invalid file mode: +004

So it would be simpler, shorter, more portable and probably more readable to just use the symbolic form:

$ chmod g-w,o+r test.txt

EDIT: Seeing ilkkachu's answer, made me test this, and the syntax he describes works, but the man page on my system (which I had checked) says

can be either a symbolic representation of changes to make, or an octal number representing the bit pattern for the new mode bits

which was what caused me to write the original answer (below). I guess it's a new change and that they forgot to update the man page.

Original answer below:

The title of your questions mentions "numerical notation", if by that you mean specifying modes as octal numbers, the answer is that you can't specify changes with that, only the new mode of the file(s).

As the 2nd line of man page says:

chmod g-w,o+r file

$ man chmod

CHMOD(1)                         User Commands             CHMOD(1)

NAME
       chmod - change file mode bits

SYNOPSIS
       chmod [OPTION]... MODE[,MODE]... FILE...

or the 1st line of chmod --help

Usage: chmod [OPTION]... MODE[,MODE]... FILE...