What is the Mode_switch modifier for?

Mode_switch is the old-style (pre-XKB) name of the key that is called AltGr on many keyboard layouts. It is similar to Shift, in that when you press a key that corresponds to a character, you get a different character if Shift or AltGr is also pressed. Unlike Shift, Mod_switch is not a modifier in the X11 sense because it normally applies to characters, not to function keys, so applications only need to perform a character lookup to obtain the desired effect.

ISO_Level3_Shift is the XKB version of this key. Generally speaking, XKB is a lot more complicated and can do some extra fancy stuff. XKB's mechanism is more general as it allows keyboard layouts to vary in which keys are influenced by which modifiers, it generalizes sticky (CapsLock-style) and simultaneous-press (Shift-style) modifiers and so on.

This answer is not meant to contradict @Gilles' answer, but is meant to describe another aspect of xkb Input handling which I feel is "under-illuminated" in his answer.

You can use both ISO_Level3_Shift(Alt-Gr) and Mode_switch at the same time with different results. It requires some configuration, but you can have them both, and use them to produce 6 different keys per physical keyboard key!

You can e.g. configure Tab to be Mode_switch when held with xcape ⌂ and xmodmap, then e.g. the key labeled "H" on the keyboard could produce following characters/keys:

h
H
← (Cursor)
← (Symbol)
┘
┐ with H
with ShiftH
with AltGrH
with AltGrShiftH
with TabH
with TabShiftH

So, to come back around to your question: IL3S and Mode_switch can be configured to mean different things, so you can have cursor keys and ~200 useful characters from the (accented) languages, box-drawing, and technical character set on the 35 keys of the 3 letter rows.

To accomplish the above:

# Initial Setup
xmodmap -e 'keycode 23 = Mode_switch ISO_Left_Tab Tab ISO_Left_Tab Tab ISO_Left_Tab'
xmodmap -e 'keycode any = Tab'
xcape -e "Mode_switch=Tab"

example for H:

xmodmap -pke | grep 'h H'
# Make note of the keycode, note the 8 columns after the `=`,
# then edit and feed back into xmodmap with:
xmodmap -e 'keycode  43 = h H lowrightcorner uprightcorner Left leftarrow h H'

Mode_shift in Xkb is a shift key for groups or layouts. If you have a layout with multiple groups, you can hold down this key to access the next group. If you have several layouts, you can hold down this key to shift between layouts (†1).

So Mode_shift is to groups what Shift (or Shift_L and Shift_R, to use the keysym names) is to levels.

This might become more clear if we consider that Mode_shift has an alias called ISO_Group_Shift (†2). There are also keys called ISO_Group_Latch and ISO_Group_Lock which correspond to ISO_Level2_Latch (latch shift) and Shift_Lock, respectively (†3).

There are also a number of aliases for different scripts:

• kana_switch
• Arabic_switch
• Greek_switch
• Hebrew_switch
• Hangul_switch

†1: This is based on my understanding of Xkb’s “groups” and “layouts”. It seems that they are treated the same as far as selecting between them with keys like this.

†2: I am basing this on the file keysymdef.h in this Git repository at revision ab8666661fc6 (“Add XF86XK_AudioPreset”, 2017-05-10). On installations you will probably find this file at /usr/include/X11/keysymdef.h.

†3: There is also a key ISO_Lock, a kind of general-purpose lock key.

The meaning of the AltGr key

Gilles mentioned in his answer the connection with the AltGr key:

Mode_switch is the old-style (pre-XKB) name of the key that is called AltGr on many keyboard layouts.

That lead me to suspect that “AltGr” might stand for “alternate/alternative group”, since Mode_switch is used to shift groups. This turned out to be wrong, however. According to the Wikipedia page on the AltGr key, “AltGr” seems to come from IBM keyboards, and in their manuals it says that it is an abbreviation for “alternate graphics”. This makes sense, since it seems likely that using the keyboard to draw “graphics” (box-drawing characters used in text user interfaces) came well before there was good support for switching quickly between multiple scripts.