Question about exercise 20.7 in the TeXbook

The exercise is about interpreting

\def\!!1#2![{!#]#!!2}

when [, ] and ! have category code 1, 2 and 6, respectively.

The paragraph before the quoted one reads

Incidentally, if you display this with \tracingmacros=1, TeX says

\!!1#2[->{##]!!#2[
 #1<-x
 #2<-[y]

The duplication of the category code 6 characters is evident in the top line: the expansion of \! is shown to be

{##]!!#2

which means that the input stream will receive

{1#6]2!6[1y11]2

and a left brace is expected.


That part of the answer refers to the output of \tracingmacros=1. It also contains a mistake. (Edit: I got carried away by another discrepancy in the answer. The OP pointed out the correct interpretation; there is no mistake in this sentence.)


Exercise 20.7 on page 205, marked with a double “dangerous bend” symbol, asks:

Suppose that ‘[’, ‘]’, and ‘!’ have the respective catcodes 1, 2, and 6, as do ‘{’, ‘}’, and ‘#’. See if you can guess what the following definition means:

\def\!!1#2![{!#]#!!2}

What token list will result when ‘\! x{[y]][z}’ is expanded?

As mentioned on page 37, catcode 1 is “beginning of group” (like {). Catcode 2 is “end of group” (like }). Catcode 6 is parameter (like #). So to answer the question first:

\def\!!1#2![{!#]#!!2}
┗━┳┛┗┫┗┫┗┫┃┃┃┗┫┃┗┫┗┫┃
  ┃  ┃ ┃ ┃┃┃┃ ┃┃ ┃ ┃┃
  ┗━━╋━╋━╋╋╋╋━╋╋━╋━╋╋━━━━━ \def primitive defines a macro.
     ┗━╋━╋╋╋╋━╋╋━╋━╋╋━━━━━  \!  means that the command being defined is \! (like the \foo in \def\foo{bar} etc).
       ┗━╋╋╋╋━╋╋━╋━╋╋━━━━━  !1  is like #1 and means parameter 1, the first parameter passed to the macro when used.
         ┗╋╋╋━╋╋━╋━╋╋━━━━━  #2  means parameter 2, the second parameter passed to the macro when used.
          ┗╋╋━╋╋━╋━╋╋━━━━━  !   (as it's of catcode 6 and followed by a catcode 1 token) means, by the special rule mentioned after 20.5,
           ┃┃ ┃┃ ┃ ┃┃           that a token [ namely catcode 1, char 91 is inserted at end of both parameter text and replacement text.
           ┗╋━╋╋━╋━╋╋━━━━━  [   as it has catcode 1 is like { so this marks end of parameter text / start of replacement text of the macro
            ┗━╋╋━╋━╋╋━━━━━  {   means the token { namely catcode 1, char 123
              ┗╋━╋━╋╋━━━━━  !#  as it is two consecutive chars with catcode 6, means the token # namely catcode 6, char 35 -- HERE THE LATTER CHAR IS USED
               ┗━╋━╋╋━━━━━  ]   means the token ] namely catcode 2, char 93
                 ┗━╋╋━━━━━  #!  as it is two consecutive chars with catcode 6, means the token ! namely catcode 6, char 33 -- HERE THE LATTER CHAR IS USED
                   ┗╋━━━━━  !2  is like #2 and means parameter 2, the second parameter passed to the macro when used.
                    ┗━━━━━  }   marks the end of the replacement text and macro definition, but recall there's a { at end, by the special rule.

In case your browser doesn't display that properly:

answer

When you invoke the expansion mentioned in the question, e.g. with

\catcode`[=1
\catcode`]=2
\catcode`!=6
\def\!!1#2![{!#]#!!2}

\tracingmacros=1
\message{Expansion gives \! x{[y]][z} ok?}

\end

You get the first parameter set to x, then the second parameter set to [y] (namely {[y]] without the surrounding begin-group / end-group tokens), and the [ is consumed, and all this is replaced with the sequence of tokens {, #, ], !, [y], [. The log file shows

\!!1#2[->{##]!!#2[
!1<-x
#2<-[y]

(not #1<-x as the exercise mentions: this is a bug either in TeX or in The TeXbook, not sure which one will DEK will consider it to be). Here, in this \tracingmacros output:

  • “Category codes are not shown, but a character of category 6 always appears twice in succession.” — For example, the ## and !! that are shown, each representing a single catcode-6 token with char code that of # and ! respectively.

  • “A parameter token in the replacement text uses the character code of the final parameter in the parameter text.” — This refers to the fact that the output parameter in the first log line (on the right-hand side of the -> in \!!1#2[->{##]!!#2[) is shown as #2 rather than !2. If we have \def\foo#1!2{#1#2} then we see \foo #1!2->!1!2, and if we have \def\foo!1#2{#1#2} then we see \foo !1#2->#1#2. That is, every parameter token shown in the replacement text by \tracingmacros uses the last catcode-6 character that was used for a parameter in the parameter-text part of the macro definition. [Edit: This interpretation was pointed out by the OP; removed incorrect explanation that was here.]