What is TeX's gullet?

The terminology isn't that useful really but as far as I can make out.

The mouth is where tokens get recognised so abc becomes three character tokens, and \abc becomes one control token

Unexpandable tokens are handled in the stomach so the gullet is basically the expansion process.

The texbook only uses gullet twice (and tex.web not at all):

expandable tokens are converted to unexpandable ones in TeX's ''gullet'' by a process similar to regurgitation.

and

What would be wrong with

 \def\%{\char`%}

? The % would be treated as a comment character, because its category code is 14; thus, no % token or } token would get through to the gullet of TeX where numbers are treated.

So as well as macros being replaced by their replacement text, the two tokens ``a being interpreted as a number (the character code of a) is also being seen as a gullet process. Conversely \char itself is stomach.


I don't think that the comparison TeX with a living being having eyes, mouth, stomach and bowels should be taken so seriously as to find an analogy with other body parts.

I like better the description Victor Eijkhout makes in “TeX by Topic” of TeX as operating on four levels (see pages 21 and 22):

The four levels are (corresponding roughly to the ‘eyes’, ‘mouth’, ‘stomach’, and ‘bowels’ respectively in Knuth’s original terminology) as follows.

  1. The input processor. This is the piece of TeX that accepts input lines from the file system of whatever computer TeX runs on, and turns them into tokens. Tokens are the internal objects of TeX: there are character tokens that constitute the typeset text, and control sequence tokens that are commands to be processed by the next two levels.

  2. The expansion processor. Some but not all of the tokens generated in the first level – macros, conditionals, and a number of primitive TeX commands – are subject to expansion. Expansion is the process that replaces some (sequences of) tokens by other (or no) tokens.

  3. The execution processor. Control sequences that are not expandable are executable, and this execution takes place on the third level of the TeX processor. One part of the activity here concerns changes to TeX’s internal state: assignments (including macro definitions) are typical activities in this category. The other major thing happening on this level is the construction of horizontal, vertical, and mathematical lists.

  4. The visual processor. In the final level of processing the visual part of TeX processing is performed. Here horizontal lists are broken into paragraphs, vertical lists are broken into pages, and formulas are built out of math lists. Also the output to the dvi file takes place on this level. The algorithms working here are not accessible to the user, but they can be influenced by a number of parameters.

Under this interpretation, the “gullet” would be the channel by which levels 2 and 3 communicate in both directions, for example when tokens sent to level 3 by \lowercase or \uppercase return to level 2 after having been converted according to the values in the \lccode and \uccode arrays. However, I think this would add nothing to the understanding of how things work.

There is another aspect that's not covered by the living being allegory: the process whereby TeX sends to level 2 the macro replacement texts. Level 3 is responsible for storing macro definitions, which level 2 accesses at.

Finally, I would reject comparing the process whereby DVI or PDF files are produced, along with the log and write streams, to another body part that I won't name. Such an analogy would be ridiculous.


Quoting from TeX For The Impatient, page 16, emphasis added:

The eyes transform an input file into a sequence of characters. The mouth transforms the sequence of characters into a sequence of tokens, where each token is either a single character or a control sequence. The gullet expands the tokens into a sequence of primitive commands, which are also tokens. The stomach carries out the operations specified by the primitive commands, producing a sequence of pages. Finally, the intestines transform each page into the form required for the .dvi file and send it there. These actions are described in more detail in Section 4 under “anatomy of TeX” (p. 46).

So, TeX's 'gullet' effectively performs expansion.

Tags:

Books

Tex Core