What does \begingroup\expandafter…\endgroup do?

Let's look step by step

\begingroup\expandafter\expandafter\expandafter\endgroup
\expandafter\ifx\csname directlua\endcsname\relax
  A
\else
  B
\fi

This becomes

(\begingroup)\expandafter\endgroup
\ifx\directlua\relax
  A
\else
  B
\fi

The \begingroup has already been digested, so I leave it in parentheses just to remember a group has been opened. Another step, now, where we have to distinguish between cases.

Case 1: \directlua is not defined, so the token produced by\csname directlua\endcsname is equivalent to \relax.

(\begingroup)\endgroup A\else B\fi

Now \endgroup is digested and this removes the assignment of the meaning \relax to \directlua. A is examined, the expansion of \else B\fi is empty.

Case 2: \directlua is defined.

(\begingroup)\endgroup B\fi

Again \endgroup is digested, but does not restore anything. The expansion of \fi is empty.

Why not doing this inside a group? The key point is that at the end \directlua is not defined if it wasn't at the start of the process. The same would be true if the code is

\begingroup\expandafter\ifx\csname directlua\endcsname\relax A\else B\fi\endgroup

However the purpose of A and B is doing some assignments. In this case A would probably be \luatexfalse, after having said before \newif\ifluatex, and B would be \luatextrue. The triple \expandafter inside the group dispenses from a global assignment, following the good practice that assignments to a variable should be always global or always local (so long as it's possible). Of course in this case a global assignment would not be that important, in other cases it might have consequences on the save stack.

The suggested alternative

{\expandafter}\expandafter\ifx\csname directlua\endcsname\undefined
  A
\else
  B
\fi

(with \undefined, not \relax) is less attractive, because it relies on a certain token to be undefined. One could object that the code we're analyzing assumes \relax has its primitive meaning, but some assumptions need to be made.

If e-TeX can be assumed, the simpler test

\ifdefined\directlua
  A
\else
  B
\fi

is even fully expandable.

The reason here is that \csname ...\endcsname will define ... as a macro equal to \relax should it not already exists. This feature is used with \ifx which compares it to \relax. This test is true if ... wasn't defined before (or was \let to \relax).

However, it isn't good practice to define macros even to \relax just for testing their existence. e-TeX provides \ifcsname ...\endcsname for this. Without e-TeX the a group can be used together with \expandafter to process both the \csname and the \ifx inside it to keep the macro definition local.

Note that when an \if... is true TeX simply goes on with processing the following tokens. It remembers looking for the closing \fi which is simply removed or an \else branch which should be skipped. If the test is false everything till \else is immediately skipped and TeX again remembers to look for an closing \fi. Therefore all the \expandafter trickery works very well. The \ifx is expanded and TeX already has chosen which branch it will execute. Then the \endgroup is insert and that branch is executed.

The benefit for wrapping the whole expression inside a groups is very clear: the actual content can define/change local settings!

Note that that in TeX if-statements and groups are independent (which is not the case in almost any other programming language). You can therefore also write the following to keep the \csname ...\endcsname statement local:

\begingroup
\expandafter\ifx\csname directlua\endcsname\relax
    \endgroup
    …
\else
    \endgroup
    …
\fi

Only one \endgroup is ever executed here.