What is meant by a shell is in "vi" mode or "emacs" mode?

In "vi" mode you can edit/navigate on the current shell prompt like a line in the vi editor. You can look at it like a one-line text file. Analogously in "emacs" mode you can edit/navigate the current command line using (some) of Emacs' shortcuts.

Example

For example in vi-mode you can do something like (in bash):

$ set -o vi
$ ls hello world
<ESC>
bbdw # results in
$ ls world

In emacs-mode you can hit e.g. Ctrl+A to jump at the start of a line (vi: Ctrl+[, 0 or ESC,0). You can turn on emacs mode via set -o emacs (in bash, ksh, zsh etc.).

Readline

A lot of interactive command line programs (including bash) use the readline library. Thus, you can configure which input mode to use (vi or emacs) and other options in one place such that every program using readline has the exact same editing/navigating interface.

For example my readline configuration looks like:

$ cat ~/.inputrc 
set editing-mode vi
set blink-matching-paren on

For example zsh/ksh does not use readline as far as I know, but also support vi/emacs modes that are very much like the bash/readline one.

Of course, the vi/emacs mode in a command line shell is just a subset of the complete editor feature set. Not every feature makes sense in a command line shell, and some features are more complicated to support than others.

Canonical Mode

Before vi/emacs modes of interactive command line shells 'were invented' your shell would use just the canonical mode of your terminal which only provides a limited set of editing commands (e.g. Ctrl+W to delete the last word.

You'll notice that when you run cat at a shell prompt on a terminal, cat being supposed to write to stdout what it reads from stdin, and press a, you see a a echoed back by the terminal driver, but cat doesn't write that a (you see only one a, the one echoed by the terminal driver).

However, if you type a Backspace b Enter, you don't see cat outputting a\010b\015, but b\012 (b and newline).

That is because the terminal driver (we're talking software in the kernel, not in the terminal emulator like xterm) implements a very basic line editor when in canonical mode. The terminal driver can be configured using ioctl() system calls like when using the stty command. For instance, to leave the canonical mode, you can do stty -icanon. If you do:

stty -icanon; cat

Then, you'll see both the echo (which you could have disabled with stty -echo) and the cat output at the same time.

That editor is a line editor. That is, it is for the user to edit one line of text until it's sent to the application reading the terminal device upon pressing Enter.

The editing capabilities of that editor are very limited. In most implementations, there are only 4 editing keys (actually characters) also configurable with stty:

• erase (^H or ^? usually): erase the previous character
• kill (^U usually): empty (kill) the line entered so far
• werase (^W): erase the previous word
• lnext (^V): enter the next character literally (cancel the special meaning of all of the above)

Back in the old days, it was thought that that terminal driver line editor would be extended with fancier capabilities. That is why none of the early shells has any command line editing capabilities (you'd get the same line editing capabilities at the shell prompt than when running cat like we did above).

However, that really never happened, maybe part of the reason being the mess with different terminals not sending the same characters upon some key presses which made it evident that that should not be implemented in kernel space.

So some shells started dropping the canonical mode of the terminal driver and implementing their own line editor. At the time, emacs and vi were the most popular visual text editors with completely different key binding and operation mode. In vi, you have one mode for entering text, and one for editing. In emacs, you're always in entering text mode, but editing is done by pressing key combinations (like ^b to move the character backward).

There was no point for shells at the time to come up with their own different key binding. That would have caused frustration for people to have to learn a different one. However, choosing one (emacs or vi) style over the other would have been a sure way to alienate the users of the other editor.

According to https://www.usenix.org/legacy/publications/library/proceedings/vhll/full_papers/korn.ksh.a:

The popular inline editing features (vi and emacs mode) of ksh were created by software developers at Bell Laboratories; the vi line editing mode by Pat Sullivan, and the emacs line editing mode by Mike Veach. Each had independently modified the Bourne shell to add these features, and both were in organizations that wanted to use ksh only if ksh had their respective inline editor. Originally the idea of adding command line editing to ksh was rejected in the hope that line editing would move into the terminal driver. However, when it became clear that this was not likely to happen soon, both line editing modes were integrated into ksh and made optional so that they could be disabled on systems that provided editing as part of the terminal interface.

So instead, they implemented both and an interface for users to choose between the two. ksh was most probably the first in the early 80s (reusing code that had been written separately to add a vi mode and an emacs mode to the Bourne shell as seen above) followed by tcsh (tcsh initially only had emacs key binding, vi mode was added later) and later bash and zsh in the early 90s.

You switch between the two modes in bash, zsh or ksh with set -o vi or set -o emacs, and with bindkey -e or bindkey -v in tcsh or zsh.

POSIX actually specifies the vi mode and not the emacs mode for sh (the story has that Richard Stallman objected to POSIX specifying the emacs mode for sh).

The default mode for bash, the public domain variants of ksh (pdksh, mksh, oksh), tcsh and zsh is the emacs mode (though with zsh, it's vi if your $EDITOR is vi), while in the AT&T ksh, it's the dumb mode unless $EDITOR or $VISUAL mentions vi or emacs.

ksh also later added a gmacs mode to accommodate users of Gosling emacs that handled Ctrl+T differently.

Now the handling of ^W in emacs or in tcsh emacs mode probably predates the werase character in the terminal line editor, so we can't really blame them for that and my statement about "departing..." may be seen as misleading. It's just that I find it irritating when things like emacs, tcsh or info behave differently from everything else when you type Ctrl-W. You can imagine I found it a lot more irritating when some applications started to close their window when you typed Ctrl-W.