How can I run /usr/bin/Xorg without sudo?

To determine who is allowed to run X configure it with

dpkg-reconfigure x11-common

There are three options: root only, console users only, or anybody. The entry is located in /etc/X11/Xwrapper.config.

---

Since Debian 9 and Ubuntu 16.04 this file does not exist. After installing xserver-xorg-legacy, the file reappears and its content has to be changed from:

allowed_users=console

to:

allowed_users=anybody
needs_root_rights=yes

You also need to specify the virtual terminal to use when starting X, otherwise, errors may occur. For example:

Xorg :8 vt8

X needs access to input devices (mouse and keyboard) and to the monitor and graphics card for output.

To achieve this for non-root X, you can change the group of Xorg from root to input, set the setgit bit, and add your user to group video:

chown root:input /usr/lib/xorg/Xorg
chmod g+s /usr/lib/xorg/Xorg
adduser YOURUSERNAME video

(Instead you could add your user to both video and input, but that is discouraged. A user in group input can spy on inputs of other users. E.g. a GUI in X could probably spy on root password typed into console/tty.)