Are there any GUI's for Linux that doesn't use X11?

No. X is the only usable GUI on Linux.

There have been competing projects in the past, but none that gained any traction. Writing something like X is hard, and it takes a lot of extra work to get something usable in practice: you need hardware drivers, and you need applications. Since existing applications speak X11, you need either a translation layer (so… have you written something new, or just a new X server?) or to write new applications from scratch.

There is one ongoing project that aims to supplant X: Mir. It's backed by Canonical, who want to standardize on it for Ubuntu — but it hasn't gained a lot of traction outside Ubuntu, so it may not succeed more than Wayland (which was designed for 3D performance, not for security) did. Mir does aim to improve on the X security model by allowing applications limited privileges (e.g. applications have to have some kind of privilege to mess with other applications' input and output); whether that scales when people want to take screenshots and define input methods remains to be seen.

You can run a few graphical applications on Linux without X with SVGAlib. However that doesn't bring any extra security either (in addition to numerous other problems, such as poor hardware support, poor usability, and small number of applications). SVGAlib has had known security holes, and it doesn't get much attention, so it probably has many more. X implementations get a lot more attention, so you can at least mostly expect that the implementation matches the security model.

X has a very easily understood security model: any application that's connected to the X server can do anything. (That's a safe approximation, but a fairly realistic one.) You can build a more secure system on top of this, simply by isolating untrusted applications: put them in their own virtual environment, displaying on their own X server, and show that X server's display in a window. You'll lose functionality from these applications, for example you have to run things like window managers and clipboard managers in the host environment. There's at least one usable project based on this approach: Qubes.

The applications that you mention (Chrome, Flash, LibreOffice) are all built to run in X11 or a similar display server like Wayland.

It is possible to run all manner of applications[1] in the console, for example:

• email (mutt)
• web browsing (w3m)
• music (mpd/ncmpcpp)
• file managers (vifm)
• movies (mplayer), etc.

In combination with a terminal multiplexer, like tmux, you can manage the various 'windows' that the applications run in.

All of these applications are available in the repositories of most distributions.

[1] For each example I have provided, there are probably at least two to three more applications in each category that will do much the same thing... You will need a framebuffer to make this work well.

DirectFB is used in quite a lot in embedded applications. It provides a local-only windows and graphics environment with hardware acceleration etc. You can run X11 applications under it using XDirectFB