How to get near-perfect screen recording quality?

If your HDD allows, you can try to do it this way:

First write uncompressed file:

ffmpeg -f x11grab -s SZ -r 30 -i :0.0 -qscale 0 -vcodec huffyuv grab.avi

here SZ is your display size (e.g. 1920x1080).

After that you can compress it at any time you want:

ffmpeg -i grab.avi grab.mkv

Of course, you can change compression, select codec and so on.

I've had success with:

ffmpeg -f x11grab -video_size 1920x1080 -framerate 30 -i :1 \
       -vcodec libx264 -preset ultrafast -qp 0 -pix_fmt yuv444p \
       video.mkv

Key points:

• -qp 0: x264 lossless mode
• -preset ultrafast: smallest cpu usage, large size
• -pix_fmt yuv444p: the default, but ffmpeg suggests yuv420p, which is lossy

Hint: capture screen and webcam overlay with:

ffmpeg -f x11grab -thread_queue_size 64 -video_size 1920x1080 -framerate 30 -i :1 \
       -f v4l2 -thread_queue_size 64 -video_size 320x180 -framerate 30 -i /dev/video0 \
       -filter_complex 'overlay=main_w-overlay_w:main_h-overlay_h:format=yuv444' \
       -vcodec libx264 -preset ultrafast -qp 0 -pix_fmt yuv444p \
       video.mkv

https://trac.ffmpeg.org/wiki/Capture/Desktop

[edit 20180418] Update for Gnome Shell and Wayland:

#!/usr/bin/env python3
from dbus import SessionBus, Interface
from os import getcwd, path
from sys import argv

BUS_NAME = 'org.gnome.Shell.Screencast'
PATH_NAME = '/org/gnome/Shell/Screencast'
INTERFACE_NAME = BUS_NAME
PIPELINE='x264enc pass=qual quantizer=0 speed-preset=ultrafast ! queue ! matroskamux'

if path.isabs(argv[1]):
    name = argv[1]
else:
    name = path.join(getcwd(), argv[1])

bus = SessionBus()
screen_cast = bus.get_object(BUS_NAME, PATH_NAME)
screen_cast = Interface(screen_cast, INTERFACE_NAME)

ret, name = screen_cast.Screencast(name, {'pipeline': PIPELINE})

if not ret:
    print('Error starting screencast.')
    raise SystemExit(1)

try:
    i = input()
except (EOFError, KeyboardInterrupt):
    pass
finally:
    screen_cast.StopScreencast()