How can I use CRF encoding with nvenc in ffmpeg?

For CRF-based encodes, pass the following arguments in the snippet below to FFmpeg:

-c:v h264_nvenc -rc:v vbr_hq -cq:v 19 -b:v 2500k -maxrate:v 5000k -profile:v high

Of course, you'll need to adjust for target bit rates and a fixed cq value. 19 is the recommended setting as its' visually identical to 0, yet preserves good compression trade off to file size. See this write-up for more on what CRF does.

Note that the -cq scale is logarithmic, meaning that 0 is essentially lossless and 51 would be the absolute worst.

Quality can be further improved upon by adding options such as B-frames (limit this to 3, at most, and this requires the H.264 Main profile and above. Baseline profiles do not support B-frames. To do this, pass -bf {uint} to the video encoder, such that -bf:v 4 would result in the encoder using 4 B-frames.

The key parts here are the -cq:v 19 and the -rc:v vbr_hq arguments, which allow you to tune the encoder with both a preset variable bitrate and a maximum allowable bitrate (-b:v and -maxrate:v) while adhering to a CRF value of 19.

And now, small notes about NVENC, and tuning it for high quality encodes:

NVENC, like any other hardware-based encoder, has several limitations, and in particular with HEVC, here are the known limitations:

  1. On Pascal:

    For HEVC encodes, the following limitations apply:

    • CTU sizes above 32 are not supported.
    • B-frames in HEVC are also not supported.
    • The texture formats supported by the NVENC encoder limit the color spaces that the encoder can work with. For now, we have support for 4:2:0 (8-bit) and 4:4:4 (for 10-bit). Extraneous formats such as 4:2:2 10-bit are not supported. This will affect some workflows where such colorspaces are required.
    • Look ahead control is also limited to 32 frames. You may want to look at this editorial for more details.

Turing has all the enhancements available to Pascal, with the addition of B-frame support for HEVC and the ability to use B-frames as a reference. See this answer for an example on this capability.

  1. And on Maxwell Gen 2 (GM200x series GPUs):

    HEVC encoding lacks the following features:

    • Sample Adaptive Offset (SAO) loop filter capabilities.
    • Adaptive quantization
    • Look-ahead rate control.

The impact here for Maxwell is that motion heavy scenes with HEVC under constrained bitrates may suffer from artifacting (blockiness) due to the missing lookahead functions and adaptive sample offset (SAO) loop filtering capabilities. Pascal has somewhat improved on this capability, but depending on the version of the SDK that the video encoder was built with, not all features may be available.

For instance, weighted prediction mode for H.264 encodes on Pascal requires NVENC SDK 8.0x and above, and this encode mode will also disable B-frame support. Likewise, the combination of hardware-based scalers running off the Nvidia Performance Primitives (NPP) with NVENC may introduce performance improvements with video scaling applications at the cost of scaling artifacting, particularly with upscaled content. The same also impacts the video encode pipeline as NPP's scaling functions run off the CUDA cores on the GPU, and as such, the performance impact introduced by the extra load should be analyzed on a case-by case basis to determine if the performance-quality trade-off is acceptable.

Keep this in mind: A hardware-based encoder will always offer somewhat lesser customization than an equivalent software-based implementation, and as such, your mileage and acceptable output quality will always differ.

And for your reference:

With FFmpeg, you can always refer to an encoder's settings for customization by:

ffmpeg -h encoder {encoder-name}

So, for NVENC-based encoders, you can run:

ffmpeg -h encoder=hevc_nvenc

ffmpeg -h encoder=h264_nvenc

You can also see all the NVENC-based encoders and NPP-based scalers (if built as such) by running:

for i in encoders decoders filters; do
    echo $i:; ffmpeg -hide_banner -${i} | egrep -i "npp|cuvid|nvenc|cuda"
done

Sample output on my testbed:

encoders:
 V..... h264_nvenc           NVIDIA NVENC H.264 encoder (codec h264)
 V..... nvenc                NVIDIA NVENC H.264 encoder (codec h264)
 V..... nvenc_h264           NVIDIA NVENC H.264 encoder (codec h264)
 V..... nvenc_hevc           NVIDIA NVENC hevc encoder (codec hevc)
 V..... hevc_nvenc           NVIDIA NVENC hevc encoder (codec hevc)
decoders:
 V..... h263_cuvid           Nvidia CUVID H263 decoder (codec h263)
 V..... h264_cuvid           Nvidia CUVID H264 decoder (codec h264)
 V..... hevc_cuvid           Nvidia CUVID HEVC decoder (codec hevc)
 V..... mjpeg_cuvid          Nvidia CUVID MJPEG decoder (codec mjpeg)
 V..... mpeg1_cuvid          Nvidia CUVID MPEG1VIDEO decoder (codec mpeg1video)
 V..... mpeg2_cuvid          Nvidia CUVID MPEG2VIDEO decoder (codec mpeg2video)
 V..... mpeg4_cuvid          Nvidia CUVID MPEG4 decoder (codec mpeg4)
 V..... vc1_cuvid            Nvidia CUVID VC1 decoder (codec vc1)
 V..... vp8_cuvid            Nvidia CUVID VP8 decoder (codec vp8)
 V..... vp9_cuvid            Nvidia CUVID VP9 decoder (codec vp9)
filters:
 ... hwupload_cuda     V->V       Upload a system memory frame to a CUDA device.
 ... scale_npp         V->V       NVIDIA Performance Primitives video scaling and format conversion

For -crf replacement from libx264 may be -cq or -qp from h264_nvenc:

-crf Select the quality for constant quality mode

-cq Set target quality level (0 to 51, 0 means automatic) for constant quality mode in VBR rate control

-qp Constant quantization parameter rate control method (from -1 to 51) (default -1)

Fastest hardware accelerated encode method:

ffmpeg -hwaccel cuvid -c:v h264_cuvid -resize 640x480 -i input.mp4 -c:v h264_nvenc -cq 21 -c:a copy output.mp4

-resize resolution on input (in hardware); no need for ffmpeg to be compiled with --enable-libnpp for scale_npp filter.

For more info:

ffmpeg -h encoder=h264_nvenc

ffmpeg -h denoder=h264_cuvid

I believe I found a solution:

ffmpeg -hwaccel auto -i in.mp4 -c:v h264_nvenc -preset llhq -rc constqp -qp 21 -c:a copy out.mp4

It seems that h264_nvenc uses -qp instead of -crf. This option only works while -rc is set to constqp.