Replace ctest command with "ctest --output-on-failure" permanently for a specific project in CMakeLists.txt

In CMake 3.17 release notes, there's a new variable CMAKE_CTEST_ARGUMENTS that you can set to pass any command-line arguments to CTest, including --output-on-failure. In your specific case, you can now simply add this to your CMakeLists.txt:

list(APPEND CMAKE_CTEST_ARGUMENTS "--output-on-failure")

I went with make check per solution: CMake: setting an environmental variable for ctest (or otherwise getting failed test output from ctest/make test automatically)

add_custom_target(check ${CMAKE_COMMAND} -E env CTEST_OUTPUT_ON_FAILURE=1
                  ${CMAKE_CTEST_COMMAND} -C $<CONFIG> --verbose
                  WORKING_DIRECTORY ${CMAKE_BINARY_DIR})