What's the difference between CMAKE_INSTALL_PREFIX and CMAKE_INSTALL_RPATH

On a system which supports paths of the form c:/ABC/DEF (i.e. Windows), none. Windows binaries don't have a notion of rpath.

On systems which do have DT_RPATH and DT_RUNPATH (= those which use ELF binaries), the CMake variable CMAKE_INSTALL_RPATH is used to set up the value of DT_RPATH (or DT_RUNPATH) tags which will be written into the binaries at installation.

This is explained at CMake RPATH handling.

On Unix systems, dynamic libraries are searched for in a system-defined list of directories. (/etc/ld.so.conf -- Windows does this in its own way that is so convoluted that it usually boils down to "just use PATH". ;-) )

If you install a library (like the one you just compiled) in a custom directory not in that list, it will not be found if you run a dependent executable. RPATH is one way to fix this.

See the Wiki page linked above for details.

Firstly, CMAKE_INSTALL_PREFIX determines a "root" for the installed location of headers, libraries, executables, and other resources.

On a system which does not support the notion of a "search hierachy" for dependencies, CMAKE_INSTALL_RPATH is not used. However, on ELF-based systems (e.g. Linux) and Mach-based systems (e.g. macOS 10.5 and later) a set of additional locations to search can be set in executables and dynamic libraries (e.g. .so/.dylib files); this is the "Rpath" and you can set it during cmake's install phase, either for all targets by setting CMAKE_INSTALL_RPATH or for individual targets by setting INSTALL_RPATH on that target.

Static libraries are not dynamic (obviously!) so, CMAKE_INSTALL_RPATH has no utility at all for static libraries.

When installing dynamic objects, CMake will write the Rpath into the dynamic object provided CMAKE_SKIP_RPATH and CMAKE_SKIP_INSTALL_RPATH are both false. By default, the Rpath written will be set to CMAKE_INSTALL_PREFIX followed by the library destination, e.g. CMAKE_INSTALL_PREFIX/lib. On Linux systems, this would by default see an Rpath of /usr/local/lib written as Rpath.

You can examine the Rpath on Linux thus:

readelf -d libmylib.so

which produces something like:

0x000000000000000f (RPATH)              Library rpath: [/usr/local/lib]

or on macOS:

otool -l libmylib.dylib | grep -A 2 LC_RPATH

which produces something like:

          cmd LC_RPATH
      cmdsize 40
         path @loader_path/../Frameworks (offset 12)

To override the install Rpath you can set the variable CMAKE_INSTALL_RPATH. E.g. on Linux:

set(CMAKE_INSTALL_RPATH "\$ORIGIN/../lib")

or on macOS:

set(CMAKE_INSTALL_RPATH "@loader_path/../lib")