Updating the Linux kernel, while leaving rest of system as is

Linus Torvalds has a very strong opinion against kernel changes resulting in userspace regressions (see the question "The Linux kernel: breaking user space" for details).

Interface between userspace and kernel is provided by system calls. Newer kernels can have more system calls, and changes in exiting ones when those changes do not break existing applications. When a system call interface has a flag parameter, new kernels often expose the new functionality with a new bit flag. This way kernel maintains backwards compatibility to old applications.

When it has not been possible to alter existing interface without breaking userspace, additional system calls have been added that provide the extended functionality. This is why there are three versions of dup and two versions of umount system call.

The policy of having a stable userspace is the reason why kernel updates rarely cause issues in userspace applications and you do not generally expect issues after upgrading the kernel.

However same API stability is not guaranteed for kernel interfaces and other implementation details. Sysfs (on /sys) and procsfs (on /proc/) expose kernel implementation details on runtime configuration, hardware, network, processes etc. which are used by low-level applications. It is possible for those interfaces to change in an incompatible way between kernel versions if there is a good reason to. Changes still try to minimize incompatibilities if possible and there are rules for how applications can use the interfaces in a way least likely to cause issues. The impact is also limited because non low-level applications shouldn't be using these interfaces.

@PeterCordes pointed out if a change in procfs or sysfs breaks an application used by your distributions init scripts, you could have a a problem.

This depends somewhat on how your distribution updates kernel (long term support or mainline) and even then the issues are relatively rare as distributions usually ship the updated tools at the same time.

@StephenKitt added that upgraded userspace might require a newer version of the kernel, in which case the system might not be able to boot with the old kernel and that distribution release notes mention this when appropriate.

The Linux kernel and the user space of a Linux distribution, which historically was dominated by user tools developed by the GNU project, are loosely coupled. In part this is by design, and in part it is by necessity.

Unlike the BSDs, where the kernel as well as the base user space are designed and maintained together, the Linux kernel and its user space were developed and are maintained by different people. So keeping them tightly coupled together would be difficult, even if the community desired it, which I don't think it does.

And @sebasth also makes the excellent point that a major policy of the Linux kernel project is that it must not break user space. Which of course is another factor enforcing loose coupling.

However, there is still some degree of coupling. A sufficiently old Linux kernel will not work with modern distributions.

My understanding is that Linux is the kernel, and everything else is ancillary. You definitely can upgrade the kernel independently of (many) installed packages, as they are generally tied to libraries rather than the kernel itself. You will generally only see such coupling between kernel versions and hardware drivers (e. g. GPU drivers). Some software is only compatible with certain versions of the kernel, but that should be specified in those programs' individual documentation.

It's rather common to have two kernel package suites installed on a system - the currently used package, and the previously installed package. When upgrading, after ensuring the new version works properly, the oldest can be removed and you still have a known-safe fallback. Red Hat (and its cousins) for example, has package-cleanup --oldkernels --count 2 to do this automagically.