How to trigger action on low-memory condition in Linux?
Yes, the Linux kernel does provide a mechanism for this: memory pressure notification. This is documented in https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt, section Memory Pressure.
In short, you register an eventfd file descriptor in /sys/fs/cgroup/memory/memory.pressure_level on which you want to receive notifications. These notifications can be low, medium, or critical. A typical use case would be to free some or all internal caches in your process when you receive a notification, in order to prevent an impending OOM kill.
What you are asking is, basically, a kernel-based callback on a low-memory condition, right? If so, I strongly believe that the kernel does not provide such mechanism, and for a good reason: being low on memory, it should immediately run the only thing that can free some memory - the OOM killer. Any other programs can bring the machine to an halt.
Anyway, you can run a simple monitoring solution in userspace. I had the same low-memory debug/action requirement in the past, and I wrote a simple bash which did the following:
monitor for a soft watermark: if memory usage is above this threshold, collect some statistics (processes, free/used memory, etc) and send a warning email;
monitor for an hard watermark: if memory usage is above this threshold, collect some statistics and kill the more memory hungry (or less important) processes, then send an alert email.
Such a script would be very lightweight, and it can poll the machine at small interval (ie: 15 seconds)