How to read oom-killer syslog messages?

The OOM killer suggests that in fact, you've run out of memory.

If you say it's got more memory than it needs then maybe some system event is creating a memory leak somewhere, but the OOM killer will not tell why there is a memory leak, only that it's run out of memory and now tries to kill the least important things (based on oom_score).

And if the case is that there is a memory leak, then maybe the oom-killer will only kill procs so that the rouge one can allocate more and more memory.

So what I would do in case, is

  1. Configure kdump, which will create a crash dump vmcore after a kernel panic. (it's described more here)
  2. Setting vm.panic_on_oom=1 kernel parameter. This will cause a kernel panic should the machine run out of memory.
  3. Next time you get a panic, you can open up the vmcore file created by kdump, and look at the process table, and it will reveal the culprit.