Spark execution memory monitoring

Answering my own question for future reference:

We are using Mesos as cluster manager. In the Mesos UI I found a page that lists all executors on a given worker and there one can find a Memory usage of the executor. It seems to be a total memory usage storage+execution. I can clearly see that when the memory fills up the executor dies.

To access:

• Go to Agents tab which lists all cluster workers
• Choose worker
• Choose Framework - the one with the name of your script
• Inside you will have a list of executors for your job running on this particular worker.
• For memory usage see: Mem (Used / Allocated)

The similar can be done for driver. For a framework you choose the one with a name Spark Cluster

If you want to know how to extract this number programatically see my response to this question: How to get Mesos Agents Framework Executor Memory