Apache Spark upgrade from 1.5.2 to 1.6.0 using homebrew leading to permission denied error during execution

I ran into the same issue and the easiest fix is to set $SPARK_HOME to /usr/local/Cellar/apache-spark/<your_spark_version>/libexec/ instead.

You can also build from source directly and you can find the instructions here.

Basically just do

git clone https://github.com/apache/spark/`
cd spark
git checkout origin/branch-X.Y

build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package

You'll need to set the $SPARK_HOME to the top level directory of the spark source code.

A ticket on the project page helped:

unset SPARK_HOME && spark-submit

https://github.com/Homebrew/homebrew/issues/48898#issuecomment-180633967