Pig keeps trying to connect to job history server (and fails)

It seems that most Hadoop installation/configuration guides neglect to mention configuring the Job History Server. It seems that Pig, in particular, relies on this server. It also seems like the default (local) settings for the JHS won't work in a multi-node cluster.

The solution was to add the hostname of the server into the configuration in mapred-site.xml to make sure it could be accesses from the other machines. (In my version of the file, the lines had to be added as "new" ... there were no previous settings.)

<property>
  <name>mapreduce.jobhistory.address</name>
  <value>cm:10020</value>
  <description>Host and port for Job History Server (default 0.0.0.0:10020)</description>
</property>

Then restart the job history server:

mr-jobhistory-daemon.sh stop historyserver
mr-jobhistory-daemon.sh start historyserver

If you get a bind exception (port in use), it means the stop didn't work. Either

1. Use ps ax | grep -e JobHistory to get the process and kill it manually with kill -9 [pid]. Then call the start command above again. Or

2. Use a different port in the configuration

Pig should pick up the new settings automatically. Run a Pig script and hope for the best.