Datanode process not running in Hadoop

You need to do something like this:

  • bin/stop-all.sh (or stop-dfs.sh and stop-yarn.sh in the 2.x serie)
  • rm -Rf /app/tmp/hadoop-your-username/*
  • bin/hadoop namenode -format (or hdfs in the 2.x serie)

the solution was taken from: http://pages.cs.brandeis.edu/~cs147a/lab/hadoop-troubleshooting/. Basically it consists in restarting from scratch, so make sure you won't loose data by formating the hdfs.


I faced similar issue while running the datanode. The following steps were useful.

  1. In [hadoop_directory]/sbin directory use ./stop-all.sh to stop all the running services.
  2. Remove the tmp dir using rm -r [hadoop_directory]/tmp (The path configured in [hadoop_directory]/etc/hadoop/core-site.xml)
  3. sudo mkdir [hadoop_directory]/tmp (Make a new tmp directory)
  4. Go to */hadoop_store/hdfs directory where you have created namenode and datanode as sub-directories. (The paths configured in [hadoop_directory]/etc/hadoop/hdfs-site.xml). Use

    rm -r namenode
    
    rm -r datanode
    
  5. In */hadoop_store/hdfs directory use

    sudo mkdir namenode
    
    sudo mkdir datanode
    

In case of permission issue, use

   chmod -R 755 namenode 

   chmod -R 755 datanode
  1. In [hadoop_directory]/bin use

     hadoop namenode -format (To format your namenode)
    
  2. In [hadoop_directory]/sbin directory use ./start-all.sh or ./start-dfs.sh to start the services.
  3. Use jps to check the services running.

I ran into the same issue. I have created a hdfs folder '/home/username/hdfs' with sub-directories name, data, and tmp which were referenced in config xml files of hadoop/conf.

When I started hadoop and did jps, I couldn't find datanode so I tried to manually start datanode using bin/hadoop datanode. Then I realized from error message that it has permissions issue accessing the dfs.data.dir=/home/username/hdfs/data/ which was referenced in one of the hadoop config files. All I had to do was stop hadoop, delete the contents of /home/username/hdfs/tmp/* directory and then try this command - chmod -R 755 /home/username/hdfs/ and then start hadoop. I could find the datanode!