Detect an executable file in java

I suspect that, apart from the extension-checking method you have already mentioned, there will be no way to catch every possible case. Executable files are ultimately sequences of machine instructions which make them largely indistinguishable from any other data.

Despite this however, there are things you could look for in certain types of executable. For example:

• Windows uses the Portable Executable format, which should always start with the magic number 4d5a (ASCII characters MZ)
• ELF format executable used by Linux start with 7f454c46
• Java class files always begin with cafebabe (that's hex, not ASCII!).
• As far as I can see, Mach-O files used by Mac-OSX have the magic number feedface (hex again)

I suggest you create a FileInputStream or similar and read the first few bytes of the file, checking for these magic numbers. It doesn't detect any file which contains executable code, but it should stop files in these standard executable formats from being allowed, which I think is what you hoped for.

So for example:

public static boolean isExecutable(File file) {
  byte[] firstBytes = new byte[4];
  try {
    FileInputStream input = new FileInputStream(file);
    input.read(firstBytes);

    // Check for Windows executable
    if (firstBytes[0] == 0x4d && firstBytes[1] == 0x5a) {
      return true;
    }
    return false;
  }
  catch (Exception e) {
    e.printStackTrace();
  }
}

Also beware that it is possible to get a false positive, where you reject a file which was not executable. I don't know what type of file you are intending to have uploaded so you should consider how likely it is for this to happen.