Drupal - Private files not found 404 error

Ok, I got this issue fixed.

The Vanish installed on server was causing the issue more specifically the following code.

# Always cache the following file types for all users.

    if (req.url ~ "(?i)\.(pdf|woff|eot|svg|ttf|png|gif|jpeg|ico|swf|css|js|html|htm)(\?[a-z0-9]+)?$") {
      unset req.http.Cookie;
      # Stop processing further for those file types. This way, we ensure to have
      # X-Varnish-Cache HIT for those requests.
      return (lookup);
    }

You can either remove the file extensions from above code or comment the entire block. This should solve the issue.