Git:Heroku Repository or object not found:
After alot of googling I found that the problem is with Heroku. If you are using Heroku, then Heroku doesn't support LFS and you have to look for alternatives.
I uploaded my long text file to dropbox and accessing it from there. Heroku should definitely look in to this issue.
git push heroku master --no-verify
This disables git-lfs pre-push hook. Metadata files are commited but binaries are not uploaded.
Then, as per upendra's answer, a buildpack can be used to download the files.