gcsfuse Input/Output error

It appears from the Insufficient Permission errors in your debug output that gcsfuse doesn't have sufficient permissions to your bucket. Probably it has read-only access.

Be sure to read the credentials documentation for gcsfuse. In particular, if you're using a service account on a GCE VM make sure to set up the VM with the storage-full access scope.

You problem does stem from insufficient permissions, but you do not need to destroy and re-create the VM with a different scope to solve this problem. Here is another approach that is more suitable for production systems:

1. Create a service account
2. Create a key for the service account, and download the JSON file
3. Grant an appropriate role to the service account
4. Grant the appropriate permissions to the service account on the bucket
5. Upload the JSON credentials for the service account to the VM

Finally, define an environment variable that contains the path to the service account credentials when calling gcsfuse from the command line:

GOOGLE_APPLICATION_CREDENTIALS=/root/credentials/service_credential_file.json gcsfuse bucket_name /my/mount/point

Use the key_file option to accomplish the same thing in fstab. Both of these options are documented in the gcsfuse credentials documentation. (EDIT: this option is documented, but won't work for me.)

Interestingly, you need to use the environment variable or key_file option even if you have configured the service account on the VM using:

gcloud auth activate-service-account --key-file /root/credentials/service_credential_file.json

For some reason, gcsfuse ignores the active credentialed account.

Using the storage-full scope when creating a VM has security and stability implications, because it allows that VM to have full access to every bucket that belongs to the same project. Should your file storage server really be able to over-write the logs in a logging bucket, or read the database backups in another bucket?