Google Cloud Storage with a service account in Java - 403 Caller does not have storage.objects.list access to bucket

TL;DR - If you're using Application Default Credentials (which BTW you are when you do StorageOptions.getDefaultInstance().getService();), and if you need to use the credentials from a service account, you can do so without changing your code. All you need to do is set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the full path of your service account json file and you are all set.

Longer version of the solution using Application Default Credentials

• Use your original code as-is

  Storage storage = StorageOptions.getDefaultInstance().getService();
  Bucket b = storage.get( "mybucketname" );
  

• Set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the full path of your json file containing the service account credentials.

  export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service_account_credentials.json
  

• Run your java application once again to verify that it is working as expected.

Alternate solution using hard-coded Service Account Credentials

The code example you posted for initializing ServiceAccountCredentials looks valid to me on a quick glance. I tried the following code snippet and it is working for me as expected.

String SERVICE_ACCOUNT_JSON_PATH = "/path/to/service_account_credentials.json";

Storage storage =
    StorageOptions.newBuilder()
        .setCredentials(
            ServiceAccountCredentials.fromStream(
                new FileInputStream(SERVICE_ACCOUNT_JSON_PATH)))
        .build()
        .getService();
Bucket b = storage.get("mybucketname");

When specifying a service account credential, the project ID is automatically picked up from the information present in the json file. So you do not have to specify it once again. I'm not entirely sure though if this is related to the issue you're observing.

Application Default Credentials

Here is the full documentation regarding Application Default Credentials explaining which credentials are picked up based on your environment.

How the Application Default Credentials work

You can get Application Default Credentials by making a single client library call. The credentials returned are determined by the environment the code is running in. Conditions are checked in the following order:

1. The environment variable GOOGLE_APPLICATION_CREDENTIALS is checked. If this variable is specified it should point to a file that defines the credentials. The simplest way to get a credential for this purpose is to create a Service account key in the Google API Console:

   a. Go to the API Console Credentials page.

   b. From the project drop-down, select your project.

   c. On the Credentials page, select the Create credentials drop-down, then select Service account key.

   d.From the Service account drop-down, select an existing service account or create a new one.

   e. For Key type, select the JSON key option, then select Create. The file automatically downloads to your computer.

   f. Put the *.json file you just downloaded in a directory of your choosing. This directory must be private (you can't let anyone get access to this), but accessible to your web server code.

   g. Set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of the JSON file downloaded.

2. If you have installed the Google Cloud SDK on your machine and have run the command gcloud auth application-default login, your identity can be used as a proxy to test code calling APIs from that machine.

3. If you are running in Google App Engine production, the built-in service account associated with the application will be used.

4. If you are running in Google Compute Engine production, the built-in service account associated with the virtual machine instance will be used.

5. If none of these conditions is true, an error will occur.

IAM roles

I would recommend going over the IAM permissions and the IAM roles available for Cloud Storage. These provide control at project and bucket level. In addition, you can use ACLs to control permissions at the object level within the bucket.

• If your use case involves just invoking storage.get(bucketName). This operation will require just storage.buckets.get permission and the best IAM role for just this permission is roles/storage.legacyObjectReader.

• If you also want to grant the service account permissions to get (storage.objects.get) and list (storage.objects.list) individual objects, then also add the role roles/storage.objectViewer to the service account.

Thanks to @Taxdude's long explanation, I understood that my Java code should be all right, and started looking at other possible reasons for the problem.

One of additional things I've tried were the permissions set to the service account, and there I've found the solution – it was unexpected, actually.

When a service account is created, it must not be given permissions to read from Google Storage, because then it will have read permissions to ALL buckets, and it is impossible to change that (not sure why), because the system marks these permissions as "inherited".

Therefore, you have to:

• Create a "blank" service account with no permissions, and
• Configure permissions from the bucket configuration

To do so:

• Open Google Cloud Web console
• Open Storage Browser
• Select your bucket
• Open the INFO PANEL with Permissions
• Add the service account with the Storage Object Viewer permission, but there are also permissions named Storage Legacy Object Reader and Storage Legacy Bucket Reader

Because of the word "Legacy" I thought those should not be used – they look like something kept for backward compatibility. And after experimenting and adding these "legacy" permissions, all of a sudden the same code I was trying all the time started working properly.

I'm still not entirely sure what is the minimal set of permissions I should assign to a service account, but at least now it works with all three "read" permissions on the bucket – two "legacy" and one "normal".