Can't access Google Cloud Datastore from Google Kubernetes Engine cluster

So what I learnt in the process of debugging this issue was that:

• During the creation of a Kubernetes Cluster you can specify permissions for the GCE nodes that will be created.

• If you for example enable Datastore access on the cluster nodes during creation, you will be able to access Datastore directly from the Pods without having to set up anything else.

• If your cluster node permissions are disabled for most things (default settings) like mine were, you will need to create an appropriate Service Account for each application that wants to use a GCP resource like Datastore.

• Another alternative is to create a new node pool with the gcloud command, set the desired permission scopes and then migrate all deployments to the new node pool (rather tedious).

So at the end of the day I fixed the issue by creating a Service Account for my application, downloading the JSON authentication key, creating a Kubernetes secret which contains that key, and in the case of Datastore, I set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path of the mounted secret JSON key.

This way when my application starts, it checks if the GOOGLE_APPLICATION_CREDENTIALS variable is present, and authenticates Datastore API access based on the JSON key that the variable points to.

Deployment YAML snippet:

  ...
  containers:
  - image: foo
    name: foo
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /auth/credentials.json
    volumeMounts:
    - name: foo-service-account
      mountPath: "/auth"
      readOnly: true
  volumes:
  - name: foo-service-account
    secret:
      secretName: foo-service-account