Composer auto scaling?

We currently don't support auto-scaling but it's on our roadmap. You could however manually scale up/down the GKE cluster by updating the nodeCount value.

I just published an article showing how to enable autoscale in its underlying kubernetes https://link.medium.com/AMUlwUIkD0.

Basically:

1. Enable autoscale on the node level

2. Apply HorizontalPodAutoscaler to airflow-worker deployment

3. Increase some airflow config parameter to remove the bottleneck

This has been tested in composer-1.7.2 and composer-1.7.5 but might be applicable to other version as well

Do check it out