I came across a very odd and aggravating issue when developing and testing a Google Cloud Composer Terraform module today.

It's definitely a Google Composer issue, not a Terraform issue. When updating a Cloud Composer environment, which causes a GKE cluster to be recreated, it fails.

Resource name projects/$PROJECT_ID/locations/europe-west2/environments/test-composer-dev

Error message Failed precondition (HTTP 400): Multiple errors occurred. Google Compute Engine: The subnetwork resource 'projects/$PROJECT_ID/regions/europe-west2/subnetworks/test' is already being used by 'projects/$PROJECT_ID/regions/europe-west2/nats/nat-rtr-Nat'. Could not configure workload identity because of another error Could not delete inverting proxy assignment because of another error

This is a private composer environment so I'm using Cloud NAT to allow egrees to the internet. It seems that Cloud NAT is using the subnet primary and secondary ranges that Cloud Composer creates for the GKE cluster, which then stops it from being able to update or destroy the environment, a race condition I guess.

To get around this I had to delete the Cloud NAT resource and then proceed with the change and/or deleting of the environment. Essentially freeing up cluster resources from the Cloud NAT resource that was attached to the subnet and IP ranges. Frustrating to say the least.

I don't have any experience with using or spinning up Cloud Composer before, from what I have read there are quite a few layers and resources which can cause clashes or issues I guess, there seem to be some "known issues" with composer.

Thought I would note this down, would be interesting to see or hear if anyone else has this issue or similar.