GCP Cloud Composer issue
Infrastructure Engineer with a Linux SysAdmin and SRE & DevOps background, previously a Google Cloud authorised trainer, who's excited and enthusiastic about Kubernetes, IaC, CI/CD, DevOps and SRE!
Experienced project infrastructure lead, project technical lead, and former Google Cloud Authorized trainer. Guiding organisations on cloud adoption, DevOps and SRE implementation. Mentor to junior engineers and people looking to change careers from a non-technical background or looking to get back into tech.
I'm passionate about building and deploying Cloud native infrastructure, automation, driving change and empowering people in learning and development.
I came across a very odd and aggravating issue when developing and testing a Google Cloud Composer Terraform module today.
It's definitely a Google Composer issue, not a Terraform issue. When updating a Cloud Composer environment, which causes a GKE cluster to be recreated, it fails.
Resource name projects/$PROJECT_ID/locations/europe-west2/environments/test-composer-dev
Error message Failed precondition (HTTP 400): Multiple errors occurred. Google Compute Engine: The subnetwork resource 'projects/$PROJECT_ID/regions/europe-west2/subnetworks/test' is already being used by 'projects/$PROJECT_ID/regions/europe-west2/nats/nat-rtr-Nat'. Could not configure workload identity because of another error Could not delete inverting proxy assignment because of another error
This is a private composer environment so I'm using Cloud NAT to allow egrees to the internet. It seems that Cloud NAT is using the subnet primary and secondary ranges that Cloud Composer creates for the GKE cluster, which then stops it from being able to update or destroy the environment, a race condition I guess.
To get around this I had to delete the Cloud NAT resource and then proceed with the change and/or deleting of the environment. Essentially freeing up cluster resources from the Cloud NAT resource that was attached to the subnet and IP ranges. Frustrating to say the least.
I don't have any experience with using or spinning up Cloud Composer before, from what I have read there are quite a few layers and resources which can cause clashes or issues I guess, there seem to be some "known issues" with composer.
Thought I would note this down, would be interesting to see or hear if anyone else has this issue or similar.
