So It's not all about writing and shouting about all the cool stuff you've done or how smart you are. It's about writing about the times you've f*&%£d up and how you learnt from it.
So Friday, I completely nuked a GCP project IAM policy with Terraform and locked everything and everyone out!
Quite spectacular for a Friday morning, oh yeah I didn't mention this was on a Friday! :FacePalm
So how did I manage this?
I was testing some changes related to de-privileging an App Engine default service account, it automatically gets an editor role assigned which isn't great to have hanging around.
I'm currently using Terraform as the tool of choice for deploying infrastructure and Cloud Build for actually running the deployment.
I had tested this in a sandpit project, fairly new and just used a block of code similar to this:
resource "google_project_iam_policy" "project" {
project = "your-project-id"
policy_data = data.google_iam_policy.admin.policy_data
}
data "google_iam_policy" "admin" {
binding {
role = "roles/viewer"
members = [
"serviceAccount:default_appengine@googleserviceaccount.com",
]
}
}
Now, the mistake I spotted after applying this was this set the IAM policy for the entire project, not just the member referenced. Again completely my fault for not correctly reading the docs and the very clearly stated warning :
You can accidentally lock yourself out of your project using this resource. Deleting a google_project_iam_policy removes access from anyone without organization-level access to the project. Proceed with caution. It's not recommended to use google_project_iam_policy with your provider project to avoid locking yourself out, and it should generally only be used with projects fully managed by Terraform. If you do use this resource, it is recommended to import the policy before applying the change.
I mentioned I tested this, didn't I ?!
I did, In a project which was pretty clean and the test worked and I still had access, so what gives?!
Luckily the org my sandpit project was in had some well-thought-out permissions set on the folder where my project lives, so inheritance preserved my IAM permissions on the project, but as it was a clean project I overlooked the missing Google services service accounts that had been removed.
I thought it looked good and proceeded to apply my changes for real!
The build started and then suddenly failed and access was lost, I thought that was very coincidental and then to my horror realised what I had done.
Oh crap!
Essentially I had removed all IAMs from the project and replaced the whole project IAM policy with just a single viewer role on a dedicated user-managed service account intended for App engine.
Reading back through the documentation, it made perfect sense, in previous experience I had only ever used google_project_iam_member Terraform resource. which is non-authoritative.
Own it. Get help
Essentially, I'm writing this to highlight that bad stuff happens to most people and it all depends on how you deal with it.
I reached out for help once I realised I was locked out and had made a pretty big derp.
I got the help I needed and luckily got access back to the project that also luckily wasn't actually in production, I was preparing it for test use.
Sitting on it and stewing on it worrying about getting in trouble will never help the situation and remember, everyone has made mistakes before as long as no one dies and it wasn't intentional, most people will be understanding.
The team I'm working with had a bit of a laugh and it also made a good story where my other teammates told some of their war stories.
After I had access again, I then had to re-add the service accounts to the IAM service agent roles. Bit of a pain as was a lot of trial and error.
Some resources stopped working and took some troubleshooting to work out what was missing but I got there in the end.
But as that was the worst of it and took a couple of days to put what I'm hoping, most of it right again, I probably got away with it!
