Deeper dive into Argo CD

You may have read my post from earlier this year “Flux CD Vs Argo CD“, where I took a comparison look at both.

I like both; both do a great job at reconciliation of your Kubernetes deployments and both can be at the core of your GitOps journey (maybe not both together, but you get what I mean….pick one).

I'll be diving deeper into both over the next few months, starting now with this deep dive into Argo CD.

Currently, I have a Kubernetes cluster running on VMs running on a Proxmox server I use for my homelab.

I am running Argo CD on it as my deployment platform of choice. So I’ll run through some more of using Argo CD in a bit more depth.

Authentication to a private Git repository

First, we won’t always be deploying from public facing Github repos, nor will we want to have to expose our company or personal repos to the rest of the world, so let’s take a look at deploying from private GitHub repos.

I’ll be using the official Argo CD docs to help get set up with authenticating to my private GitHub repo.

First, I’ll set up a quick and easy application with some Kubernetes manifest files, just a basic 2-tier frontend and backend deployments and some services., importantly, in a private repo. Just to test Argo CD deploying from a private repo.

There are a few methods we can use to authenticate to GitHub from Argo: Username and password/access token, TLS, SSH key and GitHub app credentials.

I’m not going to go through all of them right now, we’ll just settle on just creating an access token, as this is the simplest way to get started in my opinion, the GitHub app credential is a bit involved and more suited to something in production, so a simple single access token for the repo will be fine for developing for now.

Select ‘developer settings‘ from your GitHub settings page, then personal access tokens, under that “Fine-grained“ tokens.

Give your token a name, description, make sure “resource owner” is set to the owner of the repo, set your expiration, select your repos that Argo will have access to and the permissions. I’ve set mine to “contents” read-only. Then generate your token.

Security Disclaimer!

This should go without saying….DO NOT share this token, upload this token to Git or anywhere public.

This GitHub personal access token is the key to your personal and private repo. Even if your permissions are read-only, that can change and whoever has your token will have a key to your repo.

If you suspect your key has been compromised or uploaded to anything public, like accidentally pushed to a public repo (it happens, just don’t let it happen to you!), revoke it straight away from the same settings page. PSA disclaimer done!

Create the repository resource in Argo CD using declarative or using the admin interface. I’m using declarative YAML to apply to the cluster here.

apiVersion: v1
kind: Secret
metadata:
  name: demo-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: https://github.com/my-github/my-argoc-demo-repo
---
apiVersion: v1
kind: Secret
metadata:
  name: private-repo-creds
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repo-creds
stringData:
  type: git
  url: https://github.com/my-github/
  password: github_pat_12345_key_here_do_not_upload_to_git!!!!
  username: your_username

The first secret demo-app adds the specific repository we want Argo CD to track. The second secret is a template for how Argo CD can authenticate with my GitHub account using the Personal Access Token (password field).

Crucially, remember that the namespace for these Argo CD objects has to be argocd (or whatever namespace Argo CD is deployed to).

Once applied, check your Argo CD UI to confirm it's connected successfully.

Credential Templates

That may have seemed like a fair effort to get just one repo auth’d with Argo, chances are you’ll be working with and deploying from multiple repos; that’s where credential templates come in. These set the general authentication method for the repos in your account that you want Argo to work with.

App deployment

Let’s try it out, add a new application in Argo CD using the admin console or declarative manifest YAML files. I’ll show you what the console looks like, as it is pretty helpful and nice to look at when it’s applying!

Give it a name, add it to the default project and tick any of the options you might want.

Add your source repo, point it at the correct branch, cluster (local cluster Argo CD is running on for me currently)and the target namespace.

In no time at all, the application has auto-synced as I configured and deployed my Kubernetes manifest YAML files.

Now, when any changes are made to my private repo, for example, the number of pods I have for the backend, Argo CD will reconcile.

5 mins later…

GitOps at its finest from a private GitHub repo!

Psst!

You probably don’t want to be creating all your Argo CD applications with the GUI console, so here’s some declarative YAML manifest to do the same thing.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-private-k8s
  namespace: argocd
spec:
  destination:
    namespace: argo-demo-app ## Dont forget to add your target namspace!
    server: https://kubernetes.default.svc
  project: default
  source:
    path: .
    repoURL: https://github.com/my-github/your_repo_url_here
    targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

App of apps

So we’re now applying our applications to Argo CD, but might be thinking “But I’m still manually creating the Argo CD application?“.

Feels like we’re kicking the automation can down the road? This is where the App of Apps approach comes in to help us enable the automation of creating our applications.

Wouldn’t it be nice to bootstrap those clusters with the core or common deployments to get them up and running? RBAC? Namespaces? Istio or Prometheus, let’s say, for example.

You can deploy an app in Argo CD that consists of all your …. Apps! The app of apps approach.

I've created a new private repository called argocd-app-of-apps with an apps directory inside it. This apps directory will hold the YAML manifests for all my sub-applications.

I will also need to configure the Argo CD to deploy from this new GitHub repo, like we did before with the demo-private-k8s repo. You can do this via the console or just add it to the repo manifest file and apply again.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-of-apps
  namespace: argocd
spec:
  destination:
    namespace: argocd
    server: https://kubernetes.default.svc
  project: default
  source:
    path: apps
    repoURL: https://github.com/my-github/argocd-app-of-apps
    targetRevision: HEAD
  syncPolicy:
    automated:
      enabled: true
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

And in that repo is an apps directory where the application YAML manifests, which are referred to in path.

I’ve been playing with Bitnami’s Sealed Secrets recently, so I want that on my cluster and potentially any other clusters I have too, so I’ll add a sealed-secrets.yaml to the apps directory.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io/foreground
spec:
  project: default
  source:
    repoURL: https://bitnami-labs.github.io/sealed-secrets
    chart: sealed-secrets
    targetRevision: '2.17.7'
    helm:
      parameters:
        - name: installCRDs
          value: 'true'
  destination:
    namespace: kube-system
    server: https://kubernetes.default.svc
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=true

I’ll add the Argo CD guestbook example app there too, just for demonstration.

Push to the new app-of-apps repo and now apply the app-of-apps.yaml manifest file to the cluster; this should be a one-time thing that a cluster admin would do.

kubectl apply -f app-of-apps.yaml

Now the argocd-app-of-apps is created in Argo CD and that itself is now deploying the YAML manifests from the apps directory in the GitHub repo

Guestbook and sealed-secrets are now deployed onto the cluster. Let’s go one further and add another; after all, we don’t want to be adding new repos manually or imperatively.

I’ll add a demo-private-repo.yaml file to the apps directory:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-private-k8s
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io/foreground
spec:
  project: default
  source:
    repoURL: https://github.com/my-github/argo-private-application
    targetRevision: HEAD
    path: .
  destination:
    server: https://kubernetes.default.svc
    namespace: argo-demo-app
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

And there we have applications we can use to bootstrap and deploy to clusters in a GitOps way.

Hopefully, these are some helpful ideas and ways of getting the most out of your Argo CD deployments.

Multi-cluster deployment

With all the apps we’re deploying, it’s doubtful you’ll only be deploying to a single cluster.

Chances are, you might want to test on a cluster or have a pre-prod or staging cluster? Argo CD also lets you deploy to other clusters.

To add a cluster to Argo CD using the argocd CLI, I grabbed the kube config file from my 2nd cluster, test-cluster and ran this: argocd cluster add test-cluster --kubeconfig=.kube/test-config.

You can do this declaratively using Kubernetes secrets. You can run kubectl -n argocd get secrets and see your cluster as a secret that Argo CD uses to deploy to.

A quick test, I’ll deploy the demo-private-k8s application to the new test-cluster. I’ve done this via the console and on the cluster drop-down pointed it to my test-cluster.

Let’s check the cluster for the new pods:

Here’s the Argo application YAML manifest, you’ll need to be careful and make sure it is named differently from the already deployed application; otherwise, it will overwrite the existing demo-private-k8s application (that absolutely happened to me and I took me too long to realise…..)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-demo-private-k8s
  namespace: argocd
spec:
  destination:
    namespace: argo-demo-app
    server: https://192.168.1.106:6443
  project: default
  source:
    path: .
    repoURL: https://github.com/my-github/argo-private-application
    targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

You could apply this with your app-of-apps repo; it would be a fair bit of copy paste if you wanted to have your manifest files deployed to 2,3, etc clusters.

I’ve seen examples of using Kustomize, which helps template Kubernetes deployments so you can change variables for deployments to multiple clusters in multiple environments using the same core deployment manifests code.

It uses base code and overlays for specifying specific variables like labels, replicas etc would be different per cluster and/or environment.

I won’t get into it now, it’s something I’m learning about and this looks to be a great use case for Kustomize, maybe I’ll do a write up on Kustomize?!

Best practices and security

We’ve covered some really interesting concepts to advance the deployment of our Kubernetes deployments using Argo CD.

This section includes some best practices and some security thoughts and considerations.

Projects: Principle of Least Privilege

You may have noticed the Default project when deploying. Argo CD Projects are logical groupings that create clear trust and security boundaries.

Not all applications should be available to everyone. By creating projects for different teams or environments (staging, production), you can restrict:

• Which source repos can be deployed.

• Which destination clusters/namespaces can be deployed to.

• What Kubernetes resource kinds can be created (e.g., deny the creation of ClusterRoles).

This is key to implementing the Principle of Least Privilege.

Here is an example AppProject manifest:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: exmaple-project
  namespace: argocd
  # Finalizer that ensures that project is not deleted until it is not referenced by any application
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  # Project description
  description: Example Project

  # Allow manifests to deploy from any Git repos
  sourceRepos:
  - 'https://github.com/my-github/argo-private-application'

  # Only permit applications to deploy to the 'guestbook' namespace or any namespace starting with 'guestbook-' in the same cluster
  # Destination clusters can be identified by 'server', 'name', or both.
  destinations:
  - namespace: testing-projects-scope
    server: https://192.168.1.106:6443
    name: test-cluster

  # Deny all cluster-scoped resources from being created, except for Namespace
  clusterResourceWhitelist:
  - group: ''
    kind: Namespace

Let’s try an application deployment. I’ll try something to the correct cluster and from the expected Git repo, restricted to the testing-project-scope I configured in the project manifest above:

Now I’ll try deploying the Argo Guestbook application to that same cluster and the correct namespace using the Argo CD example-project project.

The application was not deployed as the example-project is not permitted to deploy from the Argo GitHub repo, as configured in the example-project I created.

To wrap this up, here are a few other essential security and advanced topics to consider as you deepen your Argo CD usage:

• Argo CD RBAC: Add users/teams to your projects, limiting who can perform actions like syncing, deleting, or rolling back applications.

• Secret Management: Implement a solution like the Sealed Secrets I hinted at earlier, or use HashiCorp Vault to securely store the sensitive information (like the GitHub tokens) that Argo CD needs.

• Sync Hooks: Look into Argo CD's sync phases and hooks to define custom actions (e.g., running database migrations) that must occur before or after the main deployment is applied.

Final Thoughts and Next Steps

We've successfully moved from a basic Argo CD install to a scalable, automated, and secure deployment platform capable of:

1. Handling Private Repositories securely.

2. Bootstrapping new clusters using the App of Apps pattern.

3. Deploying to Multiple Clusters.

4. Enforcing organisational and security policies using Projects.

Thanks for sticking with me, this was a bit of a long one! I was aiming for something people could follow along with, so I hope you enjoyed this journey!

Leave a comment if you think I missed something, got it wrong, or have anything to add or I should check out!