Anyway, Cilium is something I looked into when I was studying for the CKS: Certified Kubernetes Security Certification and we’re using it at my new place, so it seems like a good time to get better hands on with it.

Not an intro….. Introduction

Now this isn’t meant to be an intro to Cilium, I’ll just say it’s an open source project that provides networking, security and observability for container orchestrations, Kubernetes etc.

But yeah, pretty much it’s much more feature rich cluster networking than the standard networking CNI offerings but also improves security and works using a new Linux kernel technology called eBPF, which enables the dynamic insertion of powerful security, visibility, and networking control logic into the Linux kernel.

eBPF is used to provide high-performance networking, multi-cluster and multi-cloud capabilities, advanced load balancing, transparent encryption, extensive network security capabilities, transparent observability, and much more.

In other words, traditional CNIs rely on the Linux kernel’s iptables or IPVS (IP Virtual Server)to shuffle packets around using basic IP addresses; Cilium uses eBPF to bypass these bottlenecks entirely.

Definitely not an intro….. And this blog post is definitely longer than I intended it to be, you know how it is when you get in the zone!

So what am I actually doing with Cilium?

So I’m using Cilium to network 2 separate Kubernetes Clusters so I can loadbalance and failover requests for workloads between them, achieving “Multi Cloud” Kubernetes. I want to have my workloads spread across 2 separate clusters, potentially in 2 different cloud providers, with requests being able to respond locally or from the other cluster.

Anyway, back to networking 2 clusters together…. I’m using Kind to spin up 2 clusters on my homelab Docker host (my Mac Pro ProxMox is maxed out and my larger Dell PowerEdge ProxMox server, which has my prod and test 4 node Kubeadm clusters on, is having hardware issues…).

You don’t have to configure 2 workers if you don’t have the resources. (I had to trim mine down to 1 worker each towards the end…)

For Cilium ClusterMesh to work, your podSubnet must not overlap across clusters, so make sure to configure them to be unique CIDRs.

# My "GCP" Kind cluster gcp-cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: gcp-cluster
networking:
  disableDefaultCNI: true
  podSubnet: "10.10.0.0/16" # Unique Pod Range
  serviceSubnet: "10.11.0.0/16"
nodes:
- role: control-plane
- role: worker
- role: worker

# My "AWS" Kind cluster aws-cluster.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: aws-cluster
networking:
  disableDefaultCNI: true
  podSubnet: "10.20.0.0/16" # Unique Pod Range
  serviceSubnet: "10.21.0.0/16"
nodes:
- role: control-plane
- role: worker
- role: worker
containerdConfigPatches:
- |
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

Workaround… I had some issues running all these on one Docker host. When it came to adding the second cluster with 3 nodes, I was getting errors.

I0125 16:24:56.130526 233 round_trippers.go:560] GET https://aws-cluster-control-plane:6443/api/v1/nodes/aws-cluster-worker?timeout=10s 404 Not Found in 3 milliseconds

So after some digging around, I had to add the containerd runtime config in the AWS cluster nodes containerdConfigPatches yaml to use the systemd cgroup driver. It appears the nodes weren’t joining and the nesting of containers and resources on a single Docker host proved too much. By adding SystemdCgroup = true It prevents the "double management" of resources. Without this, both systemd and the container runtime try to manage the same processes.

On my Ubuntu Docker host server, I had to increase inotify limits. I’m guessing this is because we have a bunch of the same or similar applications or processes watching the same directories on the Docker host.

# Increase inotify limits (KIND's recommendation for multi-cluster)
sudo sysctl fs.inotify.max_user_watches=524288
sudo sysctl fs.inotify.max_user_instances=512

Kind often flakes out when running 4+ nodes on a single Linux host, so with these settings, we should be good to go!

Creating and preparing the clusters

Spin the clusters up kind create cluster --config gcp-cluster.yaml and kind create cluster --config aws-cluster.yaml

And you should hopefully get…..

$ kubectl cluster-info --context kind-gcp-cluster
Kubernetes control plane is running at https://127.0.0.1:39765
CoreDNS is running at https://127.0.0.1:39765/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

$ kubectl cluster-info --context kind-aws-cluster
Kubernetes control plane is running at https://127.0.0.1:41725
CoreDNS is running at https://127.0.0.1:41725/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

$ kubectl get nodes
NAME                        STATUS     ROLES           AGE     VERSION
aws-cluster-control-plane   NotReady   control-plane   2m45s   v1.32.2
aws-cluster-worker          NotReady   <none>          2m30s   v1.32.2
aws-cluster-worker2         NotReady   <none>          2m30s   v1.32.2
ferris@micro-ubuntu:~/Documents/kind_cilium_clusters$ kubectl get nodes --context kind-gcp-cluster
NAME                        STATUS     ROLES           AGE   VERSION
gcp-cluster-control-plane   NotReady   control-plane   28m   v1.32.2
gcp-cluster-worker          NotReady   <none>          28m   v1.32.2
gcp-cluster-worker2         NotReady   <none>          28m   v1.32.2

Now, if you've created a cluster before, you’ve seen this before… Don’t panic!

We configured the clusters without any CNI because we’re going to be using Cilium!

Installing Cilium

Using Helm, I’ll install Cilium:

helm repo add cilium https://helm.cilium.io/
helm repo update

# Install on GCP first
helm install cilium cilium/cilium --version 1.18.6 \
  --namespace kube-system \
  --set cluster.name=gcp-cluster \
  --set cluster.id=1 \
  --set ipam.mode=kubernetes \
  --set operator.replicas=1 \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=$(kubectl get nodes --context kind-gcp-cluster -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}') \
  --set k8sServicePort=6443 \
  --kube-context kind-gcp-cluster

When that’s done, we need to configure some certificate secrets for the second cluster. Cilium ClusterMesh won’t trust another Cilium node with different certificates from one cluster to another.

kubectl get secret -n kube-system cilium-ca -o yaml --context kind-gcp-cluster > cilium-ca.yaml

# Delete the context-specific metadata so we can apply it to AWS
sed -i '/resourceVersion/d;/uid/d;/creationTimestamp/d;/namespace/d' cilium-ca.yaml

# Apply it to AWS
kubectl apply -f cilium-ca.yaml -n kube-system --context kind-aws-cluster

I ran into a Helm issue when installing for the second AWS cluster:

Error: INSTALLATION FAILED: Unable to continue with install: Secret "cilium-ca" in namespace "kube-system" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "kube-system"

Turns out it was a Helm "ownership" conflict. Because I manually applied the cilium-ca Secret using kubectlHelm is refusing to take control of it because it's missing the labels and annotations that say, "This belongs to the Cilium Helm Chart."

Since we want the Secret to be there (it's the shared key that makes the mesh work), we need to "adopt" it into Helm's management.

kubectl annotate secret cilium-ca -n kube-system \
  meta.helm.sh/release-name=cilium \
  meta.helm.sh/release-namespace=kube-system \
  --context kind-aws-cluster

secret/cilium-ca annotated

kubectl label secret cilium-ca -n kube-system \
  app.kubernetes.io/managed-by=Helm \
  --context kind-aws-cluster

secret/cilium-ca not labeled

So let’s try again:

helm install cilium cilium/cilium --version 1.18.6 \
  --namespace kube-system \
  --set cluster.name=aws-cluster \
  --set cluster.id=2 \
  --set ipam.mode=kubernetes \
  --set operator.replicas=1 \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=$(kubectl get nodes --context kind-aws-cluster -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}') \
  --set k8sServicePort=6443 \
  --kube-context kind-aws-cluster

These configurations for Helm are essentially saying single replicas for the operator because this is Kind, it’s a small cluster, so we will replace the kubeproxy with Cilium by configuring with kubeProxyReplacement=true.

In a multi-cluster world, Cilium identifies every Pod using a combination of its Namespace, Labels, and a Cluster ID. If both clusters have the same ID, Cilium's eBPF maps will collide, and you'll get "IP already exists" or routing loop issues.

Traditional firewalls use IP addresses. Cilium uses Security Identities. When a packet leaves Cluster A, Cilium attaches a numerical identity to it. Cluster B checks its local BPF map for that ID, not the IP, to decide if the traffic is allowed. This is why the unique cluster.id is so important!

Make sure you install Cilium CLI (Probably should have done this first :shrug)

Enable cluster mesh on both clusters.

cilium clustermesh enable --context kind-gcp-cluster --service-type NodePort
cilium clustermesh enable --context kind-aws-cluster --service-type NodePort

Then connect them.

cilium clustermesh connect --context kind-gcp-cluster --destination-context kind-aws-cluster

And verify….

cilium clustermesh status --context kind-gcp-cluster
⚠️  Service type NodePort detected! Service may fail when nodes are removed from the cluster!
✅ Service "clustermesh-apiserver" of type "NodePort" found
✅ Cluster access information is available:
  - 172.18.0.2:32379
✅ Deployment clustermesh-apiserver is ready
ℹ️  KVStoreMesh is disabled

✅ All 3 nodes are connected to all clusters [min:1 / avg:1.0 / max:1]

🔌 Cluster Connections:
  - aws-cluster: 3/3 configured, 3/3 connected

Looking good!

Testing it out

Deploy a workload to cluster 1.

Something like the echoserver, so we can test later. Here’s a snippet of what I used; change it how and what you want.

kubectl --context kind-gcp-cluster create deployment backend --image=k8s.gcr.io/echoserver:1.10
kubectl --context kind-gcp-cluster scale deployment backend --replicas=2

And a workload to cluster 2.

kubectl --context kind-aws-cluster create deployment backend --image=k8s.gcr.io/echoserver:1.10
kubectl --context kind-aws-cluster scale deployment backend --replicas=2

Then we need the global service.

apiVersion: v1
kind: Service
metadata:
  name: backend
  annotations:
    service.cilium.io/global: "true"
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: backend

Global Services must exist in both clusters with the exact same name/namespace for the mesh to "link" them. Apply to both clusters, then we test the functionality:

Deploy a tester pod to cluster 1.

kubectl --context kind-gcp-cluster run tester --image=alpine --restart=Never -- /bin/sh -c "apk add curl && sleep 3600"

Then we curl the service from the tester pod and see what and where the responses are:

kubectl --context kind-gcp-cluster exec tester -- sh -c "for i in \$(seq 1 10); do curl -s backend | grep 'Hostname'; done"
Hostname: backend-5974d998f8-47ddk
Hostname: backend-5974d998f8-bd5wg
Hostname: backend-5974d998f8-7brr9
Hostname: backend-5974d998f8-fcrcw
Hostname: backend-5974d998f8-47ddk
Hostname: backend-5974d998f8-fcrcw
Hostname: backend-5974d998f8-7brr9
Hostname: backend-5974d998f8-7brr9
Hostname: backend-5974d998f8-7brr9
Hostname: backend-5974d998f8-fcrcw

kubectl --context kind-gcp-cluster get pods
NAME                       READY   STATUS    RESTARTS   AGE
backend-5974d998f8-47ddk   1/1     Running   0          6m27s
backend-5974d998f8-7brr9   1/1     Running   0          6m27s
tester                     1/1     Running   0          43s
kubectl --context kind-aws-cluster get pods
NAME                       READY   STATUS    RESTARTS   AGE
backend-5974d998f8-bd5wg   1/1     Running   0          6m30s
backend-5974d998f8-fcrcw   1/1     Running   0          6m29s

We see results from backend pods from both clusters.

To make this clearer, I edited my deployments to include the cluster type/name in the pod’s name to make it more obvious

kubectl --context kind-gcp-cluster exec tester -- sh -c "for i in \$(seq 1 20); do curl -s backend | grep 'Hostname'; done"
Hostname: aws-backend-5c69d5f85-vxwx9
Hostname: aws-backend-5c69d5f85-vxwx9
Hostname: aws-backend-5c69d5f85-7f68c
Hostname: gcp-backend-57d9f789dd-hmqs6
Hostname: gcp-backend-57d9f789dd-hmqs6
Hostname: gcp-backend-57d9f789dd-hmqs6
Hostname: gcp-backend-57d9f789dd-dbbsp
Hostname: gcp-backend-57d9f789dd-dbbsp
Hostname: gcp-backend-57d9f789dd-dbbsp
Hostname: aws-backend-5c69d5f85-7f68c
Hostname: aws-backend-5c69d5f85-7f68c
Hostname: aws-backend-5c69d5f85-7f68c
Hostname: gcp-backend-57d9f789dd-dbbsp
Hostname: gcp-backend-57d9f789dd-hmqs6
Hostname: gcp-backend-57d9f789dd-hmqs6
Hostname: gcp-backend-57d9f789dd-dbbsp
Hostname: aws-backend-5c69d5f85-7f68c
Hostname: aws-backend-5c69d5f85-vxwx9
Hostname: gcp-backend-57d9f789dd-dbbsp
Hostname: aws-backend-5c69d5f85-vxwx9

kubectl --context kind-gcp-cluster get pods
NAME                           READY   STATUS    RESTARTS   AGE
gcp-backend-57d9f789dd-dbbsp   1/1     Running   0          4m47s
gcp-backend-57d9f789dd-hmqs6   1/1     Running   0          4m47s
tester                         1/1     Running   0          2m59s
kubectl --context kind-aws-cluster get pods
NAME                          READY   STATUS    RESTARTS   AGE
aws-backend-5c69d5f85-7f68c   1/1     Running   0          5m9s
aws-backend-5c69d5f85-vxwx9   1/1     Running   0          5m9s

We created the service with an annotation service.cilium.io/global: "true" This can be configured for other behaviours. Here are some examples and use cases I found interesting:

I’ll add the affinity local annotation to my service that I added to both clusters earlier and reapply it to both clusters again to reconfigure:

apiVersion: v1
kind: Service
metadata:
  name: backend
  annotations:
    service.cilium.io/affinity: "local" ## Failover if the local pods are down/unavailable
    service.cilium.io/global: "true"
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: backend

kubectl exec tester --context kind-gcp-cluster -- sh -c "for i in \$(seq 1 5); do curl -s backend | grep 'Hostname'; done"

Hostname: gcp-backend-57d9f789dd-8nwss
Hostname: gcp-backend-57d9f789dd-s7kp5
Hostname: gcp-backend-57d9f789dd-8nwss
Hostname: gcp-backend-57d9f789dd-s7kp5
Hostname: gcp-backend-57d9f789dd-8nwss

The responses are staying local to the cluster as preferred, configured with the service annotation.

If I simulate an outage on the gcp-cluster and run the tester pod again, which is on the gcp-cluster:

kubectl --context kind-gcp-cluster scale deployment gcp-backend --replicas 0
deployment.apps/gcp-backend scaled

kubectl --context kind-gcp-cluster get pods
NAME     READY   STATUS    RESTARTS   AGE
tester   1/1     Running   0          56s

kubectl exec tester --context kind-gcp-cluster -- sh -c "for i in \$(seq 1 5); do curl -s backend | grep 'Hostname'; done"
Hostname: aws-backend-5c69d5f85-vxwx9
Hostname: aws-backend-5c69d5f85-7f68c
Hostname: aws-backend-5c69d5f85-vxwx9
Hostname: aws-backend-5c69d5f85-7f68c
Hostname: aws-backend-5c69d5f85-vxwx9

# Once again for good measure!
kubectl exec tester --context kind-gcp-cluster -- sh -c "for i in \$(seq 1 5); do curl -s backend | grep 'Hostname'; done"
Hostname: aws-backend-5c69d5f85-7f68c
Hostname: aws-backend-5c69d5f85-vxwx9
Hostname: aws-backend-5c69d5f85-7f68c
Hostname: aws-backend-5c69d5f85-vxwx9
Hostname: aws-backend-5c69d5f85-7f68c

Then bring it back up the gcp-backend pods and back in action.

kubectl --context kind-gcp-cluster scale deployment gcp-backend --replicas 2
deployment.apps/gcp-backend scaled

kubectl --context kind-gcp-cluster get pods
NAME                           READY   STATUS    RESTARTS   AGE
gcp-backend-57d9f789dd-49x4p   1/1     Running   0          4s
gcp-backend-57d9f789dd-9dhvg   1/1     Running   0          4s
tester                         1/1     Running   0          2m46s

kubectl exec tester --context kind-gcp-cluster -- sh -c "for i in \$(seq 1 5); do curl -s backend | grep 'Hostname'; done"
Hostname: gcp-backend-57d9f789dd-49x4p
Hostname: gcp-backend-57d9f789dd-49x4p
Hostname: gcp-backend-57d9f789dd-9dhvg
Hostname: gcp-backend-57d9f789dd-49x4p
Hostname: gcp-backend-57d9f789dd-49x4p

We’ve just configured and tested Kubernetes cluster failover!

Observability with Hubble

To see what’s going on with Cilium, we can use Hubble.

You can enable Hubble with the Cilium CLI or with Helm.

Running an aws-tester pod and running a curl to its own aws-backend pods, seeing as it’s set to prefer local, and I have scaled down the gcp-backend pods. Give us more information on the traffic and requests:

Summary

So there you have it! Cilium is much more than an updated network policy controller or Kubernetes firewall configuration.

We installed Cilium on 2 Kind clusters and enabled ClusterMesh!

Bonus Round!

Enjoy this fine piece of AI generated bonus stage image. I was losing the will trying to prompt it to be more like the Street Fighter II bonus stage…

As a “Thank You” for reading this far (if indeed, you still are….) Here’s some b-b-bonus material!

We’re going to see the real advantage of Cilium network policy in action with ClusterMesh!

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "simple-path-blocker"
spec:
  endpointSelector:
    matchLabels:
      run: tester
  egress:
    - toEndpoints:
        - matchLabels:
            "k8s:io.kubernetes.pod.namespace": kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*"
    - toEndpoints:
        - matchLabels:
            app: backend
      toPorts:
        - ports:
            - port: "80"
              protocol: TCP
            - port: "8080"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/public.*"

curling the backend pods from the tester pod (This will work if the tester pod is running in either cluster, as both backend pod deployments are labelled app: backend):

kubectl exec tester --context kind-gcp-cluster --   curl -v http://backend/admin

  % Total    % Received % Xferd  Average Speed  Time    Time    Time   Current
                                 Dload  Upload  Total   Spent   Left   Speed
  0      0   0      0   0      0      0      0                              0* Host backend:80 was resolved.
* IPv6: (none)
* IPv4: 10.11.57.143
*   Trying 10.11.57.143:80...
* Established connection to backend (10.11.57.143 port 80) from 10.10.1.123 port 50356 
* using HTTP/1.x
Access denied
> GET /admin HTTP/1.1
> Host: backend
> User-Agent: curl/8.18.0
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 403 Forbidden
< content-length: 15
< content-type: text/plain
< date: Mon, 02 Feb 2026 21:33:07 GMT
< server: envoy
< 
{ [15 bytes data]
100     15 100     15   0      0   1479      0                              0
* Connection #0 to host backend:80 left intact

kubectl exec tester --context kind-gcp-cluster --   curl -v http://backend/public

  % Total    % Received % Xferd  Average Speed  Time    Time    Time   Current
                                 Dload  Upload  Total   Spent   Left   Speed
  0      0   0      0   0      0      0      0                              0* Host backend:80 was resolved.
* IPv6: (none)
* IPv4: 10.11.57.143
*   Trying 10.11.57.143:80...


Hostname: gcp-backend-57d9f789dd-4jcng

Pod Information:
    -no pod information available-

Server values:
    server_version=nginx: 1.13.3 - lua: 10008

Request Information:
    client_address=10.10.1.134
    method=GET
    real path=/public
    query=
    request_version=1.1
    request_scheme=http
    request_uri=http://backend:8080/public

Request Headers:
    accept=*/*
    host=backend
    user-agent=curl/8.18.0
    x-envoy-expected-rq-timeout-ms=3600000
    x-envoy-internal=true
    x-forwarded-proto=http
    x-request-id=f42e5b24-41aa-4da5-a755-023ddcfdb7d9

Request Body:
    -no body in request-

* Established connection to backend (10.11.57.143 port 80) from 10.10.1.123 port 39048 
* using HTTP/1.x
> GET /public HTTP/1.1
> Host: backend
> User-Agent: curl/8.18.0
> Accept: */*
> 
* Request completely sent off
< HTTP/1.1 200 OK
< date: Mon, 02 Feb 2026 21:33:13 GMT
< content-type: text/plain
< server: envoy
< x-envoy-upstream-service-time: 1
< transfer-encoding: chunked
< 
{ [577 bytes data]
100    565   0    565   0      0  18802      0                              0
* Connection #0 to host backend:80 left intact

So to summarise what happened here:

• Evidence of the Proxy: In both outputs, from curling the backend /admin and /public We can see < server: envoy. This proves that traffic is no longer just "flowing" through the network; it is being actively intercepted and inspected by the Cilium-managed Envoy proxy.

• The "Secret" Headers: Notice the x-envoy-internal: true and x-request-id headers in the /public output. These are injected by the proxy and are great visual aids to show that "the network is now intelligent." Or more intelligent at least….

• The L7 Block: In the /admin request, we can see Access denied followed by the 403 Forbidden. Because the "server" is still envoy It proves the block happened at the Network Policy level, not because the application itself rejected you.

How Cilium L7 network policies work

With the typical regular Kube network policy, it’s quite broad; it’s either ports 80 and 8080 that are open to the labels we configure. With the Cilium network policy, we can define those ports that are available, but we can also configure a particular operation to a URL path. /public and the implied deny takes care of requests to /admin.

Cilium uses eBPF to intercept the packet at the virtual Ethernet interface. Cilium then identifies the source as run: tester and the destination as app: backend based on security identities rather than just IP addresses.

Cilium’s eBPF sees that an L7 HTTP policy is configured and active for this destination. Instead of forwarding the packet to the network, eBPF redirects the traffic to a local Envoy proxy listener on the node. This is transparent Injection, meaning we don’t have to make any changes to our client applications.

Envoy terminates the TCP connection, parses the HTTP headers and checks the requested path e.g. /public and /admin as we tried, against the configured and active CiliumNetworkPolicy.

What we then see is Success with /public. Envoy sees /public is whitelisted, initiates a new connection to the backend pod on port 8080 and injects those "secret" headers like x-request-id.

And we see a Block on /admin Envoy sees /admin is NOT whitelisted. It immediately generates an HTTP 403 Forbidden response with the server: envoy header and sends it back to the tester pod.

By using eBPF to transparently redirect traffic to Envoy, Cilium gives us 'Service Mesh' capabilities, like fine-grained HTTP control—without the overhead of managing sidecar containers in every pod.

I promise this is actually the end now…

Again, thank you for reading and following along, if indeed, you still are!

I’d been meaning to do a deeper dive into Cilum and find trying something as interesting as a multi cluster mesh more interesting than “Pod 1 can or can’t talk to pods 2 and 3….“ And writing about topics like this really motivates me to learn this fundamentally (What’s more motivating than not looking publicly stupid).

That being said, I am open to any comments, questions or a heads-up if I have missed anything, gotten the wrong end of the stick on something, something doesn’t work or what I should look at next!

Disclaimer (Can you tell I was a consultant!)

I’ve tried to be as accurate and factual as I can with Cilium and Kubernetes, etc. I am learning this and writing guides, hows to’s etc helps me grasp the “why” on new topics!

To be transparent, I used AI to help me troubleshoot and peer review some parts. I have fact checked and actually tested all this code and config.

This is by no means production ready (It’s literally Kind on a micro PC!) Your mileage may vary! Don’t go exposing your server, workloads etc to the internet.

Links to helpful documentation

Kind quickstart

Kind troubleshooting and common errors

Cilium quick installation

Cilium Helm installation

Cilium clustermesh setup

Cilium Getting Started with the Star Wars Demo (A great demo for an introduction to Cilium network policies)

Hubble set up

What is eBPF

I like both; both do a great job at reconciliation of your Kubernetes deployments and both can be at the core of your GitOps journey (maybe not both together, but you get what I mean….pick one).

I'll be diving deeper into both over the next few months, starting now with this deep dive into Argo CD.

Currently, I have a Kubernetes cluster running on VMs running on a Proxmox server I use for my homelab.

I am running Argo CD on it as my deployment platform of choice. So I’ll run through some more of using Argo CD in a bit more depth.

Authentication to a private Git repository

First, we won’t always be deploying from public facing Github repos, nor will we want to have to expose our company or personal repos to the rest of the world, so let’s take a look at deploying from private GitHub repos.

I’ll be using the official Argo CD docs to help get set up with authenticating to my private GitHub repo.

First, I’ll set up a quick and easy application with some Kubernetes manifest files, just a basic 2-tier frontend and backend deployments and some services., importantly, in a private repo. Just to test Argo CD deploying from a private repo.

There are a few methods we can use to authenticate to GitHub from Argo: Username and password/access token, TLS, SSH key and GitHub app credentials.

I’m not going to go through all of them right now, we’ll just settle on just creating an access token, as this is the simplest way to get started in my opinion, the GitHub app credential is a bit involved and more suited to something in production, so a simple single access token for the repo will be fine for developing for now.

Select ‘developer settings‘ from your GitHub settings page, then personal access tokens, under that “Fine-grained“ tokens.

Give your token a name, description, make sure “resource owner” is set to the owner of the repo, set your expiration, select your repos that Argo will have access to and the permissions. I’ve set mine to “contents” read-only. Then generate your token.

Security Disclaimer!

This should go without saying….DO NOT share this token, upload this token to Git or anywhere public.

This GitHub personal access token is the key to your personal and private repo. Even if your permissions are read-only, that can change and whoever has your token will have a key to your repo.

If you suspect your key has been compromised or uploaded to anything public, like accidentally pushed to a public repo (it happens, just don’t let it happen to you!), revoke it straight away from the same settings page. PSA disclaimer done!

Create the repository resource in Argo CD using declarative or using the admin interface. I’m using declarative YAML to apply to the cluster here.

apiVersion: v1
kind: Secret
metadata:
  name: demo-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: git
  url: https://github.com/my-github/my-argoc-demo-repo
---
apiVersion: v1
kind: Secret
metadata:
  name: private-repo-creds
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repo-creds
stringData:
  type: git
  url: https://github.com/my-github/
  password: github_pat_12345_key_here_do_not_upload_to_git!!!!
  username: your_username

The first secret demo-app adds the specific repository we want Argo CD to track. The second secret is a template for how Argo CD can authenticate with my GitHub account using the Personal Access Token (password field).

Crucially, remember that the namespace for these Argo CD objects has to be argocd (or whatever namespace Argo CD is deployed to).

Once applied, check your Argo CD UI to confirm it's connected successfully.

Credential Templates

That may have seemed like a fair effort to get just one repo auth’d with Argo, chances are you’ll be working with and deploying from multiple repos; that’s where credential templates come in. These set the general authentication method for the repos in your account that you want Argo to work with.

App deployment

Let’s try it out, add a new application in Argo CD using the admin console or declarative manifest YAML files. I’ll show you what the console looks like, as it is pretty helpful and nice to look at when it’s applying!

Give it a name, add it to the default project and tick any of the options you might want.

Add your source repo, point it at the correct branch, cluster (local cluster Argo CD is running on for me currently)and the target namespace.

In no time at all, the application has auto-synced as I configured and deployed my Kubernetes manifest YAML files.

Now, when any changes are made to my private repo, for example, the number of pods I have for the backend, Argo CD will reconcile.

5 mins later…

GitOps at its finest from a private GitHub repo!

Psst!

You probably don’t want to be creating all your Argo CD applications with the GUI console, so here’s some declarative YAML manifest to do the same thing.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-private-k8s
  namespace: argocd
spec:
  destination:
    namespace: argo-demo-app ## Dont forget to add your target namspace!
    server: https://kubernetes.default.svc
  project: default
  source:
    path: .
    repoURL: https://github.com/my-github/your_repo_url_here
    targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

App of apps

So we’re now applying our applications to Argo CD, but might be thinking “But I’m still manually creating the Argo CD application?“.

Feels like we’re kicking the automation can down the road? This is where the App of Apps approach comes in to help us enable the automation of creating our applications.

Wouldn’t it be nice to bootstrap those clusters with the core or common deployments to get them up and running? RBAC? Namespaces? Istio or Prometheus, let’s say, for example.

You can deploy an app in Argo CD that consists of all your …. Apps! The app of apps approach.

I've created a new private repository called argocd-app-of-apps with an apps directory inside it. This apps directory will hold the YAML manifests for all my sub-applications.

I will also need to configure the Argo CD to deploy from this new GitHub repo, like we did before with the demo-private-k8s repo. You can do this via the console or just add it to the repo manifest file and apply again.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app-of-apps
  namespace: argocd
spec:
  destination:
    namespace: argocd
    server: https://kubernetes.default.svc
  project: default
  source:
    path: apps
    repoURL: https://github.com/my-github/argocd-app-of-apps
    targetRevision: HEAD
  syncPolicy:
    automated:
      enabled: true
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

And in that repo is an apps directory where the application YAML manifests, which are referred to in path.

I’ve been playing with Bitnami’s Sealed Secrets recently, so I want that on my cluster and potentially any other clusters I have too, so I’ll add a sealed-secrets.yaml to the apps directory.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sealed-secrets
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io/foreground
spec:
  project: default
  source:
    repoURL: https://bitnami-labs.github.io/sealed-secrets
    chart: sealed-secrets
    targetRevision: '2.17.7'
    helm:
      parameters:
        - name: installCRDs
          value: 'true'
  destination:
    namespace: kube-system
    server: https://kubernetes.default.svc
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=true

I’ll add the Argo CD guestbook example app there too, just for demonstration.

Push to the new app-of-apps repo and now apply the app-of-apps.yaml manifest file to the cluster; this should be a one-time thing that a cluster admin would do.

kubectl apply -f app-of-apps.yaml

Now the argocd-app-of-apps is created in Argo CD and that itself is now deploying the YAML manifests from the apps directory in the GitHub repo

Guestbook and sealed-secrets are now deployed onto the cluster. Let’s go one further and add another; after all, we don’t want to be adding new repos manually or imperatively.

I’ll add a demo-private-repo.yaml file to the apps directory:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo-private-k8s
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io/foreground
spec:
  project: default
  source:
    repoURL: https://github.com/my-github/argo-private-application
    targetRevision: HEAD
    path: .
  destination:
    server: https://kubernetes.default.svc
    namespace: argo-demo-app
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

And there we have applications we can use to bootstrap and deploy to clusters in a GitOps way.

Hopefully, these are some helpful ideas and ways of getting the most out of your Argo CD deployments.

Multi-cluster deployment

With all the apps we’re deploying, it’s doubtful you’ll only be deploying to a single cluster.

Chances are, you might want to test on a cluster or have a pre-prod or staging cluster? Argo CD also lets you deploy to other clusters.

To add a cluster to Argo CD using the argocd CLI, I grabbed the kube config file from my 2nd cluster, test-cluster and ran this: argocd cluster add test-cluster --kubeconfig=.kube/test-config.

You can do this declaratively using Kubernetes secrets. You can run kubectl -n argocd get secrets and see your cluster as a secret that Argo CD uses to deploy to.

A quick test, I’ll deploy the demo-private-k8s application to the new test-cluster. I’ve done this via the console and on the cluster drop-down pointed it to my test-cluster.

Let’s check the cluster for the new pods:

Here’s the Argo application YAML manifest, you’ll need to be careful and make sure it is named differently from the already deployed application; otherwise, it will overwrite the existing demo-private-k8s application (that absolutely happened to me and I took me too long to realise…..)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-demo-private-k8s
  namespace: argocd
spec:
  destination:
    namespace: argo-demo-app
    server: https://192.168.1.106:6443
  project: default
  source:
    path: .
    repoURL: https://github.com/my-github/argo-private-application
    targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

You could apply this with your app-of-apps repo; it would be a fair bit of copy paste if you wanted to have your manifest files deployed to 2,3, etc clusters.

I’ve seen examples of using Kustomize, which helps template Kubernetes deployments so you can change variables for deployments to multiple clusters in multiple environments using the same core deployment manifests code.

It uses base code and overlays for specifying specific variables like labels, replicas etc would be different per cluster and/or environment.

I won’t get into it now, it’s something I’m learning about and this looks to be a great use case for Kustomize, maybe I’ll do a write up on Kustomize?!

Best practices and security

We’ve covered some really interesting concepts to advance the deployment of our Kubernetes deployments using Argo CD.

This section includes some best practices and some security thoughts and considerations.

Projects: Principle of Least Privilege

You may have noticed the Default project when deploying. Argo CD Projects are logical groupings that create clear trust and security boundaries.

Not all applications should be available to everyone. By creating projects for different teams or environments (staging, production), you can restrict:

• Which source repos can be deployed.

• Which destination clusters/namespaces can be deployed to.

• What Kubernetes resource kinds can be created (e.g., deny the creation of ClusterRoles).

This is key to implementing the Principle of Least Privilege.

Here is an example AppProject manifest:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: exmaple-project
  namespace: argocd
  # Finalizer that ensures that project is not deleted until it is not referenced by any application
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  # Project description
  description: Example Project

  # Allow manifests to deploy from any Git repos
  sourceRepos:
  - 'https://github.com/my-github/argo-private-application'

  # Only permit applications to deploy to the 'guestbook' namespace or any namespace starting with 'guestbook-' in the same cluster
  # Destination clusters can be identified by 'server', 'name', or both.
  destinations:
  - namespace: testing-projects-scope
    server: https://192.168.1.106:6443
    name: test-cluster

  # Deny all cluster-scoped resources from being created, except for Namespace
  clusterResourceWhitelist:
  - group: ''
    kind: Namespace

Let’s try an application deployment. I’ll try something to the correct cluster and from the expected Git repo, restricted to the testing-project-scope I configured in the project manifest above:

Now I’ll try deploying the Argo Guestbook application to that same cluster and the correct namespace using the Argo CD example-project project.

The application was not deployed as the example-project is not permitted to deploy from the Argo GitHub repo, as configured in the example-project I created.

To wrap this up, here are a few other essential security and advanced topics to consider as you deepen your Argo CD usage:

• Argo CD RBAC: Add users/teams to your projects, limiting who can perform actions like syncing, deleting, or rolling back applications.

• Secret Management: Implement a solution like the Sealed Secrets I hinted at earlier, or use HashiCorp Vault to securely store the sensitive information (like the GitHub tokens) that Argo CD needs.

• Sync Hooks: Look into Argo CD's sync phases and hooks to define custom actions (e.g., running database migrations) that must occur before or after the main deployment is applied.

Final Thoughts and Next Steps

We've successfully moved from a basic Argo CD install to a scalable, automated, and secure deployment platform capable of:

1. Handling Private Repositories securely.

2. Bootstrapping new clusters using the App of Apps pattern.

3. Deploying to Multiple Clusters.

4. Enforcing organisational and security policies using Projects.

Thanks for sticking with me, this was a bit of a long one! I was aiming for something people could follow along with, so I hope you enjoyed this journey!

Leave a comment if you think I missed something, got it wrong, or have anything to add or I should check out!

Sorry, that’s quite misleading! However, I’ve been experimenting with Kagent on my local Kubernetes development cluster, and it’s really neat!

I could have done with something like this during my on-call SRE days!

Kubernetes without the kubectl!

My old colleague and fellow Kubernetes enthusiast Chris Matcham suitably inspired me to give this a try. Colour me intrigued!

I followed Chris’ guide here now. I didn’t have any credit on OpenAI ChatGPT, so I wanted to try it using Google’s Gemini as my AI backend

Not a quick start-ish

So I’m not going to go over installing Kagent, Chris’ post already does a great job of that. Use Helm, use ArgoCD, however you like.

As I’m planning on using Google Gemini as my AI backend I will add what I did differently to help get you started.

First, I manually added a Kubernetes Secret in my kagent-agent namespace. I need to add my API key as a secret. So you can head over to Google AI Studio to get yourself an API key and associate it with a Google Cloud Project. I had a personal Google Cloud project I was already using for some personal dev things so was ready to go.

Then create your secret.

I then opted to install using ArgoCD to deploy. I had it running anyway. I just used the UI for testing purposes, just make sure to install the Kagent CRDs first.

Head over to aistudio.google.com and hit create API key and associate it with a Google Cloud project you have access to, then add some values to create the Gemini provider as part of the Helm deployment:

We’ll set up ArgoCD like this:

We’re telling Helm to include the Kubernetes secret I created manually that has the API key I just created.

To create the application in ArgoCD, I could have created a declarative yaml file (Chris does a great job of this in his blog post), but as I’m just tinkering with this, I’m opting for the UI.

The repo URL I’m using is ghcr.io/kagent-dev/kagent/helm and the charts for the CRDs are kagent-crds version: 0.3.15 and I’m using the same Helm repo URL and the chart name kagent for the agent deployment using version 0.3.15

Syncronise the ArgoCD application and I should have 2 running applications, 1 for the CRDs and another for the actual agent

I’m using Rancher Desktop for my local cluster, so with a quick port forward of the agent service resource and I can now access my Kagent deployment. Let’s head over to the models to make sure my changes to configure the model to Gemini worked:

Let’s open a chat and take it for a spin!

I created a deployment manually using the terminal, so let’s test if Kagent can see what’s running on the cluster:

Nice! I can interact with the cluster without using kubectl!

Let’s try something else, let’s create a pod

kubectl run something-wrong --image ngiinx:latest

Let’s see if it spots it…

It’s noticed that there is an incorrectly configured pod!

Before I can even ask, it’s fixed the incorrect image.

What is interesting is that I’ve had different outcomes from this type of issue I’ve asked to look into.

Sometimes it’s given me the “I’ve found you’ve configured the image Nginx which doesn’t look right, you should correct it“ to it correcting it, not telling me, then when I ask “Whats wrong with my pod?“ it starts going diving into Kubelet and networking possibilities because another pod is working so its probably not that type thing.

In case no one’s noticed, I’m a big Simpsons and Futurama fan…

I have tried this by creating an agent that only has read access on the cluster, so it can’t make changes.

If you're implementing using GitOps, ArgoCD etc, then that might be for the best, and using Kagnet as a troubleshooting tool.

Imagine waking up at 3 am thanks to Pager duty and you’ve got an AI assistant to summarise logs and spot things that don’t look right? Game changer! What I would have given for this type of thing when I was an on-call SRE getting alerts at 2 am!

AI Agents are the new hot jam, I’m really keen to investigate and try this out a bit more as it’s quite new to me, potentially have something like this running for my Homelab to help troubleshoot etc.

My good pal Chris does a great job getting you started with his walkthrough here. I just tried it out with a Google Cloud Gemini twist.

You can find out more about Kagent in more detail and its thriving community at their website.

For those of you who have been reading my blog posts and following along, you might have guessed a bit of a theme... If you haven't guessed, I've been on a real learning journey to learn and get my hands on as much Kubernetes as I can.

This is a write-up of my approach to the final 2 exams for me, the KCSA and the CKS.

Opening thoughts on certifications as part of development

As a goal driven individual, I learn and develop best when there is an end goal to reach for and aspire to (There is no real end to this but don’t tell my brain that…)

In my case, the Kubestronaut Certification, one blue jacket to rule them all!

Which is why certifications matter to me. There is a bit of a debate with certs: “certifications don’t maketh the engineer”, which is right, they don’t, but they are a great way to validate skills and experience.

So I get both sides of the coin.

I’ve met and worked with some amazing DevOps, Data engineers, and developers who don’t have a cert to their name and never will; they don’t see the point in it.

I’ve also interviewed some people who had 5 Google Cloud certs but couldn’t explain a simple architecture from a design they clearly “liberated” from the internet and had clearly exam dumped to pass certs without the experience to back it up.

So, I get it.

For me, I want to learn the real fundamentals of a thing, so with Kubernetes Security, I refreshed my Docker skills and went back to basics, breaking and fixing clusters before diving into the security topics of the Certified Kubernetes Security Specialist.

Sorry for the long ramble of an intro, I was a cloud trainer for 2 and a half years, and it’s questions and thoughts I got asked about a lot, so I thought I’d add my 2 pence worth.

TL;DR certs are good, but not the be-all and end-all; experience and fundamental learning absolutely matter.

The Kubernetes and Cloud Security Associate (KCSA)

While I was preparing and learning the technical security skills for the CKS, I'll add my thoughts after this. I wanted to get the KCSA out of the way, so here's my take and how I approached it.

It's a multiple-choice exam and focuses more on fundamental and overview understanding of cloud-native security with Kubernetes and container-based design at the forefront. Ergo, it's a nice introductory cert into Kubernetes Cloud Native Security.

This exam is an online, proctored, multiple-choice exam.

If you’ve been in the industry long enough and have some experience with a cloud-native culture and tools, chances are you probably have some fundamental knowledge already in the topic areas of Overview of Cloud Native Security, Platform Security, and Compliance and Security Frameworks.

Compliance and Threat Modelling

I did find myself brushing up on some of the compliance frameworks, as it had been a while since I had read about any of them. I read up on what they are and what they cover: NIST, CIS, GDPR, and PCI DSS. Google is your friend for that.

I also read up on some threat modelling frameworks, STRIDE and DREAD. These were also mentioned in the exam I took.

Cluster component security

The slightly trickier areas I found I had to think about a bit more were Kubernetes Cluster Component Security, Kubernetes Security Fundamentals, and Kubernetes Threat Model.

The cluster component security and security fundamentals question areas focused on the components themselves, which are considered best practices that you apply to protect secrets around etcd, for example.

In my exam experience, there were definitely some questions I had to flag and return to, some real head-scratchers. It’s not an “easy” exam (there aren’t any), but it is an “easier” exam. I passed the first time with a decent score in the low 90s, I think. I wasn’t expecting a score that high!

Links I used for studying:

Kode Kloud KCSA course

Kubernetes Threat Modelling

Kubernetes STRIDE, DREAD & PASTA threat models

Now for the main event! Certified Kubernetes Security Specialist (CKS)

Chat GPT is getting better at drawing me pictures for my blogs

I had already passed the CKA and the CKAD. I wrote about how I prepped for the CKA and creating a cluster to practice.

In a previous life, I was a Linux SysAdmin, and I use Linux as part of my day job; it’s a fundamental skill. As I tell people who are just getting started, make sure they have grasped basic Linux skills, networking, and containers before getting into Kubernetes.

I found the jump felt smaller when I first started learning about Kubernetes years ago because I had the fundamental skills mentioned. Learning about the abstraction, context, and importantly, the “why” was much easier with those fundamental skill sets and experience in my toolbelt.

That certainly helped with the CKA and CKAD, but it is not a guaranteed pass for the CKS. This is a very tough exam, and I did not pass on my first try!

Which, at the time, stung, but it helped me focus on what I was good at and what I needed to improve on #FailureisnotFinal

CKS Exam overview

Some information about the exam:

Make sure you check out the full information on the domains and competencies on the official CKS exam page.

It’s a performance based test that requires solving multiple tasks from a command line running Kubernetes. You have 2 hours to complete the tasks.

I’m not aiming to give any direct, exact questions I got in my exam, that’s not the point of this post and it’s pretty much NDA’d and frowned upon for me to disclose that.

My focus areas. Yours may differ….

These are some areas that were new to me, needed a refresh and needed my extra focus:

• Falco (I wrote about it here) Know where to look for logs, how to update rules and find pods based on alerts firing.

• A real cheeky one, I thought at the time. I had an Istio task which does appear under Minimize Microservice Vulnerabilities domain, I’m just really glad I did a deep dive (shameless plug of blog post here)

• Bom/SBom - Tools like Trivy and the Bom utility by the Kubernetes project. Learn how to use them, you will get tasks related to and using these tools, which covers the Supply Chain Security Domain part of the exam.

• Remember updating a Cluster from your CKA? You should still remember how to do that…..

• Know about etcd and how to make configuration changes to it regarding secrets and security best practices.

• Get well practised at making changes to the kube-api-server (implementing logging, plugins and security changes) and practice getting fast and accurate at it. This helped me a lot during my re-take. I was too slow the first time around.

• Know about AppArmor and how to implement profiles on nodes and pods.

• Know about Seccomp and how it restricts which system calls are allowed.

• Practice Network Policies. While you're at it, go practice Cilium Network policies too. Network policy editor is a great tool; try it out.

• Speaking of speed, the 2 hours pass by incredibly quickly! In both attempts, I flagged 3 out of the 16 tasks that I didn't even try. I simply ran out of time.

Time is not on your side

That said, manage your time and tasks accordingly. Flag the questions you won’t answer right away or those you anticipate requiring more time and effort.

I tried to answer the smaller/easier/preferred tasks first (there were no easy questions on the CKS, only easier and personally preferred).

Then I returned to the bigger and harder tasks. I was working right up to the last second. Time is the real test of this exam.

I passed on my second attempt after failing the first time with a score of 57. I had a good understanding of what I needed to improve, which was generally completing tasks more quickly and focusing on areas like Falco, making changes to the etcd configuration, refreshing my knowledge of working with Service Accounts and Tokens, and refactoring deployments to run in a restricted namespace with Pod Security Admission.

My closing thoughts

I didn’t feel hard done by failing my first attempt, it felt hard and I didn’t feel like I knew it all enough, a bit like when you die in Dark Souls, you didn’t die because the game is unfair, just like I didn’t fail my first exam because the exam was unfair.

It’s because I wasn’t good enough (This is why I don’t play Souls games….)

I passed 2nd time around on the nose, with a very efficient 67. My failures helped me get to the point that I passed.

My advice….. Practice. Just practice for speed and accuracy. There are some great scenarios over on killercoda’s Killer Shell. I went through these a lot until I could get through them with minimal documentation help (I spent too much time searching the docs on my first attempt).

I worked and studied every day leading up to the CKS, just practising scenarios and reading. I learn best in small, frequent sessions rather than once per week for a long time. But we’re all different. Little and often is a real-time commitment on top of full-time project work and being a dad of two (with a very supportive and understanding wife!).

Sorry, this one turned into a bit of a long one. This was a real journey, and I'm super happy to be a part of the Kubestronaut program. Passing all 5 certification exams was a real test of skill and experience, and I really enjoyed the learning process.

#FailureisnotFinal #PracticeMakesPermanent

Study resources I found useful:

• I shouldn’t have to tell you, but kubernetes.io and get good and searching for what you need quickly!

• It might be obvious, but check the official CKS exam page for all the domains and competencies

• Kode Kloud - The CKS course is quite comprehensive and has some great practice labs

• Kode Kloud - Challenge labs

• Killer Coda - Killer Shell CKS scenarios

• Kubernetes CKS Full course on YouTube

• Network Policy Editor online tool

• Make use of the Exam simulator with the exam purchase, you get 2 36 hour sessions. Use them!

Let me know what you think of certifications, Kubernetes or the Kubestronaut programme!

I’ve had a couple of people message me via LinkedIn asking about learning paths and development, so don’t be a stranger!

As you can probably tell from a lot of my previous posts, I’ve been having a lot of fun with Kubernetes, I’m currently trying my hand at the Certified Kubernetes Security Specialist, known as the most challenging Kubernetes exam of them all.

Which is why I’m going to do a bit of a quick start write up getting started with installing and using Falco on GKE.

What is Falco?

Falco is a cloud native security tool that provides runtime security across hosts, containers, Kubernetes, and cloud environments. It is designed to detect and alert on abnormal behaviour and potential security threats in real-time.

It’s essentially a real time monitoring tool that alerts against preconfigured rules and custom rules configured by us administrators.

Falco deploys with some preconfigured rules that check the Linux kernel for any unusual behaviour, including but a few:

• Privilege escalation using privileged containers

• Executing shell binaries such as sh, bash, csh, zsh, etc

• Executing SSH binaries such as ssh, scp, sftp, etc

• Read/Writes to well-known directories such as /etc, /usr/bin, /usr/sbin, etc

This is a brief overview. You can find more info on what Falco is and why at the Falco docs site.

Installing Falco on GKE

Falco is fairly easy to get started with, you can install it on a VM, compute instance or Kubernetes Cluster. You’ll find a quick start style tutorial on the Falco Getting Started page.

I’ve gone for a GKE cluster. I had one already up and running, so I’ve opted to install Falco on the cluster using Helm. You can use a Linux VM or any other type of cloud managed Kubernetes provider.

kubectl create ns falco

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

helm install falco \
-n falco \
--set tty=true \
--set driver.kind=ebpf \
falcosecurity/falco

I’ve used the —set driver.kind set to ebpf as I’m using in a GKE cluster and I’m not able to load a kernel module on my GKE cluster.

The Helm chart will then deploy Falco as a DaemonSet, meaning that a pod running Falco will run on each node of the cluster.

Testing Alerting

Let’s try out Falco and see the pre-configured rules in action!

Let’s spin up a pod and have it do something to trigger a rule.

kubectl run pod test --image nginx

Let’s have the test pod do something “unusual”.

We can check the logs on the Falco pods.

kubectl -n falco logs -c falco -l app.kubernetes.io/name=falco

I’ve cut off the top of the logs so you can see the logs from the commands I just ran:

There’s a lot of info there, but what we’re looking for is timestamps, the pod that was alerted on and what the pod to alert.

10:14:56.930581844: Notice A shell was spawned in a container with an attached terminal (evt_type=execve user=root user_uid=0 user_loginuid=-1 process=sh proc_exepath=/usr/bin/dash parent=containerd-shim command=sh -c ls -la terminal=34816 exe_flags=EXE_WRITABLE|EXE_LOWER_LAYER container_id=9f03a0d2a1ed container_image=docker.io/library/nginx container_image_tag=latest container_name=test k8s_ns=default k8s_pod_name=test)
10:15:10.218050374: Notice A shell was spawned in a container with an attached terminal (evt_type=execve user=root user_uid=0 user_loginuid=-1 process=sh proc_exepath=/usr/bin/dash parent=containerd-shim command=sh -c ls -la /root terminal=34816 exe_flags=EXE_WRITABLE|EXE_LOWER_LAYER container_id=9f03a0d2a1ed container_image=docker.io/library/nginx container_image_tag=latest container_name=test k8s_ns=default k8s_pod_name=test)

For example, the logs we have above the very last part tell us the container and pod:

The start is the message “Notice A shell was spawned in a container with an attached terminal“ (We’ll use this later!).

But look into the log message more, and we can see what actually triggered the alert:

Essentially, the container “test” in the pod named “test” ran the “ls -la” and in another instance “ls -la /root“.

It's not ideal! That could signify a bad actor trying to work their way around a pod and potentially gaining access to the underlying node and our wider infrastructure. So its a good thing we have an rule triggering this event!

Logging and alerting Falco in Google Cloud Monitoring

Now with this running. In GKE, wouldn’t it be nice not to have to remember to go trawling through the logs everynow and then just to know what’s happening in our cluster? The answer is yes!

Google Cloud comes ready with a very comprehensive logging and monitoring suite of tools that we can make use of to alert on the content of a log message. GKE integrates with Google Cloud Logging out of the box so we should definitely make use of it, let’s take a look how.

Falco logs are output using stdout, so with the logging capabilities of GKE, they appear in Google Cloud Logging from the “Workloads” page in the GKE cluster. I can choose “Falco” and look at the logs produced from the pods:

What’s neat is that you can fine tune your Logging query to find the logs you are interested in right now, clicking the “View in Logs Explorer“.

Now in Logs Explorer, we can see the LQL query and make some changes to find the log content we’re interested in:

resource.type="k8s_container"
resource.labels.project_id="gcp-project-id"
resource.labels.location="europe-west2-b"
resource.labels.cluster_name="cks-cluster"
resource.labels.namespace_name="falco"
labels.k8s-pod/app_kubernetes_io/instance="falco"
labels.k8s-pod/app_kubernetes_io/name="falco" severity>=DEFAULT
textPayload:"Notice A shell was spawned in a container with an attached terminal"

In the last hour, we can see the logs (plus another I did!) that Falco alerted on from shell sessions spawned in pods on the cluster, reducing the noise to specify the testPayload to search for logs with the following testPayload “Notice A shell was spawned in a container with an attached terminal".

Much easier to find what we’re looking for!

With this log query, we can also create a log based alert, super simple! Click the “Actions” button and then “Create log alert“.

Give the alert a name, it will grab our log query and we set the frequency and the channel to notify.

Let’s fire off some more Kubectl exec and create some alerts……

That was quick!

Let’s look at the actual incident that’s been created.

In the incident, you can view the log/s that caused the alert and you can pop out to the Google Logs Explorer page again.

And the email alert got sent, so if I wasn’t staring at the monitoring dashboards, I certainly know about it now!

The alert also doesn’t have to be an email, Google CLoud Monitoring has a choice of notification channels, could be Google Chat, Pager Duty for the really important alerts, or PubSub for something completely different.

As a former oncall SRE, all I ask is that you alert and wake up engineers for something serious that they need to be present for!

Creating custom rules

Now, lets say we have some scenarios which are not covered by the rules that come configured in Falco? Thats where custom rules come in. We can create our rules for Falco to alert on.

The default Falco configuration will load rules from /etc/falco/falco_rules.yaml, /etc/falco/falco_rules.local.yaml and /etc/falco/rules.d.

My current deployment of Falco (and yours if your following along…) was done via Helm, so it’s a case of creating a custom rule yaml file and updating the Helm deployment.

I borrowed this from the Falco quick start to get going with some minor changes:

customRules:
  custom-rules.yaml: |-
    - rule: Write into etc
      desc: An attempt to write to the /etc directory
      condition: >
        (evt.type in (open,openat,openat2) and evt.is_open_write=true and fd.typechar='f' and fd.num>=0)
        and fd.name startswith /etc
      output: "Stop what your doing and look at this!! File below /etc opened for writing (file=%fd.name pcmdline=%proc.pcmdline gparent=%proc.aname[2] ggparent=%proc.aname[3] gggparent=%proc.aname[4] evt_type=%evt.type user=%user.name user_uid=%user.uid user_loginuid=%user.loginuid process=%proc.name proc_exepath=%proc.exepath parent=%proc.pname command=%proc.cmdline terminal=%proc.tty %container.info)"
      priority: WARNING
      tags: [filesystem, mitre_persistence]

As a brief overview of the rule we’re creating here, in the condition, we’re stating that the event type of the syscall open, openat or openat2 and the event type Macro used is opened for writing (evt.is_open_write equaling true), which starts with /etc .

The output section then determines what is logged out to what we see in the Falco pod logging and also in this case, Google Cloud Logging. You can use event fields for the outputs to make things easier to interpolate rather than hardcoding. The documentation is quite comprehensive.

Finally tags, these are optional but handy for organising rules into categories. Example, you could you could tell Falco to skip all rules with a particular stage in a dev environment. Tags are a handy way of controlling the rules in use

Now to update the Helm deployment to include the custom rules yaml file with the Falco deployment.

helm upgrade --namespace falco falco falcosecurity/falco --set tty=true -f custom_rules.yaml

Wait for the pods to restart.

Let’s trigger an alert by exec’ing into the test pod and try to write to /etc

Checking the logs in the Falco pod or in Google Cloud Logs Explorer shows the attempt:

Screenshots are hard to read….. Here’s a copy paste of the log that was output to Google Cloud Logging.

2025-05-12 11:14:47.152 BST

10:14:47.147029349: Warning Stop what your doing and look at this!! File below /etc opened for writing (file=/etc/test.txt pcmdline=sh -c touch /etc/test.txt gparent=containerd-shim ggparent=systemd gggparent=<NA> evt_type=openat user=root user_uid=0 user_loginuid=-1 process=touch proc_exepath=/usr/bin/touch parent=sh command=touch /etc/test.txt terminal=34817 container_id=71ee5d3c8123 container_image=docker.io/library/nginx container_image_tag=latest container_name=test k8s_ns=default k8s_pod_name=test)

You can see the custom message I added to output “Stop what your doing and look at this!!“

More info on creating custom rules here.

We’re just scratvhing the surface on making the Falco rules work for us, there are further more advanced things you can do that the documentation points to in more depth here, including how to override, add exceptions and writing your own rules etc.

Summary

Thats it for this quick intro into getting Falco up and running in a GKE Cluster.

To summarise, we had a GKE cluster up and running and we deployed Falco using helm. Any Kubernets Cluster or even a Linux VM should be fine to use. Your install/deployment method will vary.

Once running we tested the default built in rule engine by running a simple nginx pod and tried to sh exec commands reading and writing to root directories.

We then checked the Falco logs for alerts and then created a log based alert in Google Cloud Monitoring, to alert us when a rule had been triggered.

We then checked out custom rules.

I hope you found this useful, feel free to comment with any of your findings or thoughts!

Useful Links

Falco Quick start

Basic elements of Falco rules

Write your first custom rule

Default Macros

This continues from my previous blog post “Getting Started with Istio Service Mesh“.

We’ll be diving a bit deeper, specifically into Istio's architecture, traffic management, security, and observability features.

Recap

Let’s recap what a service mesh is and what Istio can help us with.

Istio is a service mesh that acts as an infrastructure layer for managing networking, security, and traffic control across microservices. It enables these capabilities without requiring any changes to the application code. Instead, Istio abstracts this functionality into a dedicated control plane and data plane, allowing teams to implement repeatable, complex configurations such as zero-trust security and advanced traffic management, independent of the applications running in their Kubernetes cluster or clusters.

Architecture

So what makes up Istio?

The Data plane

The Envoy proxy is typically injected into pods as a sidecar container. This sidecar container essentially takes over all the networking, working very closely with the other containers in the pod, all network calls in and out go via the Istio injected sidecar container. This is known as the Data Plane.

The Control Plane

The Control Plane part of this architecture is powered by the Istiod deployment in the istio-system namespace, which provides the service discovery, the configuration management and the certificate management.

Hopefully, this diagram helps

• Istiod (shown with the Istio logo at the bottom) is the control centre, storing and distributing configurations.

• Pods (e.g., App/Pod 1 and 2) contain both the application container and the Envoy sidecar proxy, injected at pod creation.

• All ingress and egress traffic flows through the Envoy proxies, where traffic management, authentication, and security policies are enforced based on configurations propagated from the Control Plane (Istiod).

With these core components running and working together, we now have a proxy layer which are the Envoy proxies containers running in the pods, injected at pod deployment they can then handle the traffic control features, failover and fault injection, security and authentication features like enforcing security policies and access control.

You can start to imagine if we wanted to introduce this and add these capabilities to our application code it would start to look very different and get more complex very quickly.

Instead, Istio decouples these concerns from the application itself, allowing teams to focus on business logic while Istio handles network, security, and observability at scale.

Traffic Management

Now we’ve had a bit of a recap of what Istio is and why it might be a good idea to introduce a decoupled network layer to configure, intercept and mediate our mesh network, let’s deep dive into just some of the traffic management capabilities it offers.

Let’s start at Gateways, after all, we probably want to learn how we get traffic and requests from outside the Kubernetes cluster to our applications and how we can control and configure them to behave how we want.

When you first install and run Istio on a cluster you can start with a demo profile which will create an Ingress Gateway and an Egress Gateway. They are deployed as Kubernetes objects and essentially act as load balancers (I’m working with Google’s GKE, so that will create a Google Cloud Load Balancer) for incoming and outgoing network requests at the outer edges of the cluster.

Here’s mine running on a GKE cluster.

The load balancer service it creates in GKE which is a network load balancer in Google Cloud which forwards the requests to the nodes in the GKE cluster.

This is created when Istio is installed with the default profile, I don’t have an istio-egressgateway, using kubectl describe on the service in the istio-system namespace gives us some more info about the load balancer. The istio=ingressgateway label is useful for when we create our Ingress Gateway resource which acts as a load balancer at the edge of our cluster.

I’ll create a Gateway object. This will configure and point to the Istio ingressgateway which we have above and sets up a proxy to configure the Istio Ingress Gateway and tell it where to send the traffic request.

We’ll configure the hosts for *. I added this as a wildcard so I can access it without a domain name because I haven’t gotten around to setting up the DNS yet (It’s Friday, what are you gonna do?! :shrug)

But if we wanted to access the ingress based on a domain, for example ferrishall.dev. I would add that to the hosts field in the yaml file and add the istio-ingressgateway external IP address as an A record in a domain's DNS settings.

This is a super simple gateway code snippet I used to create the Gateway object. I’ll list all the sources that helped me learn and write this up at the bottom of this post.

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - '*'

I was reading that there is support for both Istio Gateway and Kubernetes Gateway APIs and the Kubernetes Gateway API will soon be the default so I’ll probably have to re-learn and re-write this…. Documentation

I’ll spin up a hello-world application first so I have something to manage the traffic to.

The hello-world pod has 2 containers, 1 for the hello-world application and the 2nd for the Istio Envoy proxy sidecar.

Not shown, the hello-world container, but you can see just some of the istio-proxy container info when you run kubectl describe pod hello-world-xxxx.

So I’ve got an Ingress load balancer, a gateway and an application…. how do we tell Istio when it gets traffic from the ingress load balancer to route that request to hello-world? We need a Virtual Service.

Virtual Service

A Virtual Service is used to configure the actual routing rules to the backend services we want to send the traffic to, the hello-world service we just created for example.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: hello-world-vs
spec:
  hosts:
    - "*"
  gateways:
    - gateway
  http:
    - route:
        - destination:
            host: hello-world.default.svc.cluster.local
            port:
              number: 80

Again, I’m using the * in the hosts field because I haven’t set up DNS etc and we’re telling the virtual service that it is bound to the Gateway object called….. gateway and to please route requests to the backend service hello-world which I’m using the fully qualified domain name in the cluster to avoid any potential confusion with namespaces etc

Destination Rules & Virtual Services

Let’s try something else because that seems like a lot of work to get a hello-world application working!

I’ve deployed a web frontend and a customers backend application onto my cluster.

Now say there is a customers deployment version v2 which we want to test, we want to configure a percentage of some of the traffic to the v2 labelled pods to test some new features, ie we want to perform some canary testing.

We can create a Destination Rule for our service and define the two subsets representing v1 and v2 versions. A Destination Rule defines a policy that is applied to the traffic which is intended for a service but after the routing has taken place, rules can specify load balancing configuration. like ROUND_ROBIN, connection pool settings etc.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: customers-dr
spec:
  host: customers.default.svc.cluster.local
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2

This Destination Rule is telling Istio there are 2 versions of the customers service, which are labelled version: v1 and version: v2, known as subsets. Both pod versions are under the same customer ClusterIP service.

So when I create my Virtual Service to direct the traffic to my customer’s service, I can configure it with some rout weighting options.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: customers-vs
spec:
  hosts:
    - 'customers.default.svc.cluster.local'
  http:
    - route:
        - destination:
            host: customers.default.svc.cluster.local
            port:
              number: 80
            subset: v1
          weight: 50
        - destination:
            host: customers.default.svc.cluster.local
            port:
              number: 80
            subset: v2
          weight: 50

So now Istio knows to send 50% of traffic to version: v1 and the other 50% to version: v2 of the customer’s pods in the customers service.

We didn’t have to change any of the application code or Kubernetes deployment manifest files, it’s all configured and deployed with these Istio yaml manifest files and decoupled from the application code and Kubernetes manifest yaml files.

I’m just scratching the surface and there is a lot more that can be done like matching traffic requests to a service based on request header content, traffic mirroring etc.

I’m just trying to keep it simple and in context to what hopefully, the people reading this might find helpful and relatable. When you get playing with it you’ll find your own advanced use cases and content!

Security

Istio also helps us apply security to our distributed applications using mutual TLS.

Here is an image taken from my Kiali dashboard, to demonstrate I can access the customer service directly and I can also access via the gateway IP address which is managed by Istio.

Peer Authentication

What I’ve done here is reset some things, I deployed the web-frontend application without the Istio proxy injection and deployed the customers application which does have the Istio proxy injected. I also updated the virtual service to send traffic to the customers application via the istio-ingressgateway load balancer IP.

The unknown entry in the dashboard is the web-frontend application because it doesn’t have the Istio proxy injected, Istio doesn’t know what or where it is and importantly, doesn’t have the security lock symbol on the traffic, it’s not protected with mTLS.

Proxies send plain text traffic between services that do not have the sidecar injected.

I updated the virtual service for customers to point to the gateway and send a curl command with the customers service URI in the header, the Gateway object has an Istio proxy sidecar injected so it will send the request traffic using mTLS.

(In a nutshell Mutual TLS is where both the client and the server have to authenticate and verify there identity, with TLS its one way where the server’s identity is authenticated by the client. This article on mTLS explains the differences nicely.)

So how can we enforce mTLS security? We can apply peer authentication configuration.

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: default
spec:
  mtls:
    mode: STRICT

Now our applications, via the Istio Envoy proxy will only accept and transmit requests via mTLS connections but what does this mean for our web-frontend we deployed without the Istio proxy sidecar?….

Nope, not happening.

The current web-frontend hasn’t been deployed with the Istio proxy sidecar container injected into the pod so it’s trying to send requests to the customers service application pod via plain text. Istio configuration which has the PeerAuthentication configured to strict is saying no thanks, let’s redeploy the web-fontend with the Istio injection enabled….

That looks better!

Authorisation Policies

We can also control the flow of requests, just because all the pods in our cluster are deployed and managed by us does that mean they should have access to everything running the cluster? Does everything need access to that database? Or our customers application?

No of course not we want to be specific and use the principle of least privilege not just to users, service accounts and the permissions they have but to network connections and requests of the workloads running the in the cluster.

There’s where Authorisation Policies can help (I’m not writing Authorization, you can’t make me….)

We can configure what pods in which particular namespace and which principal (eg service account) is allowed or denied access to the pod or anything in the whole namespace right down to which operation they are allowed to perform on a particular path.

In our case, say we want to lock down access to the customers application to only accept requests from the web-frontend application and that is only allowed to come via the istio-ingressgateway load balancer.

First, we’ll flat out deny everything, giving us a bit of a clean slate.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
 name: deny-all
 namespace: default
spec:
  {}

Then we can apply the first authorisation policy.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-ingress-frontend
  namespace: default
spec:
  selector:
    matchLabels:
      app: web-frontend
  action: ALLOW
  rules:
    - from:
        - source:
            namespaces: ["istio-system"]
        - source:
            principals: ["cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"]

That covers the istio-ingressgateway authorised allowed to access the pods labelled app: web-frontend, we need the web-frontend to be allowed to access the customers application.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-web-frontend-customers
  namespace: default
spec:
  selector:
    matchLabels:
        app: customers
        version: v1
  action: ALLOW
  rules:
  - from:
    - source:
        namespaces: ["default"]
      source:
        principals: ["cluster.local/ns/default/sa/web-frontend"]

This covers the requests coming from web-frontend being allowed to access the customers pods which are labelled app: customers and the requests are coming from the default namespace and pods running as the web-frontend service account.

Let’s test that the deny rule is still working, I’ve just created a pod running a busybox curl image

This pod doesn’t have the expected principal or labels we specified in the Authorisation Policies so it’s getting RBAC: denied. Just as expected!

You might be thinking “Why not just use the network policy Kubernetes object?“ you absolutely can, but the key difference is network policies are layer 4 IP based network policies and the Istio authorisation policies are layer 7 so you can add HTTP header checks and other attributes. So you just get more granular options with Istio authorisation policies than network policies, all depends on your use case at the end of the day.

Again, I’m just scratching the surface here. There are a lot more security and Authorisation Policies you can adopt and deploy with Istio. More info you can find here

Observability

Finally, we come to observability. We’ve done some cool things with Istio but another feature it adds to is giving us insight using logs and metrics showing us what is actually happening within our distributed network.

Also, wouldn’t it be nice to measure the performance and alert us when performance starts degrading before it gets unacceptable to our users? That’s what we’ll take a look at in this final section of our deep dive.

Proxy Metrics

The Istio Envoy Proxies produce metrics and logs. Collecting valuable insight in the form of metrics about the traffic passing in and out of the proxies. Documentation here.

Service Metrics

Istio also provides metrics at the service level, the Istio addons via the GitHub repo supply some very handy default dashboards for visualising in Grafana.

Control Plane Metrics

Istio itself should also be monitored, so you can keep an eye on the performance and that it is behaving as expected, especially as you scale.

Kiali dashboard

Now who doesn’t love a good dashboard?! The Kiali dashboard is a nice way to get started and get some visual insight into your applications and the Istio proxies.

Grafana

You might be acquainted with Prometheus and Grafana so I won’t spend any time explaining what and why. You can find some default dashboards with the samples/addons directory of the Istio GitHub repo that provides visibility using the metrics collected from Prometheus.

Visualise the metrics for the services we have the Envoy Proxies injected into. The customers application metrics.

You’ll find a really nice set of addons in the Istio GitHub repo to help you get started with observability in Istio and the applications or services we are using and proxying with Istio.

I’ll leave observability there, there’s a lot more to discover but it’s important to take away that metrics, logs etc are all things that Istio adds. This is super handy as our applications and architecture get more complex, more distributed and start scaling, that we have as much insight into performance and will help us troubleshoot any issues that might arise.

The more data we have, the more informed technical and business decisions we can make!

Conclusion

Sorry, that was a bit longer than I was aiming for but you know how it is when you get into it.

In conclusion, Istio provides a robust and flexible service mesh solution that enhances the management of microservices in Kubernetes environments. By decoupling network, security, and observability concerns from application code, Istio allows development teams to focus on business logic while ensuring efficient traffic management, robust security through mutual TLS, and comprehensive observability.

The integration of tools like Kiali and Grafana further enriches the user experience and observability by providing valuable insights into service performance and network behaviour.

As you continue to explore Istio, especially in multi-cluster environments, you'll discover even more advanced capabilities that can further optimize your microservices architecture.

That’s all folks!

Hopefully, this was a handy deep dive into Istio, I’ve been learning and tinkering with Istio for the last few months and really having some fun with it (proper nerdy, I know…) writing about it helps me understand and commit to memory!

Hopefully, my words help explain some of the parts and use to beginners starting with Istio or just why would you use it in the first place.

I’m going to be taking my learning to multi cluster Istio service mesh, so hopefully I’ll write something up about that in the near future!

As always, really interested to hear what people’s thoughts are and what alternatives to try out. (Linkerd is next on my list) and if I got anything wrong please let me know!

Helpful links

Istio documentation

Getting started with Istio

Istio GitHub repository

Introduction to Istio Linux Foundation course (Free!)

Linux Foundation Intro to istio GitHub repository

Disclaimer: I have no affiliation with the Linux Foundation, this course or its authors, or claim to have created any of this code, but it is very helpful to use for learning this topic and recommend it very highly!

To carry on with my current learning and discovery with Kubernetes, I’ve written about creating a cluster the same guide that I used to get my homelab cluster running for practising with passing the CKS and CKAD certifications. I’ve also written and talked about GitOps with Argo CD. I even wrote about my trials and tribulations with Flux CD and Argo CD (Shameless plugs to previous articles over…).

So let’s check out networking! Everyone loves networking… when it’s working.

So why would networking with Kubernetes be a “thing” then? Surely the cluster sits on a network and just works?! Well, kind of…..

With Kubernetes, we have potentially broken down a monolith application where everything lived on a single box or VM and just contacted localhost:some_port or maybe another VM on the same network, maybe some firewall rules. We’ve now broken this down to different applications, which are now pods part of deployments, potentially multiple versions for testing, maybe even on different clusters.

What I’m trying to say is that we have introduced some added complexity for some benefits, which I won't go into as I’m sure if you're reading this, you know what they are. But we have to think about how all these distributed pods and applications interact or don’t interact with each other.

So what exactly is Istio?

Istio uses a proxy to intercept all your network traffic, allowing a broad set of application-aware features based on the configuration you set.

The control plane takes your desired configuration, and its view of the services, and dynamically programs the proxy servers, updating them as the rules or the environment changes.

The data plane is the communication between services. Without a service mesh, the network doesn’t understand the traffic being sent over, and can’t make any decisions based on what type of traffic it is, or who it is from or to. The data plane comprises Envoy sidecar proxies injected into application pods, handling actual traffic routing, security, and observability.

Seriously though, why?

The more our applications scale and potentially grow to additional deployments, stateful sets or just pods (please don’t just run pods…) they all need some networking or interaction with other deployments, pods etc.

If you're starting with a hello-world app that has a frontend and backend, maybe a database then of course a service mesh will be overkill, but think about if we introduce multiple versions of these deployments. Or maybe we add more services. Like some reviews for our website? Or an ad service? Does a shopping cart get added too? It starts to add up.

With a service mesh, we can abstract the networking configuration layer and decouple the networking away from the application, this keeps the application code more reusable and we can manage the networking without having to redeploy parts of the application.

With a service mesh networking is just part of it, and its security also. Just because all the pods live on the same cluster should they all be able to contact and have access to every pod on the cluster? No! Would you have every VM on your network have access to every other VM? I hope not, and the same should apply to your Kubernetes deployments.

Additional benefits to using a service mesh and decoupling network configuration

• Traffic management

  Istio allows users to control traffic flows and API calls between services by configuring rules and routing traffic. We can be quite granular in how and what our pods and applications can interact with on the network.

• Security

  Istio provides a backbone for communications and manages security controls. Adding an additional layer of security and adding to the principle of least privilege.

• Observability

  Istio can extract telemetry data from proxy containers and send it to a monitoring dashboard. As we can see with a Grafana dashboard we now get a huge amount of observablity into how our network is performing, more data gives us more insight allowing us to make not just better technical decisions and spot trends but also business decisions and trends too.

You can do some other neat things like fault injection like adding delays to test the resiliency of your configuration and traffic shaping.

These are more advanced features I’ll be looking at in the near future but they really add to the feature set to why you would look to add Istio service mesh to your infrastructure.

Hopefully, you can start to see why you introduce a service mesh. Having the network configuration decoupled from application logic, enabling fine-grained control and observability.

Sounds good! How do I get started?

Now we have an idea of why we might want to get started with a service mesh, let’s take a look at Istio. You’ll need a cluster to install it on, the rest is fairly straightforward.

I’m not going to regurgitate the Istio quickstart guide found here you can go there and go through the guides, it’s a good showcase on getting started and the sample application gives a good account of a distributed application where having a service mesh would be worth the time, effort and added complexity.

You install the Istioctl command line tool, then install Istio itself on the cluster, deploy the sample application and then enable istio on the namespace you are working with.

The Kiali dashboard helps visualise Istio and the configured applications, you can find it in the Istio repo in the samples/addons:

Grafana dashboard for the Istio services we are monitoring via Prometheus, we now get a ton of insight and observability into how our services are performing. Grafana and Prometheus deployments and config can be found in the samples/addons part of the repo:

So I followed the guide, what did I just do? How does Istio…. Istio?

What’s neat, is how Istio works. It’s essentially a proxy where the network configuration is enabled to the workloads/applications by injecting a sidecar container to the pods, here’s the productpage pod which shows the productpage container and the istio-proxy container as a sidecar:

Istio operates by injecting a sidecar proxy (Envoy) into each pod in your application. This proxy handles all incoming and outgoing traffic for the pod, allowing Istio to control and observe traffic without requiring changes to your application code. These sidecars work together under the direction of the Istio control plane, which manages configuration, policy enforcement, and telemetry.

Add a namespace label to instruct Istio to automatically inject Envoy sidecar proxies when you deploy your application later:

kubectl label namespace default istio-injection=enabled

For example, when you deploy the productpage Pod, Istio injects an Envoy proxy alongside the application container. This proxy intercepts traffic, applying Istio’s routing rules, security policies, and telemetry collection:

Pod: productpage
|-- Container: productpage
|-- Container: istio-proxy

This architecture ensures that network configuration remains decoupled from application logic, enabling fine-grained control and observability.

The first time I tried to work with Istio, nothing was happening…. I forgot to label the namespace I was working in! :face_palm

also, keep an eye out for any existing resource quotas and/or network policies in place that might stop Istio from working or behaving as expected.

k label namespace dev-three-tier istio-injection=enabled
k -n dev-three-tier scale deployments backend --replicas 0
k -n dev-three-tier scale deployments frontend --replicas 0
k -n dev-three-tier get deployments.apps
k -n dev-three-tier scale deployments backend --replicas 2
k -n dev-three-tier scale deployments frontend --replicas 1

Here I’ve just added the label to another namespace in my cluster dev-three-tier (haven’t added the db yet!) which is a very simple frontend backend and soon, DB. I then scaled down and scaled back the replicas in the deployment and Isto has now injected the sidecar proxy container into each of the pods.

Send some traffic to the frontend. while :; do curl -s http://192.168.1.43 ; sleep 1; done

I haven’t added a gateway or any rules yet but it’s a very simple app and Istio is probably overkill, I just wanted to show how you can add Istio to other workloads in your cluster quite simply.

I’m still learning about Istio but really enjoying seeing some real world context and use case, I wanted to use this article to go through how to get started, what I found useful and write some things down in the hope that this helps anyone, maybe demystify and explain in words and context that is more approachable.

As always, drop a comment if I got this wrong or missed something or if there is something else I should check out, I’m planning on checking out Linkerd in more detail but if anything else, I’m all ears!

Links:

Istio GitHub repo

Learning Microservices with Kubernetes and Istio

Istio Sidecar mode getting started

So it got me thinking, I should probably put some best practice tips on here. I'm not going to be writing about what a VM is and what the cloud is, I'm assuming most reading this have been there already.

So this is more of a "I've spun up a VM using default config, now what?" next step post. Well, this is where I'll add some pointers.

IAM and least privilege

So I'll start with Compute Engine, that's pretty much where everyone starts?! You may have noticed that when you create a Compute Instance, it'll come configured as a default to use the project's default Compute Engine service account, easy right?! Well yes but not overly best practice or secure.

Take a look at the IAM page and you'll notice that the default Compute Engine service account comes locked and loaded with the "Editor" role, again great what's the problem?! I can use that to do what I need and no more config is needed, awesome! Well, that is the problem!

The editor role is way too overpowered and broad for a VM that might just need access to a Cloud Storage bucket or maybe Cloud SQL.

To get into the habit of best practice we should be using IAM securely and only giving Google Cloud Identities like users and service accounts IAM roles to access services or do the jobs they need to, nothing more.

That way if our Compute Engine VM or the service account attached to it were to become comprised or mistakes during deployment happens, the blast radius is smaller and no unnecessary access can be leveraged.

Our Compute Engine VM, depending on what it is being used for, say for this example a web server. We would create a dedicated service account that has the IAM role roles/storage.objectAdmin this then allows the Compute Engine VM the permission to create, update, delete etc objects within Google Cloud Storage buckets.

Great! This ensures that the VM has no unnecessary IAM roles to do anything or have access to anything that we don't intend for it to do.

The practice of least privilege should be used for all IAM roles for identities and service accounts when creating Google Cloud resources, there's more info on this on the IAM Google doc which explains it nicely.

VPC Network

While we're at it with our Compute Engine VM, when you first spin one up you probably attached it to a network, right? the default VPC?

That's fine but we can level this up a bit by getting rid of the default VPC network and creating a custom VPC network with subnets in the region we're working in.

This controls what our subnet IP ranges are and what region we have our subnets created in. We can also get more control over the firewall rules and not have the default firewall rules available to use.

We really don't want to have firewalls that we don't intend on using and we don't want a "default-allow-ssh" firewall rule to all instances in our VPC that is available to 0.0.0.0.

Again, this all points to best practices and ensuring security by design, having those security considerations in mind when creating resources in Google Cloud.

I didn't include any step-by-step instructions on how to do these steps, it's too late in the evening, but it's always worth taking a look the next time you're in a Google Cloud project (just don't test in production!)

Please feel free to reach out if you have any thoughts or feedback, I'm always happy to get constructive feedback and to keep learning!