Why?!

I've personally used and tinkered with Kubernetes and GKE for the last 3-4 years and in the last couple of years it has been more in a professional context, building some GKE clusters and Cloud Composer clusters that use GKE, but it was in the last couple of years I really wanted to learn about Kubernetes, not just how to use it and get by.

Now, there are loads of ways to get a Kubernetes cluster up and running, these days it's super simple with MicroK8s, MiniKube, and even spinning up a cluster using GKE (Google Cloud Kubernetes Engine) GCP's managed Kubernetes service.

I wanted to learn more about what Kubernetes is actually made of! What are the moving parts that make Kubernetes and how do they all work together?

There are some really good guides out in the wild, ranging from a complete DIY guide "Kubernetes the hard way" (I tried this a couple of years or so ago, it's really interesting and good fun, you can find Kelsey Hightower's repo here.) or running a Raspberry Pi Microk8s cluster A great way if you have the spare boards or even just one.

For me, time is a factor so I'll be using Kubeadm and some VMs to get a cluster up and running. You can think of Kubeadm as the packaged version of all the components that you need to create a cluster, it's also what the CKA & CKAD exams use.

I'm using this page to keep track of what I found worked for me after using a couple of guides to get my test cluster up and running.

Usual this isn't production-ready public disclaimer!

This by no means is a production-ready tutorial or even an opinion on a production-ready Kubernetes cluster, just what I found worked for me in getting a development cluster working in a fairly quick time using various articles and tutorials.

I wanted a cluster where I could practice upgrading, breaking and fixing to help me prepare for the Certified Kubernetes Administrator exam.

Pre-requisites

First, I created 3 VMs for my cluster, 1 control plane or master node, and 3 nodes for workloads. I created the VMs on my home lab hypervisor server, using ProxMox (Maybe I'll do a quick article on my home lab setup?!).

My VMs are installed with Ubuntu 22.04 OS and will prep and configure all with the following commands.

Steps for all VMs

Swap

First, we disable swap.

sudo swapoff -a

You'll need to edit the /etc/fstab swap or in my case swamp.img entry, just adding a "#" to comment out the line that mounts the swap should be fine. This will stop swap re-enabling at VM boot.

But wait, why does swap need to be disabled?!

Swap needs to be disabled for performance reasons and part of the scheduling that kube-scheduling performs to "score" and "pick' a node for a pod or workload, as they could potentially utilise the swap memory of a node..... or something like that.

Kubelet also failed to start and run correctly on my VMs until I disabled swap.

You'll need to do this for all VM control plane/s and worker nodes.

That being said swap support is being worked on apparently, I found some interesting info here about it, if you're interested.

Container runtime

Now, we configure Contained, the container runtime the cluster will be using.

sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF

Then run the modprobe commands to load some kernel modules

sudo modprobe overlay
sudo modprobe br_netfilter

The overlay module provides overlay filesystem support, Kubernetes needs this for its pod network abstraction.
The br_netfilter module enables bridge netfilter support in the Linux kernel, Kubernetes needs this for networking and policy.

We then need to add some configuration to the sysctl.d/kubernetes.conf file

sudo tee /etc/sysctl.d/kubernetes.conf <<EOT
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOT
sudo sysctl --system

Install packages

Now, to install some packages!

sudo apt install -y curl gnupg2 software-properties-common apt-transport-https ca-certificates
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/docker.gpg
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

Now some repos for Containerd so we can install that too.

sudo apt update
sudo apt install -y containerd.io
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl enable containerd

Now time to add the Kubernetes package to our repo, we're opting for version 1.28 here (which leaves me room to practice upgrading my cluster nodes! #PracticeMakesPermenant)

$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt install -y kubelet kubeadm kubectl

So what's happening here? First, we download the public GPG key then we add the Kubernetes apr repo as it's not natively available on Ubuntu.

Make note of "pkgs.k8s.io" in the package repo URL, there has been a recent update as to where the repos are stored and accessed from instead of the previously Google hosted repos, this happened around a year ago, just so it doesn't trip you up if you're reading some other articles about setting up a cluster. More on the upgrade here.

Optional: You can also run the command sudo apt-mark hold kubelet kubeadm kubectl to pin the version of kublet, kubeadm and kubectl to 1.28 so we don't accidentally upgrade them. This is for dev purposes so you don't have to worry about it for now.

Just the control plane node / master node VM

Kubeadm init

Now this next command is specific for your nominated control plane node or master node, whatever you want to call it.

sudo kubeadm init

You could also manually specify the DNS name or IP of your control plane if you wanted to refer to it with the kube config file using sudo kubeadm init --control-plane-endpoint=YOUR_CONTROLPLANE_VM_NAME

But this simple init command will start to initialise your cluster, once finished it should give you some information about your cluster, and joining nodes to it but for now we're interested in these 2 (we'll get to joining nodes in a minute):

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

First, we make sure the .kube directory exists in our home directory, Kubernetes will look for the cluster config file here.

We're then copying the cluster config file from the Kubernetes directory into the .kube directory which has all the information for kubectl to interact with our cluster kubeapiserver component, we then change the file permissions owner to our user so we can use the config file.

Testing the control plane / master node

Now, let's test it out!

kubectl cluster-info
kubectl get nodes

These commands will display our cluster information the control plane endpoint information etc.

Nearly done! Just the worker nodes now...

Kubeadm join

Now we need some nodes to complete our cluster, don't worry we're nearly done!

On each one of our VMs that will join as a node, we run the kubeadm join command that was displayed after the kube init command on the control plane.

sudo kubeadm join kube-master-vm-name:6443 --token 123456.vcwibsv...
or
sudo kubeadm join 192.168.1.100:6443 --token 123456.vcwibsv...

If your token has expired or you have decided to add another node at a later date, that's not a problem, you can run kubeadm token create --print-join-commandfrom the control plane VM and you can use the kubeadm join command displayed after.

You should see some pre-flight checks output and then you should see This node has jkoined the cluster displayed, jump back on to your control plane/master VM and try kubectl get nodes you should see the node as well as the control plane now displayed.

Continue the same on the other nodes you are adding to your cluster until you're done.

Post cluster creation

Last thing...... you might be seeing that your nodes are "Not Ready" What gives?!

We still need to add networking to the cluster so pods and services can find each other. I suggest the Calico CNI

kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml

This is applying the manifest file from the Calico GitHub repo to our Kubernetes cluster you can read more from the repo here.

Testing. Testing. 1, 2, 3...

Now, let's check that worked...

kubectl get pods -n kube-system

Wait for the Calico pods, you should have one per node because its a DaemonSet and a calico-kube-controller which is a deployment.

And then finallykubectl get nodes should display something like......

NAME                STATUS   ROLES           AGE   VERSION
kube-dev-master-1   Ready    control-plane   16d   v1.28.10
kube-dev-node-1     Ready    <none>          16d   v1.28.10
kube-dev-node-2     Ready    <none>          15d   v1.28.10
kube-dev-node-3     Ready    <none>          14d   v1.28.10

Congrats!

You made it this far and if it looks like the above..... You've done it! Nice work!

Try creating a test pod/deployment workload using either of the following commands:

kubectl run web-test --image nginx
or
kubectl create deployment web-test --image nginx --replicas 3

This will create a single pod that is running nginx or a deployment that will ensure 3 pods are running nginx.

I've tried to create this how-to guide as best as I can from my experience creating my own cluster for preparing for the CKA exam, this is by no means a production-ready cluster!

But just for running internally at home, this should give some good insight and some good practice for cluster administration and hopefully, someone might find this useful.

Feel free to leave any questions, comments, or feedback. I'm always happy to take constructive feedback or just hear how people might do this differently!