Adding worker and control-plane nodes to the cluster

Kubernetes

In my previous post, I wrote about graceful node removal from the Kubernetes cluster.

The reason behind removing nodes was due to a hardware update I wanted to perform. Disks have been replaced, so it is time to add nodes back to the cluster.

Currently, the cluster includes 4 nodes: 2 control-plane nodes, and 2 worker ones.

❯ kubectl get nodes
NAME     STATUS   ROLES           AGE   VERSION
vm0101   Ready    control-plane   35d   v1.24.9
vm0102   Ready    <none>          35d   v1.24.9
vm0201   Ready    control-plane   34d   v1.24.9
vm0202   Ready    <none>          34d   v1.24.9

I would like the following nodes vm0301 and vm0302 to rejoin the cluster.

Adding worker node

To add a worker node, we have to obtain a join command first. Let's ssh to the existing control plane node and create one.

❯ ssh vm0101.maas
ubuntu@vm0101:~$ sudo kubeadm token create --print-join-command
kubeadm join 192.168.111.10:6443 --token mxp7jy.f55up7nxk1quuhba --discovery-token-ca-cert-hash sha256:73d3054f751a331fc9a027a582840ea46cef59552ccfb9eb2fcf2ad5a1577153

Now we can ssh to vm0302 and issue obtained command.

❯ ssh vm0302.maas
ubuntu@vm0302:~$ sudo kubeadm join 192.168.111.10:6443 --token mxp7jy.f55up7nxk1quuhba --discovery-token-ca-cert-hash sha256:73d3054f751a331fc9a027a582840ea46cef59552ccfb9eb2fcf2ad5a1577153
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

A new worker node has been added and can be seen in the node's list.

❯ kubectl get nodes
NAME     STATUS   ROLES           AGE   VERSION
vm0101   Ready    control-plane   35d   v1.24.9
vm0102   Ready    <none>          35d   v1.24.9
vm0201   Ready    control-plane   34d   v1.24.9
vm0202   Ready    <none>          34d   v1.24.9
vm0302   Ready    <none>          58s   v1.24.9

Adding control-plane node

To add another control plane node to the cluster additional step is required, to generate the certificate key. Let's ssh to the existing control plane node again. Using kubeadm to run upload-certs phase will provide us certificate key for the new control plane node.

❯ ssh vm0101.maas
ubuntu@vm0101:~$ sudo kubeadm init phase upload-certs --upload-certs
I0217 21:47:47.354890 4155900 version.go:255] remote version is much newer: v1.26.1; falling back to: stable-1.24
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
8ff3dbe4324d847b67a52f18044471d4cf005749432737824dc621c7cfad7cf6

Now it's time to run a slightly modified join command. We will have to append additional --control-plane and --certificate-key parameters to the previously obtained join command. Now we are ready to go.

❯ ssh vm0301.maas
ubuntu@vm0301:~$ sudo kubeadm token create --print-join-command
kubeadm join 192.168.111.10:6443 --token mxp7jy.f55up7nxk1quuhba --discovery-token-ca-cert-hash sha256:63d3054f751a331fc9a027a582840ea46cef59552ccfb9eb2fcf2ad5a1577153 --control-plane --certificate-key 8ff3dbe4324d847b67a52f18044471d4cf005749432737824dc621c7cfad7cf6
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local kubernetes.maas kubernetes.slys.dev vm0301] and IPs [10.96.0.1 192.168.111.117 192.168.111.10 192.168.111.97]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost vm0301] and IPs [192.168.111.117 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost vm0301] and IPs [192.168.111.117 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[check-etcd] Checking that the etcd cluster is healthy
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[etcd] Announced new etcd member joining to the existing etcd cluster
[etcd] Creating static Pod manifest for "etcd"
[etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s
The 'update-status' phase is deprecated and will be removed in a future release. Currently it performs no operation
[mark-control-plane] Marking the node vm0301 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node vm0301 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

	mkdir -p $HOME/.kube
	sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
	sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

A new control plane node has been added!

❯ kubectl get nodes
NAME     STATUS   ROLES           AGE    VERSION
vm0101   Ready    control-plane   35d    v1.24.9
vm0102   Ready    <none>          35d    v1.24.9
vm0201   Ready    control-plane   34d    v1.24.9
vm0202   Ready    <none>          34d    v1.24.9
vm0301   Ready    control-plane   68s    v1.24.9
vm0302   Ready    <none>          9m8s   v1.24.9

Conclusion

Adding new nodes to the existing Kubernetes cluster seems to be pretty straightforward. Just remember that adding a new control plane node requires a stable and healthy ETCD cluster first. Enjoy the day!