My kubernetes cluster IP address changed and now kubectl will no longer connect

I found this solution on the internet and it works for me:

    systemctl stop kubelet docker
    cd /etc/
    mv kubernetes kubernetes-backup
    mv /var/lib/kubelet /var/lib/kubelet-backup
    mkdir -p kubernetes
    cp -r kubernetes-backup/pki kubernetes
    rm kubernetes/pki/{apiserver.*,etcd/peer.*}
    systemctl start docker
    kubeadm init --ignore-preflight-errors=DirAvailable--var-lib-etcd
    #Run "kubeadm reset" on all nodes if was this error "error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
        [ERROR Port-10250]: Port 10250 is in use
        [ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists"
    cp kubernetes/admin.conf ~/.kube/config
    kubectl get nodes --sort-by=.metadata.creationTimestamp
    kubectl delete node $(kubectl get nodes -o jsonpath='{.items[(@.status.conditions[0].status=="Unknown")].metadata.name}')
    kubectl get pods --all-namespaces

After These, Join your Slaves to Master. Reference: https://medium.com/@juniarto.samsudin/ip-address-changes-in-kubernetes-master-node-11527b867e88


The following command can be used to regenerate admin.conf

kubeadm alpha phase kubeconfig admin --apiserver-advertise-address <new_ip>

However, if you use an IP instead of a hostname, your API-server certificate will be invalid. So, either regenerate your certs ( kubeadm alpha phase certs renew apiserver ), use hostnames instead of IPs or add the insecure --insecure-skip-tls-verify flag when using kubectl


You do not want to use kubeadm reset. That will reset everything and you would have to start configuring your cluster again.

Well, in your scenario, please have a look on the steps below:

  1. nano /etc/hosts (update your new IP against YOUR_HOSTNAME)

  2. nano /etc/kubernetes/config (configuration settings related to your cluster) here in this file look for the following params and update accordingly

    KUBE_MASTER="--master=http://YOUR_HOSTNAME:8080"

    KUBE_ETCD_SERVERS="--etcd-servers=http://YOUR_HOSTNAME:2379" #2379 is default port

  3. nano /etc/etcd/etcd.conf (conf related to etcd)

    KUBE_ETCD_SERVERS="--etcd-servers=http://YOUR_HOSTNAME/WHERE_EVER_ETCD_HOSTED:2379"

    2379 is default port for etcd. and you can have multiple etcd servers defined here comma separated

  4. Restart kubelet, apiserver, etcd services.

It is good to use hostname instead of IP to avoid such scenarios.

Hope it helps!

Tags:

Kubernetes

Recent Posts