Unable to run Hyperkube (kubernetes) locally via Docker
I've reproduced your issue before, and I've also successfully run the kubelet container a couple times.
Here is the exact command I am running when it succeeds:
export K8S_VERSION=v1.2.2
docker run \
--volume=/:/rootfs:ro \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:rw \
--volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
--volume=/var/run:/var/run:rw \
--net=host \
--pid=host \
--privileged=true \
--name=kubelet \
-d \
gcr.io/google_containers/hyperkube-amd64:${K8S_VERSION} \
/hyperkube kubelet \
--containerized \
--hostname-override="127.0.0.1" \
--address="0.0.0.0" \
--api-servers=http://localhost:8080 \
--config=/etc/kubernetes/manifests \
--allow-privileged=true --v=2
I removed these 2 settings from the tutorial's suggested command because DNS wasn't needed in my case:
--cluster-dns=10.0.0.10
--cluster-domain=cluster.local
Also, I started the docker SSH portal in the background before starting the kubelet container, using this command:
docker-machine ssh `docker-machine active` -f -N -L "8080:localhost:8080"
I also did not make any changes to SSL certificates.
I am able to run the kubelet container with K8S_VERSION=v1.2.2 and K8S_VERSION=1.2.3.
On a successful run, I observe all the processes are "Up"; none are "Exited":
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
42e6d973f624 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube apiserver" About an hour ago Up About an hour k8s_apiserver.78ec1de_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_5d260d3c
135c020f14b4 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube controlle" About an hour ago Up About an hour k8s_controller-manager.70414b65_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_9b338f27
873656c913fd gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/setup-files.sh IP:1" About an hour ago Up About an hour k8s_setup.e5aa3216_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_ff89fc7c
8b12f5f20e8f gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube scheduler" About an hour ago Up About an hour k8s_scheduler.fc12fcbe_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_ea90af75
93d9b2387b2e gcr.io/google_containers/etcd:2.2.1 "/usr/local/bin/etcd " About an hour ago Up About an hour k8s_etcd.7e452b0b_k8s-etcd-127.0.0.1_default_1df6a8b4d6e129d5ed8840e370203c11_d66f84f0
f6e45af93ee9 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube proxy --m" About an hour ago Up About an hour k8s_kube-proxy.9a9f4853_k8s-proxy-127.0.0.1_default_5e5303a9d49035e9fad52bfc4c88edc8_b0084efc
f6748442f2d1 gcr.io/google_containers/pause:2.0 "/pause" About an hour ago Up About an hour k8s_POD.6059dfa2_k8s-master-127.0.0.1_default_4c6ab43ac4ee970e1f563d76ab3d3ec9_f4758f9b
d515c10910c4 gcr.io/google_containers/pause:2.0 "/pause" About an hour ago Up About an hour k8s_POD.6059dfa2_k8s-etcd-127.0.0.1_default_1df6a8b4d6e129d5ed8840e370203c11_3248c1d6
958f4865df9f gcr.io/google_containers/pause:2.0 "/pause" About an hour ago Up About an hour k8s_POD.6059dfa2_k8s-proxy-127.0.0.1_default_5e5303a9d49035e9fad52bfc4c88edc8_3850b11e
2611ee951476 gcr.io/google_containers/hyperkube-amd64:v1.2.2 "/hyperkube kubelet -" About an hour ago Up About an hour kubelet
On a successful run, I also see similar log output as you when I run docker logs kubelet. In particular, I see:
Unable to register 127.0.0.1 with the apiserver: Post http://localhost:8080/api/v1/nodes: dial tcp 127.0.0.1:8080: connection refused
But, eventually, it works:
$ kubectl -s http://localhost:8080 cluster-info
Kubernetes master is running at http://localhost:8080
$ kubectl get nodes
NAME STATUS AGE
127.0.0.1 Ready 1h
192.168.99.100 NotReady 1h
localhost NotReady 1h
Other tips:
You might need to wait a little bit for the API server to start up. For example, this guy uses a while loop:
until $(kubectl -s http://localhost:8080 cluster-info &> /dev/null); do sleep 1 doneOn Mac OS X, I've noticed the Docker VM can get unstable whenever my wireless changes, or when I suspend/resume my laptop. I can usually resolve such issues with a
docker-machine restart.When experimenting with kubelet, I'll often want to stop the kubelet container and stop/remove all containers in my docker. I do that by running
docker stop kubelet && docker rm -f $(docker ps -aq)
Info about my setup, OS X El Capitan 10.11.2:
$ docker --version
Docker version 1.10.3, build 20f81dd
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.0", GitCommit:"5cb86ee022267586db386f62781338b0483733b3", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.2", GitCommit:"528f879e7d3790ea4287687ef0ab3f2a01cc2718", GitTreeState:"clean"}
[I'm not a kubernetes expert - just following my nose here].
kubelet's failure is apparently a consequent symptom of port 8080 being closed, which you noted at the beginning of your question. It's not where you should be focused.
Note the following line in the logs you showed us:
I0422 11:05:28.469355 541 kubelet.go:1137] Unable to register 172.20.34.112 with the apiserver: Post http://localhost:8080/api/v1/nodes: dial tcp 127.0.0.1:8080: connection refused
So, kubelet is trying to contact the apiserver, and getting connection refused. That's not surprising given that as you note, it has exited.
The log lines you show us for the apiserver show it complaining about not having a certificate. The certificates are normally in /var/run/kubernetes (noted here). That falls within the /var/run volume that's set up in the docker command for running kubernetes in your tutorial. I'd be looking closely at that volume specification to see if you've made any mistakes, and to see if the certificates are in there as expected.
There's a few bits at https://github.com/kubernetes/kubernetes/issues/11000 which might be useful for figuring out what's going wrong with your certs, including devurandom providing a script for creating the certs if that's what's needed.