Wrestling with Kube-Proxy on Ubuntu 20.04.1

Note: This post was originally posted on my personal blog. I have copied the content to this blog.

I recently spun up a couple hosts as Kubernetes workers and interestingly, one of them worked fine but the other would not resolve kube-dns queries. On the node itself, running this command would fail:

nslookup kubernetes 10.43.0.10

In kube-proxy logs I found this:

E0209 04:40:18.259923    1168 proxier.go:1571] Failed to execute iptables-restore: exit status 2 (iptables-restore v1.8.4 (legacy): Couldn't load target `KUBE-MARK-DROP':No such file or directory
 Error occurred at line: 131
 Try `iptables-restore -h' or 'iptables-restore --help' for more information.
 )

I found this article:

Specifically these commands:

sudo sysctl net.bridge.bridge-nf-call-iptables=1
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy

Running that seemed to fix my problems, but after a reboot it didn’t work, and running the commands again had no effect.

(Un)helpfully, there is a GitHub issue for this, which says the problem is fixed in a later version.

I ended up just blasting away the node and making sure to bring up a new one with a different name. Unsatisfying, I know.