Docker 1.8 corrupts /etc/hosts files in containers #17190
Description
related: #16619
@mavenugo - We have a problem.
We've been experiencing periodic end-to-end test failures where some containers can not resolve "localhost". Killing the container and letting it restart usually fixes the problem. Today we got it into this state and pinned down the following:
The container in question has a /etc/hosts file that is several hundred lines long. That /etc/hosts file does NOT have localhost or the container's own IP. If I add those 2 lines the tests pass.
Of course I thought of #16619 . So we looked at /etc/hosts of every container on this machine - 84 in total. Those files range in size from 175 lines (correct: 7 boilerplate + 2 per container) to 1700+ lines. Looking at the offending files, there are a lot of duplicate IPs. It looks like old containers have not been removed, but localhost, "self", and the ipv6 boilerplate has been removed.
Even looking at my local test machine (far less churn) I see the same. I have 4 containers with network namespace running, I should have 8+7=15 lines, right?
# wc -l /var/lib/docker/containers/*/hosts
15 /var/lib/docker/containers/3fc23782814513d1934ce21c0f1b096263df1fcacee6787208c3cda715822356/hosts
15 /var/lib/docker/containers/4b7e8432a368eeb7f1dc49230f16c1d7c788b471e4dfa7d8fbf709c3495f430f/hosts
19 /var/lib/docker/containers/5715aa09fc091ee0ff3d4dcf2ee300d8e52603dafcde8f24aaf1def960cf4a28/hosts
15 /var/lib/docker/containers/6e7fe564dece829de3c335400357b2ef327a1a44c638045350f0b49e5324a189/hosts
15 /var/lib/docker/containers/7845351acbacf0588142a1ebfa3fa481c3191da48a188b20c5d5fdae689cef1e/hosts
17 /var/lib/docker/containers/ba23cbabfa90700f4d9ba80c5723afb1d7479a08b12efdded74e5874991ce76b/hosts
17 /var/lib/docker/containers/c1e64038b35a0527b6aef911dad002278569724263cc922efd8e715747e83538/hosts
19 /var/lib/docker/containers/fc7c5475fd923b75a0595431dcaf6b6db5fa8a18409d0df4e6282e3c0bd8aedf/hosts
Diffing a 15 and 19 line file shows:
# diff -u -U 100 <(sort -n /var/lib/docker/containers/3fc23782814513d1934ce21c0f1b096263df1fcacee6787208c3cda715822356/hosts) <(sort -n /var/lib/docker/containers/5715aa09fc091ee0ff3d4dcf2ee300d8e52603dafcde8f24aaf1def960cf4a28/hosts)
--- /dev/fd/63 2015-10-19 23:59:20.618879795 +0000
+++ /dev/fd/62 2015-10-19 23:59:20.618879795 +0000
@@ -1,15 +1,19 @@
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
+10.244.0.10 distracted_ritchie
+10.244.0.10 distracted_ritchie.bridge
10.244.0.2 k8s_POD-7be6d81d_fluentd-cloud-logging-kubernetes-minion-isfq_kube-system_dcfde077753b8b6908952cfea6a14200_c4f00d1d
10.244.0.2 k8s_POD-7be6d81d_fluentd-cloud-logging-kubernetes-minion-isfq_kube-system_dcfde077753b8b6908952cfea6a14200_c4f00d1d.bridge
10.244.0.3 k8s_POD-7be6d81d_heapster-v10-lx8q7_kube-system_2a3fc6d6-6ddb-11e5-9249-42010af00002_cbe79e10
10.244.0.3 k8s_POD-7be6d81d_heapster-v10-lx8q7_kube-system_2a3fc6d6-6ddb-11e5-9249-42010af00002_cbe79e10.bridge
10.244.0.4 k8s_POD-c5371ceb_monitoring-influxdb-grafana-v2-koh3j_kube-system_2a43f315-6ddb-11e5-9249-42010af00002_76d3cb92
10.244.0.4 k8s_POD-c5371ceb_monitoring-influxdb-grafana-v2-koh3j_kube-system_2a43f315-6ddb-11e5-9249-42010af00002_76d3cb92.bridge
10.244.0.5 k8s_POD-9db2f941_kube-ui-v2-v4yd6_kube-system_2a42e6da-6ddb-11e5-9249-42010af00002_69a0a0c7
10.244.0.5 k8s_POD-9db2f941_kube-ui-v2-v4yd6_kube-system_2a42e6da-6ddb-11e5-9249-42010af00002_69a0a0c7.bridge
-10.244.0.5 kube-ui-v2-v4yd6
+10.244.0.6 k8s_POD-6e934112_kube-dns-v9-ee38a_kube-system_2a4c2763-6ddb-11e5-9249-42010af00002_86e49fbc
+10.244.0.6 k8s_POD-6e934112_kube-dns-v9-ee38a_kube-system_2a4c2763-6ddb-11e5-9249-42010af00002_86e49fbc.bridge
+10.244.0.6 kube-dns-v9-ee38a
127.0.0.1 localhost
Is there any reason the hosts files would be different for different containers on the same node?
I know docker is live-editing the hosts file (which is absolutely unsafe, but should only be unsafe for the users in the container). Any clues on what's going on?