Docker swarm overlay networking not working after --force-new-cluster #495
Description
- This is a bug report
- This is a feature request
- I searched existing issues before opening this one
Expected behavior
After a --force-new-cluster and subsequently adding a new node to the cluster the tasks.servicename should be resolved by internal docker dns and containers on the same overlay network should be able to reach each other.
Actual behavior
On the node on which --force-new-cluster was executed the tasks.servicename endpoint will not resolve. On the added node, the tasks.servicename does resolve but it will only resolve to the container on the one node. Also, the containers on the same overlay network cannot reach each other by their ips.
Steps to reproduce the behavior
- Using the following Docker file build an image on each node called demo.
FROM ubuntu
RUN apt update
RUN apt install dnsutils -y
CMD /bin/bash -c "while true; do nslookup tasks.demo; sleep 2; done"
- execute swarm init on one of the nodes
- create a network
docker network create --scope swarm --driver overlay --attachable test - create a service
docker service create --network test --mode global --name demo demo - verify that the tasks.demo endpoint resolves to two ip addresses
docker service logs demo - now execute
docker swarm init --force-new-clusteron one of the nodes - demote and remove the other node and also, remove the service and network
- recreate the service and network on the remaining node
- have a third node join the remaining node
- at this point node 3 will resolve tasks.demo to be it's container's ip but the tasks.demo will not resolve on the first node. Also the container on each node cannot reach the container on the other node using it's ip.
Restarting the docker daemon on the first node does resolve the issue.
Output of docker version:
`Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Wed Jun 20 21:43:51 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Wed Jun 20 21:42:00 2018
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 8
Server Version: 18.03.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: itrdsuwlqi234atk1nwc8foha
Is Manager: true
ClusterID: ysq5qap98z4gbilfi4z3o60j3
Managers: 2
Nodes: 2
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.138.0.16
Manager Addresses:
10.138.0.11:2377
35.227.182.132:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-1024-gcp
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.607GiB
Name: instance-9
ID: TEPO:ELY7:EYOT:LPCS:OQ4B:DKKA:FK2U:XJ52:RXF7:7CGN:GEXO:YLAN
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.)
gcloud instances. Have reproduced the same behavior on 18.09 ce as well.