docker logs -f <containerID> fails when container rotates log file #32038
Description
BUG REPORT INFORMATION
Using docker logs -f <containerID> to follow the output logs of a container seems to stop after a while, and sooner if the --log-opt max-size is set to a low value. What we've seen in live production systems, is that only the first log-file is properly followed, but the second it rotates, the docker logs -f command stops tracking the file. Maybe a fsnotify is not being triggered correctly?
Steps to reproduce the issue:
-
execute the following command to execute a docker container with a small max-size:
docker run -d --log-driver=json-file --log-opt max-size=1k alpine /bin/ash -c 'while true; do echo "hit CTRL+C to exit"; sleep 1; done' -
Follow the container logs using the follow command:
docker logs -f <containerID>
Describe the results you received:
The log output stops after around 10-12 lines and never pick up again, even though the container keeps echoing.
Describe the results you expected:
The logs should either keep going or stop (and close the stream) after the json-file was rotated.
Additional information you deem important (e.g. issue happens only occasionally):
This has been observed on both Docker 1.12.6 in Amazon ECS and locally running the version described below using docker version/info.
Output of docker version:
Client:
Version: 17.03.0-ce
API version: 1.26
Go version: go1.7.5
Git commit: 60ccb22
Built: Thu Feb 23 10:40:59 2017
OS/Arch: darwin/amd64
Server:
Version: 17.03.0-ce
API version: 1.26 (minimum version 1.12)
Go version: go1.7.5
Git commit: 3a232c8
Built: Tue Feb 28 07:52:04 2017
OS/Arch: linux/amd64
Experimental: true
Output of docker info:
Containers: 6
Running: 0
Paused: 0
Stopped: 6
Images: 101
Server Version: 17.03.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.12-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.952 GiB
Name: moby
ID: TP72:QUAN:KOBL:NK7N:LTOZ:6NRW:NQPN:GH7K:HZD6:QDDE:GSZJ:GSQU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 17
Goroutines: 28
System Time: 2017-03-23T13:07:49.418835015Z
EventsListeners: 1
No Proxy: *.local, 169.254/16
Username: trustpilotdockerhub
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
Docker for Mac and AWS ECS