gcplogs sends inconsistent entries to stackdriver based on metainformation available #40505
Description
gcplogs sends inconsistent entries to stackdriver based on metainformation available.
Reproduction steps:
- Create a machine with boot disk Container Optimized OS
- Execute
docker run -d --name mysite --log-driver=gcplogs -p 80:80 nginx - Execute
curl localhost - Reset the VM
- Execute
docker run -d --name mysite2 --log-driver=gcplogs -p 80:80 nginx - Execute
curl localhost
Expected behaviour:
You get two entries under "GCE VM Instance, instance-name" > "gcplogs-docker-driver"
Actual behaviour:
You get one entry under "Global" > "gcplogs-docker-driver"
You get one entry under "GCE VM Instance, instance-name" > "gcplogs-docker-driver"
Additional information:
In the actual behaviour, there's a single difference between the two entries. The first one is missing the labels instance_id and zone:
resource:
labels:
project_id: "my-project-id"
whereas the second one (after a reset) looks like this:
resource:
labels:
instance_id: "2836207941276610115"
project_id: "my-project-id"
zone: "europe-west1-b"
Bottom line
The actual behaviour can be explained as the gcplogs driver not attaching the instance id nor the zone at the initial startup and then, after a reset, attaching the missing labels and thus, logging the entry correctly under the GCE VM Instance logs with the respective instance name.
The first entry doesn't have these labels, and that's the reason Stackdriver fills these entries under Global.
Output of docker version:
Client:
Version: 19.03.5
API version: 1.40
Go version: go1.11.2
Git commit: 633a0ea
Built: Sun Feb 9 06:22:13 2020
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 19.03.5
API version: 1.40 (minimum version 1.12)
Go version: go1.11.2
Git commit: 633a0ea
Built: Sun Feb 9 06:21:33 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.9
GitCommit: d50db0a42053864a270f648048f9a8b4f24eced3
runc:
Version: 1.0.0-rc8
GitCommit: 425e105d5a03fabd737a126ad93d62a9eeede87f
docker-init:
Version: 0.18.0
GitCommit: fec3683b971d9c3ef73f284f176672c44b448662
Output of docker info:
Client:
Debug Mode: false
Server:
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 2
Server Version: 19.03.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d50db0a42053864a270f648048f9a8b4f24eced3
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683b971d9c3ef73f284f176672c44b448662
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.19.87+
Operating System: Container-Optimized OS from Google
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.611GiB
Name: instance-3
ID: Z773:IACZ:YRJO:RCYP:D2FV:N655:NZRQ:TFPZ:VW4U:GKWY:BAXR:MKQB
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://mirror.gcr.io/
Live Restore Enabled: true
Additional environment details (AWS, VirtualBox, physical, etc.):
This happens on GCP with vanilla products and the instructions mentioned above, nothing extra.