8000 fix socket handling during copy by alexcb · Pull Request #1581 · moby/buildkit · GitHub
[go: up one dir, main page]
Skip to content
  • Topics
    Explore
  • Repositories
  • Available add-ons
  • Pricing

    • Search code, repositories, users, issues, pull requests...

    Provide feedback

    We read every piece of feedback, and take your input very seriously.

    Saved searches

    Use saved searches to filter your results more quickly
    Appearance settings

    fix socket handling during copy #1581

    New issue

    Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

    Sign up for GitHub

    By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

    Already on GitHub? Sign in to your account

    Jump to bottom
    Merged
    merged 1 commit into from
    Jul 23, 2020
    Merged

    fix socket handling during copy #1581

    merged 1 commit into from
    Jul 23, 2020

    Conversation

    alexcb
    Copy link
    Collaborator
    @alexcb alexcb commented Jul 18, 2020 •
    edited
    Loading
    8000

    This reproduces an error with tar trying to archive a socket, which is similar to the one that was fixed in #1144

            client_test.go:1696: 
            	Error Trace:	client_test.go:1696
            	            				run.go:165
            	Error:      	Received unexpected error:
            	            	rpc error: code = Unknown desc = failed to compute cache key: failed to create hash for /root/socket.sock: archive/tar: sockets not supported
            	            	github.com/moby/buildkit/util/stack.Enable
            	            		/src/util/stack/stack.go:51
            	            	github.com/moby/buildkit/util/grpcerrors.FromGRPC
            	            		/src/util/grpcerrors/grpcerrors.go:162
            	            	github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor
            	            		/src/util/grpcerrors/intercept.go:22
            	            	google.golang.org/grpc.(*ClientConn).Invoke
            	            		/src/vendor/google.golang.org/grpc/call.go:35
            	            	github.com/moby/buildkit/api/services/control.(*controlClient).Solve
            	            		/src/api/services/control/control.pb.go:1321
            	            	github.com/moby/buildkit/client.(*Client).solve.func2
            	            		/src/client/solve.go:201
            	            	golang.org/x/sync/errgroup.(*Group).Go.func1
            	            		/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1357
            	            	failed to solve
            	            	github.com/moby/buildkit/client.(*Client).solve.func2
            	            		/src/client/solve.go:214
            	            	golang.org/x/sync/errgroup.(*Group).Go.func1
            	            		/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57
            	            	runtime.goexit
            	            		/usr/local/go/src/runtime/asm_amd64.s:1357
            	Test:       	TestIntegration/TestTarExporterWithSocketCopy/worker=oci-rootless
        oci.go:74: stdout: /usr/bin/sudo
        oci.go:74: stderr: /usr/bin/sudo
        oci.go:77: time="2020-07-18T04:54:13Z" level=info msg="auto snapshotter: using overlayfs"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="running in rootless mode"
        oci.go:77: time="2020-07-18T04:54:13Z" level=info msg="found worker \"0d7fqlw0dzm6coe47xuqsliqo\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:5e9fae6f5c60 org.mobyproject.buildkit.worker.sandbox:true org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/386]"
        oci.go:77: time="2020-07-18T04:54:13Z" level=warning msg="rootless mode is not supported for containerd workers. disabling containerd worker."
        oci.go:77: time="2020-07-18T04:54:13Z" level=info msg="found 1 workers, default=\"0d7fqlw0dzm6coe47xuqsliqo\""
        oci.go:77: time="2020-07-18T04:54:13Z" level=warning msg="currently, only the default worker can be used."
        oci.go:77: time="2020-07-18T04:54:13Z" level=info msg="running server on /tmp/bktest_buildkitd003312146/buildkitd.sock"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="session started"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=resolving host="localhost:38005"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="do request" host="localhost:38005" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.3.0+unknown request.method=HEAD url="http://localhost:38005/v2/library/alpine/manifests/latest"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="fetch response received" host="localhost:38005" response.header.content-length=528 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Sat, 18 Jul 2020 04:54:13 GMT" response.header.docker-content-digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65\"" response.status="200 OK" url="http://localhost:38005/v2/library/alpine/manifests/latest"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=resolved desc.digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" host="localhost:38005"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=resolving host="localhost:38005"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="do request" host="localhost:38005" request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.3.0+unknown request.method=HEAD url="http://localhost:38005/v2/library/alpine/manifests/sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="fetch response received" host="localhost:38005" response.header.content-length=528 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Sat, 18 Jul 2020 04:54:13 GMT" response.header.docker-content-digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65\"" response.status="200 OK" url="http://localhost:38005/v2/library/alpine/manifests/sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=resolved desc.digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" host="localhost:38005"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=fetch digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" mediatype=application/vnd.docker.distribution.manifest.v2+json size=528
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="do request" digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.3.0+unknown request.method=GET size=528 url="http://localhost:38005/v2/library/alpine/manifests/sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="fetch response received" digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=528 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Sat, 18 Jul 2020 04:54:13 GMT" response.header.docker-content-digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65\"" response.status="200 OK" size=528 url="http://localhost:38005/v2/library/alpine/manifests/sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=fetch digest="sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e" mediatype=application/vnd.docker.container.image.v1+json size=1509
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="do request" digest="sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.3.0+unknown request.method=GET size=1509 url="http://localhost:38005/v2/library/alpine/blobs/sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="fetch response received" digest="sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=1509 response.header.content-type=application/octet-stream response.header.date="Sat, 18 Jul 2020 04:54:13 GMT" response.header.docker-content-digest="sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e\"" response.status="200 OK" size=1509 url="http://localhost:38005/v2/library/alpine/blobs/sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=fetch digest="sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65" mediatype=application/vnd.docker.distribution.manifest.v2+json size=528
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=fetch digest="sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=2797541
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg=fetch digest="sha256:a24bb4013296f61e89ba57005a7b3e52274d8edd3ae2077d04395f806b63d83e" mediatype=application/vnd.docker.container.image.v1+json size=1509
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="do request" digest="sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.3.0+unknown request.method=GET size=2797541 url="http://localhost:38005/v2/library/alpine/blobs/sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="fetch response received" digest="sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.content-length=2797541 response.header.content-type=application/octet-stream response.header.date="Sat, 18 Jul 2020 04:54:13 GMT" response.header.docker-content-digest="sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c\"" response.status="200 OK" size=2797541 url="http://localhost:38005/v2/library/alpine/blobs/sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="unpigz not found, falling back to go gzip" error="exec: \"unpigz\": executable file not found in $PATH"
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="diff applied" d=255.743209ms dgst="sha256:df20fa9351a15782c64e6dddb2d4a6f50bf6d3688060a34c4014b0d9a752eb4c" media=application/vnd.docker.image.rootfs.diff.tar.gzip size=0
        oci.go:77: time="2020-07-18T04:54:13Z" level=debug msg="> creating bmn7txh17f5go1zdf1dg57hk2 [sh -c nc -l -s local:/root/socket.sock & usleep 100000; kill %1]"
        oci.go:77: time="2020-07-18T04:54:14Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to compute cache key: failed to create hash for /root/socket.sock: archive/tar: sockets not supported"
        oci.go:77: time="2020-07-18T04:54:14Z" level=debug msg="session finished: <nil>"
FAIL

    @alexcb alexcb force-pushed the tar-exporter-with-socket-copy-reproduce-error branch from f680d09 to ed029cf Compare July 18, 2020 05:07
    @tonistiigi
    Copy link
    Member

    @alexcb Are you working on a fix or do you want us to take over from here?

    @alexcb
    Copy link
    Collaborator Author
    alexcb commented Jul 18, 2020

    @tonistiigi I haven't started on any fix. I'm fairly new to buildkit, so it might take me a while to figure out what needs fixing. If you have time and it's any easy fix then I would gladly let you take over.

    @alexcb
    Copy link
    Collaborator Author
    alexcb commented Jul 23, 2020

    I discovered one of the fstypes.Stat structs was being directly created in buildkit rather than in fsutil/stat.go.

    tonistiigi
    tonistiigi reviewed Jul 23, 2020
    View reviewed changes
    cache/contenthash/filehash.go Outdated
    }

    // Clear the socket bit since archive/tar.FileInfoHeader does not handle it
    stat.Mode &^= uint32(os.ModeSocket)
    Copy link
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    would this make more sense inside NewFromStat ?

    Copy link
    Collaborator Author

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    If you think that's a better place we can definitely move it there.

    in NewFromStat(stat *fstypes.Stat) stat is passed as a pointer, would it be fine to modify the contents, or would it be better to make a copy of it so it's essentially passed in as a const?

    Copy link
    Member

    Choose a reason for hiding this comment

    The reason will be displayed to describe this comment to others. Learn more.

    I'm fine with the current as it is a trivial case, but if you want to be more correct you could check it the socket bit exists and only do copy then.

    @alexcb alexcb marked this pull request as ready for review July 23, 2020 17:08
    @alexcb alexcb force-pushed the tar-exporter-with-socket-copy-reproduce-error branch 2 times, most recently from 7c5a106 to 1a5fbb3 Compare July 23, 2020 17:18
    @alexcb alexcb changed the title reproduce tar exporter with socket error fix socket handling during copy Jul 23, 2020
    @tonistiigi
    Copy link
    Member

    LGTM but please squash the commits so it doesn't cause issues for git bisect in the future.

    Treat unix sockets as regular files
    5382a20 
    This fix is similar to the fix in moby#1144; but was hit in a different code
path.

Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
    @alexcb alexcb force-pushed the tar-exporter-with-socket-copy-reproduce-error branch from 1a5fbb3 to 5382a20 Compare July 23, 2020 18:26
    @alexcb
    Copy link
    Collaborator Author
    alexcb commented Jul 23, 2020

    squashed! thanks for the help with the PR.

    @alexcb
    Copy link
    Collaborator Author
    alexcb commented Jul 23, 2020 •
    edited
    Loading

    that's odd the tests passed before my squash, but after the squash it panicked in a different part of the code:

    panic: runtime error: slice bounds out of range [19792:19280]
goroutine 7169 [running]:
bytes.(*Buffer).Write(0xc000facab0, 0xc0009fa400, 0x52, 0x900, 0x3, 0x3, 0x1496f00)
	/usr/local/go/src/bytes/buffer.go:174 +0x102
fmt.Fprintf(0x15fc7a0, 0xc000facab0, 0x14b30b1, 0x14, 0xc000f62718, 0x3, 0x3, 0xc000f626f0, 0xd271a1, 0xc001c84740)
	/usr/local/go/src/fmt/print.go:205 +0xa5
github.com/moby/buildkit/util/testutil/integration.startCmd.func1(0xc000f62768, 0xc00008d980)
	/src/util/testutil/integration/util.go:39 +0x1b8
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000facb10, 0xc0005101e0)
	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
	/src/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
FAIL	github.com/moby/buildkit/frontend/dockerfile	74.189s

    @tonistiigi
    Copy link
    Member

    @alexcb That is unrelated. Restarted and will fix that in separate PR

    @tonistiigi tonistiigi mentioned this pull request Jul 23, 2020
    Merged
    @tonistiigi tonistiigi merged commit a67af5b into moby:master Jul 23, 2020
    alexcb pushed a commit to earthly/earthly that referenced this pull request Jul 23, 2020
    Remove socket file rm hack
    1d99b73 
    An issue in buildkit caused a "tar/archive: sockets not supported" error
while building the dot net example; this issue has been fixed upstream
in buildkit ( see moby/buildkit#1581 ), and we
can now remove this work-around.
    @alexcb alexcb mentioned this pull request Jul 23, 2020
    Merged
    alexcb added a commit to earthly/earthly that referenced this pull request Jul 23, 2020
    @alexcb
    Remove socket file rm hack (#137)
    ca5d8de 
    An issue in buildkit caused a "tar/archive: sockets not supported" error
while building the dot net example; this issue has been fixed upstream
in buildkit ( see moby/buildkit#1581 ), and we
can now remove this work-around.

Co-authored-by: Alex Couture-Beil <alex@earthly.dev>
    @alexcb alexcb mentioned this pull request Jul 24, 2020
    Closed
    This was referenced Jul 25, 2020
    Merged
    Merged
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Reviewers

    @tonistiigi tonistiigi tonistiigi approved these changes

    Assignees
    No one assigned
    Labels
    None yet
    Projects
    None yet
    Milestone
    No milestone
    Development

    Successfully merging this pull request may close these issues.
    2 participants
    @alexcb @tonistiigi
    0