File Corruption in Docker Image

[rty813]

Description

I have been using the Docker image I built myself without any issues. However, on one occasion after pulling it, I found that the 'ping' command was not available in container and the size of '/usr/bin/ping' was 0.

So I would like to know, 1. how this file corruption occurred and how to avoid it? 2. If it is unavoidable, is there any verification method to check if an image is intact?

Reproduce

1. docker pull xxxxxxx
2. docker run --rm xxxxxx ping -V

Expected behavior

No response

docker version

Client:
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:34:41 2023
 OS/Arch:           linux/arm
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:38:47 2023
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          v1.7.1
  GitCommit:        1677a17964311325ed1c31e2c0a3589ce6d5c30d
 runc:
  Version:          1.1.8
  GitCommit:        v1.1.8-0-g82f18fe
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc.)
    Version:  v0.9.1-beta3
    Path:     /usr/libexec/docker/cli-plugins/docker-app
  buildx: Build with BuildKit (Docker Inc.)
    Version:  v0.5.1-docker
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 6
  Running: 5
  Paused: 0
  Stopped: 1
 Images: 7
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: local
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1677a17964311325ed1c31e2c0a3589ce6d5c30d
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 4.1.15-g9f7912359-dirty
 Operating System: Ubuntu 16.04.7 LTS
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 1.963GiB
 Name: u12
 ID: 6d15ad24-9836-43b8-b928-ca173b0cdab4
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Additional Info

No response

[rty813]

The image is based on Ubuntu 20.04 and uploaded to the self-hosted registry.
Is there any other information that needs to be provided?

[thaJeztah]

You mention "on one occasion";

• is this the exact same image you were pulling before, or has it been updated (rebuild) in between?
• is this on the same machine you pulled the image from before, or a different machine?
• did anything change in between "it worked" and "it doesn't work"? (updated docker in between, or updated your system?)

If there's no steps to reproduce the issue, or (e.g.) logs from when this happened (perhaps an error occurred when extracting the image), I don't think there's much "actionable" here; it will be hard to tell what happened (which could be external to docker, for example, anti-malware may have quarantined the file).

I do notice though that you look to be running a version of Ubuntu that reached EOL, and for which we no longer provide packages;

Operating System: Ubuntu 16.04.7 LTS

Ubuntu 16.04 reached EOL in 2021, and is no longer maintained (which includes security fixes). I think docker 20.10.7 was the last built for Ubuntu 16.04; docker 23.0 and up have not been tested on it (so may not be fully compatible),

And it looks like the machine is running a custom kernel, which was built from modified source (which could be relevant if there's (e.g.) an issue with overlayfs);

Kernel Version: 4.1.15-g9f7912359-dirty

[rty813]

1. The image was not updated between the two pulls.
2. It is on another machine.
3. No changes

Due to some log reading bugs in version 20.10.7, we had to manually update to the latest Docker version via the binary file.

Or is there any way to verify that the pulled image is complete? If it's incomplete, I can pull it again.