Add import gzip support #10241

jokemanfire · 2024-05-17T12:57:34Z

when ctr import a image file , it can be gzip ~~and bzip2~~ format file.

k8s-ci-robot · 2024-05-17T12:57:43Z

Hi @jokemanfire. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jokemanfire · 2024-05-17T12:58:45Z

#9981

core/images/archive/importer.go

pkg/archive/compression/compression.go

dmcgowan

Only include the change in pkg/archive/compression, the change to core/images/archive is not needed.

dmcgowan · 2024-05-20T05:12:34Z

core/images/archive/importer.go

@@ -82,6 +80,12 @@ func ImportIndex(ctx context.Context, store content.Store, reader io.Reader, opt
 			return ocispec.Descriptor{}, err
 		}
 	}
+	s, err := compression.DecompressStream(reader)


The interface expects the caller to call DecompressStream. This would cause DecompressStream to be called twice in some cases.

Sorry, May I do not understand （Can you explain it to me clearly? thanks）. I add this code ,because when I use ctr i import any file ,not only tar , I can be imported successful. I search the code , but just image Import will call ImportIndex (Is there something I missing?) . If the reader is tar file , the DecompressStream will do nothing , I think it will not cause any problem if it is called twice.

ZhangShuaiyi · 2024-05-24T10:25:28Z

What version are you testing and how did you create the tar.gz file?
I have tested the main version and containerd is able to successfully import the tar.gz image.

save tar.gz

[root@dockervm ~]# docker save busybox:latest | gzip > busybox.tar.gz
[root@dockervm ~]# file busybox.tar.gz
busybox.tar.gz: gzip compressed data, last modified: Fri May 24 09:39:14 2024, from Unix, original size 4505600

import tar.gz

[root@devnode1 containerd]# ctr version
Client:
  Version:  v2.0.0-rc.2-6-g45e30913b
  Revision: 45e30913bc75c43148e00aade10891e5a4a6eba5
  Go version: go1.22.3

Server:
  Version:  v2.0.0-rc.2-6-g45e30913b
  Revision: 45e30913bc75c43148e00aade10891e5a4a6eba5
  UUID: 79343445-f320-4901-b203-5c2a2bb87144
[root@devnode1 containerd]# ctr image import busybox.tar.gz
docker.io/library/busybox:latest                saved
application/vnd.oci.image.manifest.v1+json sha256:1d8ee6ccd551c9d8dfa55730cba130e76f5a69ec8dbebe17fb5e1c309077dfb5
Importing       elapsed: 0.4 s  total:   0.0 B  (0.0 B/s)

jokemanfire · 2024-05-24T12:11:59Z

What version are you testing and how did you create the tar.gz file? I have tested the main version and containerd is able to successfully import the tar.gz image.

save tar.gz

[root@dockervm ~]# docker save busybox:latest | gzip > busybox.tar.gz
[root@dockervm ~]# file busybox.tar.gz
busybox.tar.gz: gzip compressed data, last modified: Fri May 24 09:39:14 2024, from Unix, original size 4505600

import tar.gz

[root@devnode1 containerd]# ctr version
Client:
  Version:  v2.0.0-rc.2-6-g45e30913b
  Revision: 45e30913bc75c43148e00aade10891e5a4a6eba5
  Go version: go1.22.3

Server:
  Version:  v2.0.0-rc.2-6-g45e30913b
  Revision: 45e30913bc75c43148e00aade10891e5a4a6eba5
  UUID: 79343445-f320-4901-b203-5c2a2bb87144
[root@devnode1 containerd]# ctr image import busybox.tar.gz
docker.io/library/busybox:latest                saved
application/vnd.oci.image.manifest.v1+json sha256:1d8ee6ccd551c9d8dfa55730cba130e76f5a69ec8dbebe17fb5e1c309077dfb5
Importing       elapsed: 0.4 s  total:   0.0 B  (0.0 B/s)

I compile containerd from main branch , make sure you have gzip file , could you check the 3 bytes front of the file is ’1f 8b 08 ‘？ hexdump -C busybox.tar.gz |head .. I trace the code in import ,but there is no decomporess , If your code is from main branch now?

ZhangShuaiyi · 2024-05-24T13:35:16Z

What version are you testing and how did you create the tar.gz file? I have tested the main version and containerd is able to successfully import the tar.gz image.

save tar.gz
[root@dockervm ~]# docker save busybox:latest | gzip > busybox.tar.gz
[root@dockervm ~]# file busybox.tar.gz
busybox.tar.gz: gzip compressed data, last modified: Fri May 24 09:39:14 2024, from Unix, original size 4505600
import tar.gz
[root@devnode1 containerd]# ctr version
Client:
  Version:  v2.0.0-rc.2-6-g45e30913b
  Revision: 45e30913bc75c43148e00aade10891e5a4a6eba5
  Go version: go1.22.3

Server:
  Version:  v2.0.0-rc.2-6-g45e30913b
  Revision: 45e30913bc75c43148e00aade10891e5a4a6eba5
  UUID: 79343445-f320-4901-b203-5c2a2bb87144
[root@devnode1 containerd]# ctr image import busybox.tar.gz
docker.io/library/busybox:latest                saved
application/vnd.oci.image.manifest.v1+json sha256:1d8ee6ccd551c9d8dfa55730cba130e76f5a69ec8dbebe17fb5e1c309077dfb5
Importing       elapsed: 0.4 s  total:   0.0 B  (0.0 B/s)
I compile containerd from main branch , make sure you have gzip file , could you check the 3 bytes front of the file is ’1f 8b 08 ‘？ hexdump -C busybox.tar.gz |head .. I trace the code in import ,but there is no decomporess , If your code is from main branch now?

[root@devnode1 containerd]# git pull origin main
From https://github.com/containerd/containerd
 * branch                main       -> FETCH_HEAD
Already up to date.
[root@devnode1 containerd]# git log -n 1
commit 45e30913bc75c43148e00aade10891e5a4a6eba5 (HEAD -> main, origin/main, origin/HEAD)
Merge: ccc41e670 65024e6fd
Author: Derek McGowan <derek@mcg.dev>
Date:   Thu May 23 21:23:18 2024 +0000

    Merge pull request #10257 from akhilerm/fix-unknown-platform

    core/image: fix usage of "unknown" platform
[root@devnode1 containerd]# ctr version
Client:
  Version:  v2.0.0-rc.2-6-g45e30913b
  Revision: 45e30913bc75c43148e00aade10891e5a4a6eba5
  Go version: go1.22.3

Server:
  Version:  v2.0.0-rc.2-6-g45e30913b
  Revision: 45e30913bc75c43148e00aade10891e5a4a6eba5
  UUID: 79343445-f320-4901-b203-5c2a2bb87144
[root@devnode1 containerd]# hexdump -C busybox.tar.gz |head -n 2
00000000  1f 8b 08 00 42 60 50 66  00 03 ec 5c 0b b4 15 d5  |....B`Pf...\....|
00000010  79 de 17 91 f7 45 44 69  7c 20 39 b9 71 2d 31 c2  |y....EDi| 9.q-1.|
[root@devnode1 containerd]# ctr image import busybox.tar.gz
docker.io/library/busybox:latest                saved
application/vnd.oci.image.manifest.v1+json sha256:1d8ee6ccd551c9d8dfa55730cba130e76f5a69ec8dbebe17fb5e1c309077dfb5
Importing       elapsed: 0.4 s  total:   0.0 B  (0.0 B/s)

main branch, revision 45e30913bc75c43148e00aade10891e5a4a6eba5
first 3 bytes 1f 8b 08

jokemanfire · 2024-05-24T14:23:21Z

docker save busybox:latest

sorry I find my mistake , my ctr use 1.7.2 version . That's such a huge mistake . When I debug , I use the latest containerd but ctr is 1.7.2 . I will change this pr , if xz and bandzip2 is need 。 thanks for u.This is a big black dragon

jokemanfire · 2024-05-24T15:41:00Z

#10260 I have take a new pr

k8s-ci-robot added needs-ok-to-test size/M labels May 17, 2024

AkihiroSuda reviewed May 18, 2024

View reviewed changes

core/images/archive/importer.go Outdated Show resolved Hide resolved

AkihiroSuda reviewed May 18, 2024

View reviewed changes

pkg/archive/compression/compression.go Outdated Show resolved Hide resolved

jokemanfire force-pushed the main branch from d725fde to 83e4a72 Compare May 18, 2024 12:32

AkihiroSuda approved these changes May 18, 2024

View reviewed changes

dmcgowan requested changes May 20, 2024

View reviewed changes

jokemanfire force-pushed the main branch from 83e4a72 to f7193f6 Compare May 22, 2024 13:13

k8s-ci-robot added size/XS and removed size/M labels May 22, 2024

samuelkarp changed the title ~~Add import gzip and bzip2 support~~ Add import gzip support May 22, 2024

jokemanfire closed this May 24, 2024

jokemanfire force-pushed the main branch from f7193f6 to 45e3091 Compare May 24, 2024 14:24

samuelkarp mentioned this pull request May 24, 2024

Add import uncompressd file support for XZ BZip2 #10260

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add import gzip support #10241

Add import gzip support #10241

jokemanfire commented May 17, 2024 •

edited by samuelkarp

k8s-ci-robot commented May 17, 2024

jokemanfire commented May 17, 2024

dmcgowan left a comment

dmcgowan May 20, 2024

jokemanfire May 20, 2024

ZhangShuaiyi commented May 24, 2024

jokemanfire commented May 24, 2024 •

edited

ZhangShuaiyi commented May 24, 2024

jokemanfire commented May 24, 2024

jokemanfire commented May 24, 2024

Add import gzip support #10241

Add import gzip support #10241

Conversation

jokemanfire commented May 17, 2024 • edited by samuelkarp

k8s-ci-robot commented May 17, 2024

jokemanfire commented May 17, 2024

dmcgowan left a comment

Choose a reason for hiding this comment

dmcgowan May 20, 2024

Choose a reason for hiding this comment

jokemanfire May 20, 2024

Choose a reason for hiding this comment

ZhangShuaiyi commented May 24, 2024

jokemanfire commented May 24, 2024 • edited

ZhangShuaiyi commented May 24, 2024

jokemanfire commented May 24, 2024

jokemanfire commented May 24, 2024

jokemanfire commented May 17, 2024 •

edited by samuelkarp

jokemanfire commented May 24, 2024 •

edited