Poor copy/sync speed from a NFS-mounted directory to Ceph S3 bucket

[zhangyeda]

The associated forum post URL from https://forum.rclone.org

https://forum.rclone.org/t/poor-copy-sync-speed-from-a-nfs-mounted-directory-to-ceph-s3-bucket/45912

What is the problem you are having with rclone?

In a 1Gbps network environment, I'm using rclone copy files from a NFS-mounted directory to a Ceph S3 bucket. The speed is only around 35MB/s. However, when I mount the Ceph S3 bucket to the same client machine as an NFS directory and use cp directly to copy files, the speed is approximately 90MB/s. And use rclone copy a local dir file to Ceph S3 bucket, the speed is also approximately 70MB/s.

What is your rclone version (output from rclone version)

rclone v1.66.0

• os/type: linux
• os/arch: arm64 (ARMv8 compatible)
• go/version: go1.22.1
• go/linking: static
• go/tags: none

Which OS you are using and how many bits?

Centos 8 Stream 64bit aarch64

Which cloud storage system are you using?

Ceph S3

The command you were trying to run

rclone copy /remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso ceph37:mybucket/ --progress --s3-disable-checksum -vv

A log from the command with the -vv flag

rclone copy /remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso ceph37:mybucket/ --progress --s3-disable-checksum -vv
2024/05/08 09:28:01 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "copy" "/remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso" "ceph37:mybucket/" "--progress" "--s3-disable-checksum" "-vv"]
2024/05/08 09:28:01 DEBUG : Creating backend with remote "/remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso"
2024/05/08 09:28:01 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/05/08 09:28:01 DEBUG : fs cache: adding new entry for parent of "/remote/ns1/CentOS-Stream-8-x86_64-20220328-dvd1.iso", "/remote/ns1"
2024/05/08 09:28:01 DEBUG : Creating backend with remote "ceph37:mybucket/"
2024/05/08 09:28:01 DEBUG : ceph37: detected overridden config - adding "{1SSjr}" suffix to name
2024/05/08 09:28:01 DEBUG : Resolving service "s3" region "us-east-1"
2024/05/08 09:28:01 DEBUG : fs cache: renaming cache item "ceph37:mybucket/" to be canonical "ceph37{1SSjr}:mybucket"
2024/05/08 09:28:01 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Need to transfer - File not found at Destination
2024/05/08 09:28:01 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: disabling buffering because source is local disk
2024/05/08 09:28:02 INFO  : S3 bucket mybucket: Bucket "mybucket" created with ACL ""
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: open chunk writer: started multipart upload: 2~7PlrmybwyWPoTKRkUmkMJAgKK4NUJAC
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Starting multi-thread copy with 2151 chunks of size 5Mi with 4 parallel streams
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 4/2151 (15728640-20971520) size 5Mi starting
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 3/2151 (10485760-15728640) size 5Mi starting
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 1/2151 (0-5242880) size 5Mi starting
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 2/2151 (5242880-10485760) size 5Mi starting
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: Seek from 5242880 to 0
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multipart upload wrote chunk 4 with 5242880 bytes and etag "0d895e9eaa21e337bca9a3fa3e14ba48"
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 4/2151 (15728640-20971520) size 5Mi finished
2024/05/08 09:28:02 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 5/2151 (20971520-26214400) size 5Mi starting
.......
2024/05/08 09:30:47 DEBUG : CentOS-Stream-8-x86_64-20220328-dvd1.iso: multi-thread copy: chunk 116/2151 (602931200-608174080) size 5Mi finished
Transferred:          585 MiB / 10.502 GiB, 5%, 35.626 MiB/s, ETA 4m45s
Transferred:            0 / 1, 0%
Elapsed time:        16.6s
Transferring:
     CentOS-Stream-8-x86_64-20220328-dvd1.iso:  5% /10.502Gi, 35.937Mi/s, 4m4

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

[zhangyeda]

rclone copy file in nfs-mounted dir to ceph bucket:

rclone copy file in local filesystem to ceph bucket:

cp nfs-mounted to ceph s3-mounted:

[ncw]

I suggest you try experimenting with these flags

 --s3-upload-concurrency int   Concurrency for multipart uploads and copies (default 4)
 --s3-chunk-size SizeSuffix   Chunk size to use for uploading (default 5Mi)

Making them larger will help a lot at the cost of more memory usage.

[zhangyeda]

I suggest you try experimenting with these flags

 --s3-upload-concurrency int   Concurrency for multipart uploads and copies (default 4)

 --s3-chunk-size SizeSuffix   Chunk size to use for uploading (default 5Mi)

Making them larger will help a lot at the cost of more memory usage.

Thanks, I will try this.