Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[POC] Implement --auth-proxy for 'serve s3' #7646

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

[POC] Implement --auth-proxy for 'serve s3' #7646

wants to merge 8 commits into from

Conversation

saw-jan
Copy link
Contributor

@saw-jan saw-jan commented Feb 22, 2024 •
edited

What is the purpose of this change?

POC for #7645

Requires: rclone/gofakes3#3

Was the change discussed in an issue or in the forum before?

Checklist

  • I have read the contribution guidelines.
  • I have added tests for all changes in this PR if appropriate.
  • I have added documentation for the changes if appropriate.
  • All commit messages are in house style.
  • I'm done, this Pull Request is ready for review :-)

@saw-jan saw-jan mentioned this pull request Feb 22, 2024
Open
@saw-jan saw-jan changed the title [POC] Implement --proxy-mode for 'rclone serve s3 <remote>' [POC] Implement --auth-proxy for 'serve s3' Feb 26, 2024
saw-jan
saw-jan commented Feb 26, 2024
View reviewed changes
cmd/serve/s3/server.go Outdated
}

// auth does proxy authorization
func (w *Server) auth(accessKeyId string) (value interface{}, err error) {
Copy link
Contributor Author

@saw-jan saw-jan Feb 26, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To test, use this example proxy:

#!/usr/bin/env python3
"""
A proxy for rclone serve s3

This takes the incoming access_key_id
and converts it into an webdav backend with bearer_token auth
"""

import sys
import json

def main():
    i = json.load(sys.stdin)
    o = {
        "type": "webdav",            # type of backend
        "_root": "",                 # root of the fs
        # NOTE: adding user doesn't work with bearer_token (?)
        # "user": i['user'],
        "bearer_token": i['pass'],
        "url": "https://localhost:9200/remote.php/webdav",
    }
    json.dump(o, sys.stdout, indent="\t")

if __name__ == "__main__":
    main()

Copy link
Contributor Author

@saw-jan saw-jan Feb 26, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ncw Could you have a look at whether the --auth-proxy implementation is correct or not?

saw-jan
saw-jan commented Feb 26, 2024
View reviewed changes
cmd/serve/s3/server.go Outdated

// auth does proxy authorization
func (w *Server) auth(accessKeyId string) (value interface{}, err error) {
VFS, _, err := w.proxy.Call(stringToMd5Hash(accessKeyId), accessKeyId, false)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stringToMd5Hash(accessKeyId) is used to generate short unique name from the accessKey which will be used as fs name

@ncw
Copy link
Member

ncw commented Mar 23, 2024

This looks like a great feature - sorry I didn't look at it before. It looks like it is doing the right thing but I have't given it a detailed review.

To ensure correctness we need it to run through the auth proxy tests.

Here is how it is done for SFTP

rclone/cmd/serve/sftp/sftp_test.go

Lines 39 to 41 in bf494d4

// TestSftp runs the sftp server then runs the unit tests for the
// sftp remote against it.
func TestSftp(t *testing.T) {

Note in particular this line which calls the tests with and without the auth proxy

rclone/cmd/serve/sftp/sftp_test.go

Line 72 in bf494d4

servetest.Run(t, "sftp", start)

Ideally we'd get serve s3 to use the same framework.

saw-jan
saw-jan commented Apr 3, 2024
View reviewed changes
cmd/serve/s3/s3_test.go Outdated
Comment on lines 231 to 225
func testListBuckets(t *testing.T, useProxy bool) {
fstest.Initialise()

var f fs.Fs
if useProxy {
Copy link
Contributor Author

@saw-jan saw-jan Apr 3, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could not make this test to work in s3

rclone/cmd/serve/sftp/sftp_test.go

Lines 39 to 41 in bf494d4

// TestSftp runs the sftp server then runs the unit tests for the
// sftp remote against it.
func TestSftp(t *testing.T) {

So instead I tried to add tests similar to

rclone/cmd/serve/http/http_test.go

Lines 92 to 93 in bf494d4

func testGET(t *testing.T, useProxy bool) {

CC @ncw

@saw-jan saw-jan force-pushed the feat/cmd-s3-serve-proxy branch 2 times, most recently from ab312d1 to eda44cb Compare April 3, 2024 10:22
@saw-jan saw-jan marked this pull request as ready for review April 15, 2024 04:13
@saw-jan
Copy link
Contributor Author

saw-jan commented Apr 15, 2024

@ncw I have updated the PR, could you have a look?

@ncw
Copy link
Member

ncw commented Apr 15, 2024

I could not make this test to work in s3

rclone/cmd/serve/sftp/sftp_test.go

Lines 39 to 41 in bf494d4

// TestSftp runs the sftp server then runs the unit tests for the
// sftp remote against it.
func TestSftp(t *testing.T) {

What was the problem? Doing this would save having to duplicate the functionality tests making it fit into the same framework at the other --auth-proxy servers would be very useful.

saw-jan
saw-jan commented Apr 16, 2024
View reviewed changes
cmd/serve/s3/s3_test.go
Comment on lines +366 to 388
func TestS3Integration(t *testing.T) {
// Configure and start the server
start := func(f fs.Fs) (configmap.Simple, func()) {
testURL, keyid, keysec, w := serveS3(f)

// Config for the backend we'll use to connect to the server
config := configmap.Simple{
"type": "s3",
"provider": "Rclone",
"endpoint": testURL,
"access_key_id": keyid,
"secret_access_key": keysec,
}

// return a stop function
return config, func() {
assert.NoError(t, w.server.Shutdown())
w.server.Wait()
}
}

servetest.Run(t, "s3", start)
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added the test similar to

rclone/cmd/serve/sftp/sftp_test.go

Lines 39 to 41 in bf494d4

// TestSftp runs the sftp server then runs the unit tests for the
// sftp remote against it.
func TestSftp(t *testing.T) {

Test failure: https://github.com/rclone/rclone/actions/runs/8703549883/job/23869967900?pr=7646

Full log: 
=== RUN   TestS3Integration
=== RUN   TestS3Integration/Normal
2024/04/16 15:51:37 NOTICE: Local file system at /tmp/rclone3384403052: Starting s3 server on [http://127.0.0.1:39719/]
2024/04/16 15:51:38 ERROR : serve s3: unexpected EOF
2024/04/16 15:51:38 ERROR : rclone-test-pofeqaq5dugicem0vemenuj4/hello? sausage/êé/: Dir.Remove not empty
2024/04/16 15:51:38 ERROR : rclone-test-pofeqaq5dugicem0vemenuj4/hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/: Dir.Remove not empty
2024/04/16 15:51:38 ERROR : rclone-test-pofeqaq5dugicem0vemenuj4/: Dir.Remove not empty
    servetest.go:97: 
        ----------
        === RUN   TestVersionLess
        --- PASS: TestVersionLess (0.00s)
        === RUN   TestMergeDeleteMarkers
        --- PASS: TestMergeDeleteMarkers (0.00s)
        === RUN   TestIntegration
            fstests.go:432: Using remote "s3test:"
        === RUN   TestIntegration/FsCheckWrap
            fstests.go:473: Not a wrapping Fs
        === RUN   TestIntegration/FsCommand
        === RUN   TestIntegration/FsRmdirNotFound
        === RUN   TestIntegration/FsString
        === RUN   TestIntegration/FsName
        === RUN   TestIntegration/FsRoot
        === RUN   TestIntegration/FsRmdirEmpty
        === RUN   TestIntegration/FsMkdir
        === RUN   TestIntegration/FsMkdir/FsMkdirRmdirSubdir
            fstest.go:244: Filtering empty directory "dir"
            fstest.go:244: Filtering empty directory "dir/subdir"
            fstest.go:244: Filtering empty directory "dir"
        === RUN   TestIntegration/FsMkdir/FsListEmpty
        === RUN   TestIntegration/FsMkdir/FsListDirEmpty
        === RUN   TestIntegration/FsMkdir/FsListRDirEmpty
        === RUN   TestIntegration/FsMkdir/FsListDirNotFound
        === RUN   TestIntegration/FsMkdir/FsListRDirNotFound
        === RUN   TestIntegration/FsMkdir/FsEncoding
        === RUN   TestIntegration/FsMkdir/FsEncoding/control_chars
            fstests.go:701: testing "␀␁␂␃␄␅␆␇␈␉␊␋␌␍␎␏␐␑␒␓␔␕␖␗␘␙␚␛␜␝␞␟␡"
        === RUN   TestIntegration/FsMkdir/FsEncoding/dot
            fstests.go:701: testing "．"
        === RUN   TestIntegration/FsMkdir/FsEncoding/dot_dot
            fstests.go:701: testing "．．"
        === RUN   TestIntegration/FsMkdir/FsEncoding/punctuation
            fstests.go:701: testing "!\"#$%&'()*+,-.／:;<=>?@[\\]^_`{|}~"
        === RUN   TestIntegration/FsMkdir/FsEncoding/leading_space
            fstests.go:701: testing " leading space"
        === RUN   TestIntegration/FsMkdir/FsEncoding/leading_tilde
            fstests.go:701: testing "~leading tilde"
        === RUN   TestIntegration/FsMkdir/FsEncoding/leading_CR
            fstests.go:701: testing "␍leading CR"
        === RUN   TestIntegration/FsMkdir/FsEncoding/leading_LF
            fstests.go:701: testing "␊leading LF"
        === RUN   TestIntegration/FsMkdir/FsEncoding/leading_HT
            fstests.go:701: testing "␉leading HT"
        === RUN   TestIntegration/FsMkdir/FsEncoding/leading_VT
            fstests.go:701: testing "␋leading VT"
        === RUN   TestIntegration/FsMkdir/FsEncoding/leading_dot
            fstests.go:701: testing ".leading dot"
        === RUN   TestIntegration/FsMkdir/FsEncoding/trailing_space
            fstests.go:701: testing "trailing space "
        === RUN   TestIntegration/FsMkdir/FsEncoding/trailing_CR
            fstests.go:701: testing "trailing CR␍"
        === RUN   TestIntegration/FsMkdir/FsEncoding/trailing_LF
            fstests.go:701: testing "trailing LF␊"
        === RUN   TestIntegration/FsMkdir/FsEncoding/trailing_HT
            fstests.go:701: testing "trailing HT␉"
        === RUN   TestIntegration/FsMkdir/FsEncoding/trailing_VT
            fstests.go:701: testing "trailing VT␋"
        === RUN   TestIntegration/FsMkdir/FsEncoding/trailing_dot
            fstests.go:701: testing "trailing dot."
        === RUN   TestIntegration/FsMkdir/FsEncoding/invalid_UTF-8
            fstests.go:701: testing "invalid utf-8\xfe"
        === RUN   TestIntegration/FsMkdir/FsEncoding/URL_encoding
            fstests.go:701: testing "test%46.txt"
        === RUN   TestIntegration/FsMkdir/FsNewObjectNotFound
        === RUN   TestIntegration/FsMkdir/FsPutError
        === RUN   TestIntegration/FsMkdir/FsPutZeroLength
        === RUN   TestIntegration/FsMkdir/FsOpenWriterAt
            fstests.go:774: FS has no OpenWriterAt interface
        === RUN   TestIntegration/FsMkdir/FsOpenChunkWriter
        2024/04/16 15:51:38 ERROR : writer-at-subdir/writer-at-file: Don't know how to set key "chunkSize" on upload
        2024/04/16 15:51:38 NOTICE: S3 bucket rclone-test-pofeqaq5dugicem0vemenuj4: Streaming uploads using chunk size 5Mi will have maximum file size of 48.828Gi
        === RUN   TestIntegration/FsMkdir/FsChangeNotify
            fstests.go:856: FS has no ChangeNotify interface
        === RUN   TestIntegration/FsMkdir/FsPutFiles
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsListDirFile2
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsListRDirFile2
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsListR
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsListRSubdir
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsListDirRoot
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsListRDirRoot
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsListSubdir
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsListRSubdir#01
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsListLevel2
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsListRLevel2
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsListFile1
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsNewObject
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsNewObjectCaseInsensitive
            fstests.go:1109: Not Case Insensitive
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsListFile1and2
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsNewObjectDir
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsPurge
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsPurgeRoot
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsListRootedSubdir
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsCopy
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsCopy/Metadata
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsMove
            fstests.go:1330: FS has no Mover interface
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsDirMove
            fstests.go:1447: FS has no DirMover interface
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsRmdirFull
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsPrecision
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectString
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectFs
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectRemote
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectHashes
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectModTime
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectMimeType
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectMetadata
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectMetadata/mtime
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectSetModTime
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectSize
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectOpen
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectOpenSeek
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectOpenRange
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectPartialRead
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectUpdate
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectStorable
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsIsFile
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsIsFile/FsRoot
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsIsFileNotFound
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FromRoot
            fstests.go:1838: Opening root remote "s3test:" path "rclone-test-pofeqaq5dugicem0vemenuj4" from "s3test:rclone-test-pofeqaq5dugicem0vemenuj4"
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FromRoot/List
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListEntries
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListR
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put/Remove
        === RUN   TestIntegration/FsMkdir/FsPutFiles/PublicLink
            fstests.go:1979: skipping directory tests as not supported on this backend
        === RUN   TestIntegration/FsMkdir/FsPutFiles/SetTier
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectCheckWrap
            fstests.go:2036: Not a wrapping Fs
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectRemove
        === RUN   TestIntegration/FsMkdir/FsPutFiles/ObjectAbout
            fstests.go:2070: FS does not support About
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsPutStream
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsPutStream/0
        === RUN   TestIntegration/FsMkdir/FsPutFiles/FsPutStream/100
        === RUN   TestIntegration/FsMkdir/FsPutFiles/Internal
        === RUN   TestIntegration/FsMkdir/FsPutFiles/Internal/Metadata
        === RUN   TestIntegration/FsMkdir/FsPutFiles/Internal/Metadata/GzipEncoding
        === RUN   TestIntegration/FsMkdir/FsPutFiles/Internal/Metadata/GzipEncoding/NoDecompress
        2024/04/16 15:51:38 NOTICE: test-metadata: Not decompressing 'Content-Encoding: gzip' compressed file. Use --s3-decompress to override
        === RUN   TestIntegration/FsMkdir/FsPutFiles/Internal/Metadata/GzipEncoding/Decompress
        === RUN   TestIntegration/FsMkdir/FsPutFiles/Internal/NoHead
        === RUN   TestIntegration/FsMkdir/FsPutFiles/Internal/Versions
            s3_internal_test.go:245: Couldn't enable versioning: NotImplemented: NotImplemented
                	status code: 501, request id: 805750E74EC52774, host id: ODA1NzUwRTc0RUM1Mjc3NDgwNTc1MEU3NEVDNTI3NzQ4MDU3NTBFNzRFQzUyNzc0ODA1NzUwRTc0RUM1Mjc3NA==
        === RUN   TestIntegration/FsMkdir/FsPutChunked
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5Mi
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5Mi/5242879
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5Mi/5242879/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5Mi/5242880
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5Mi/5242880/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5Mi/10485761
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5Mi/10485761/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5.054Mi
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5.054Mi/5299999
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5.054Mi/5299999/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5.054Mi/5300000
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5.054Mi/5300000/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5.054Mi/10600001
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5.054Mi/10600001/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5.055Mi
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5.055Mi/5300052
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5.055Mi/5300052/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5.055Mi/5300053
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5.055Mi/5300053/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5.055Mi/10600107
        === RUN   TestIntegration/FsMkdir/FsPutChunked/5.055Mi/10600107/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/6.667Mi
        === RUN   TestIntegration/FsMkdir/FsPutChunked/6.667Mi/6990505
        === RUN   TestIntegration/FsMkdir/FsPutChunked/6.667Mi/6990505/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/6.667Mi/6990506
        === RUN   TestIntegration/FsMkdir/FsPutChunked/6.667Mi/6990506/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/6.667Mi/13981013
        === RUN   TestIntegration/FsMkdir/FsPutChunked/6.667Mi/13981013/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/8Mi
        === RUN   TestIntegration/FsMkdir/FsPutChunked/8Mi/8388607
        === RUN   TestIntegration/FsMkdir/FsPutChunked/8Mi/8388607/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/8Mi/8388608
        === RUN   TestIntegration/FsMkdir/FsPutChunked/8Mi/8388608/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/8Mi/16777217
        === RUN   TestIntegration/FsMkdir/FsPutChunked/8Mi/16777217/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/10Mi
        === RUN   TestIntegration/FsMkdir/FsPutChunked/10Mi/10485759
        === RUN   TestIntegration/FsMkdir/FsPutChunked/10Mi/10485759/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/10Mi/10485760
        === RUN   TestIntegration/FsMkdir/FsPutChunked/10Mi/10485760/Streamed
        === RUN   TestIntegration/FsMkdir/FsPutChunked/10Mi/20971521
        === RUN   TestIntegration/FsMkdir/FsPutChunked/10Mi/20971521/Streamed
        === RUN   TestIntegration/FsMkdir/FsCopyChunked
        === RUN   TestIntegration/FsMkdir/FsCopyChunked/5242880
            fstests.go:2324: 
                	Error Trace:	/mnt/workspace/JT/rclone/fstest/fstests/fstests.go:2324
                	Error:      	Received unexpected error:
                	            	SerializationError: empty response payload
                	            		status code: 200, request id: 805750E74EC52897, host id: ODA1NzUwRTc0RUM1Mjg5NzgwNTc1MEU3NEVDNTI4OTc4MDU3NTBFNzRFQzUyODk3ODA1NzUwRTc0RUM1Mjg5Nw==
                	            	caused by: EOF
                	Test:       	TestIntegration/FsMkdir/FsCopyChunked/5242880
        === RUN   TestIntegration/FsMkdir/FsCopyChunked/5242881
            fstests.go:2324: 
                	Error Trace:	/mnt/workspace/JT/rclone/fstest/fstests/fstests.go:2324
                	Error:      	Received unexpected error:
                	            	SerializationError: empty response payload
                	            		status code: 200, request id: 805750E74EC528C9, host id: ODA1NzUwRTc0RUM1MjhDOTgwNTc1MEU3NEVDNTI4Qzk4MDU3NTBFNzRFQzUyOEM5ODA1NzUwRTc0RUM1MjhDOQ==
                	            	caused by: EOF
                	Test:       	TestIntegration/FsMkdir/FsCopyChunked/5242881
        === RUN   TestIntegration/FsMkdir/FsCopyChunked/10485759
            fstests.go:2324: 
                	Error Trace:	/mnt/workspace/JT/rclone/fstest/fstests/fstests.go:2324
                	Error:      	Received unexpected error:
                	            	SerializationError: empty response payload
                	            		status code: 200, request id: 805750E74EC528FB, host id: ODA1NzUwRTc0RUM1MjhGQjgwNTc1MEU3NEVDNTI4RkI4MDU3NTBFNzRFQzUyOEZCODA1NzUwRTc0RUM1MjhGQg==
                	            	caused by: EOF
                	Test:       	TestIntegration/FsMkdir/FsCopyChunked/10485759
        === RUN   TestIntegration/FsMkdir/FsCopyChunked/10485760
            fstests.go:2324: 
                	Error Trace:	/mnt/workspace/JT/rclone/fstest/fstests/fstests.go:2324
                	Error:      	Received unexpected error:
                	            	SerializationError: empty response payload
                	            		status code: 200, request id: 805750E74EC5292D, host id: ODA1NzUwRTc0RUM1MjkyRDgwNTc1MEU3NEVDNTI5MkQ4MDU3NTBFNzRFQzUyOTJEODA1NzUwRTc0RUM1MjkyRA==
                	            	caused by: EOF
                	Test:       	TestIntegration/FsMkdir/FsCopyChunked/10485760
        === RUN   TestIntegration/FsMkdir/FsCopyChunked/10485761
            fstests.go:2324: 
                	Error Trace:	/mnt/workspace/JT/rclone/fstest/fstests/fstests.go:2324
                	Error:      	Received unexpected error:
                	            	SerializationError: empty response payload
                	            		status code: 200, request id: 805750E74EC52974, host id: ODA1NzUwRTc0RUM1Mjk3NDgwNTc1MEU3NEVDNTI5NzQ4MDU3NTBFNzRFQzUyOTc0ODA1NzUwRTc0RUM1Mjk3NA==
                	            	caused by: EOF
                	Test:       	TestIntegration/FsMkdir/FsCopyChunked/10485761
        === RUN   TestIntegration/FsMkdir/FsUploadUnknownSize
        === RUN   TestIntegration/FsMkdir/FsUploadUnknownSize/FsPutUnknownSize
        === RUN   TestIntegration/FsMkdir/FsUploadUnknownSize/FsUpdateUnknownSize
        === RUN   TestIntegration/FsMkdir/FsRootCollapse
        === RUN   TestIntegration/FsMkdir/FsDirSetModTime
            fstests.go:2420: FS has no DirSetModTime interface
        === RUN   TestIntegration/FsMkdir/FsMkdirMetadata
            fstests.go:2463: FS has no MkdirMetadata interface
        === RUN   TestIntegration/FsMkdir/FsDirectory
            fstests.go:2512: Can't test if can't have empty directories
        === RUN   TestIntegration/FsShutdown
            fstests.go:2626: Shutdown method not supported
        --- FAIL: TestIntegration (462.25s)
            --- SKIP: TestIntegration/FsCheckWrap (0.00s)
            --- PASS: TestIntegration/FsCommand (0.00s)
            --- PASS: TestIntegration/FsRmdirNotFound (0.00s)
            --- PASS: TestIntegration/FsString (0.00s)
            --- PASS: TestIntegration/FsName (0.00s)
            --- PASS: TestIntegration/FsRoot (0.00s)
            --- PASS: TestIntegration/FsRmdirEmpty (0.00s)
            --- FAIL: TestIntegration/FsMkdir (462.25s)
                --- PASS: TestIntegration/FsMkdir/FsMkdirRmdirSubdir (0.00s)
                --- PASS: TestIntegration/FsMkdir/FsListEmpty (0.00s)
                --- PASS: TestIntegration/FsMkdir/FsListDirEmpty (0.00s)
                --- PASS: TestIntegration/FsMkdir/FsListRDirEmpty (0.00s)
                --- PASS: TestIntegration/FsMkdir/FsListDirNotFound (0.00s)
                --- PASS: TestIntegration/FsMkdir/FsListRDirNotFound (0.00s)
                --- PASS: TestIntegration/FsMkdir/FsEncoding (0.08s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/control_chars (0.04s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/dot (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/dot_dot (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/punctuation (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/leading_space (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/leading_tilde (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/leading_CR (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/leading_LF (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/leading_HT (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/leading_VT (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/leading_dot (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/trailing_space (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/trailing_CR (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/trailing_LF (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/trailing_HT (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/trailing_VT (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/trailing_dot (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/invalid_UTF-8 (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsEncoding/URL_encoding (0.00s)
                --- PASS: TestIntegration/FsMkdir/FsNewObjectNotFound (0.00s)
                --- PASS: TestIntegration/FsMkdir/FsPutError (0.00s)
                --- PASS: TestIntegration/FsMkdir/FsPutZeroLength (0.00s)
                --- SKIP: TestIntegration/FsMkdir/FsOpenWriterAt (0.00s)
                --- PASS: TestIntegration/FsMkdir/FsOpenChunkWriter (0.23s)
                --- SKIP: TestIntegration/FsMkdir/FsChangeNotify (0.00s)
                --- PASS: TestIntegration/FsMkdir/FsPutFiles (0.17s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListDirFile2 (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListRDirFile2 (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListR (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListRSubdir (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListDirRoot (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListRDirRoot (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListSubdir (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListRSubdir#01 (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListLevel2 (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListRLevel2 (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListFile1 (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsNewObject (0.00s)
                    --- SKIP: TestIntegration/FsMkdir/FsPutFiles/FsNewObjectCaseInsensitive (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListFile1and2 (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsNewObjectDir (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsPurge (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsPurgeRoot (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsListRootedSubdir (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsCopy (0.01s)
                        --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsCopy/Metadata (0.00s)
                    --- SKIP: TestIntegration/FsMkdir/FsPutFiles/FsMove (0.00s)
                    --- SKIP: TestIntegration/FsMkdir/FsPutFiles/FsDirMove (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsRmdirFull (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsPrecision (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectString (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectFs (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectRemote (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectHashes (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectModTime (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectMimeType (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectMetadata (0.00s)
                        --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectMetadata/mtime (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectSetModTime (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectSize (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectOpen (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectOpenSeek (0.02s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectOpenRange (0.08s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectPartialRead (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectUpdate (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectStorable (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsIsFile (0.00s)
                        --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsIsFile/FsRoot (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsIsFileNotFound (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FromRoot (0.01s)
                        --- PASS: TestIntegration/FsMkdir/FsPutFiles/FromRoot/List (0.00s)
                        --- PASS: TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListEntries (0.00s)
                        --- PASS: TestIntegration/FsMkdir/FsPutFiles/FromRoot/ListR (0.00s)
                        --- PASS: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put (0.00s)
                            --- PASS: TestIntegration/FsMkdir/FsPutFiles/FromRoot/Put/Remove (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/PublicLink (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/SetTier (0.00s)
                    --- SKIP: TestIntegration/FsMkdir/FsPutFiles/ObjectCheckWrap (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/ObjectRemove (0.00s)
                    --- SKIP: TestIntegration/FsMkdir/FsPutFiles/ObjectAbout (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsPutStream (0.00s)
                        --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsPutStream/0 (0.00s)
                        --- PASS: TestIntegration/FsMkdir/FsPutFiles/FsPutStream/100 (0.00s)
                    --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal (0.00s)
                        --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal/Metadata (0.00s)
                            --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal/Metadata/GzipEncoding (0.00s)
                                --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal/Metadata/GzipEncoding/NoDecompress (0.00s)
                                --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal/Metadata/GzipEncoding/Decompress (0.00s)
                        --- PASS: TestIntegration/FsMkdir/FsPutFiles/Internal/NoHead (0.00s)
                        --- SKIP: TestIntegration/FsMkdir/FsPutFiles/Internal/Versions (0.00s)
                --- PASS: TestIntegration/FsMkdir/FsPutChunked (20.45s)
                    --- PASS: TestIntegration/FsMkdir/FsPutChunked/5Mi (2.58s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/5Mi/5242879 (0.63s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/5Mi/5242879/Streamed (0.34s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/5Mi/5242880 (0.67s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/5Mi/5242880/Streamed (0.34s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/5Mi/10485761 (1.28s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/5Mi/10485761/Streamed (0.64s)
                    --- PASS: TestIntegration/FsMkdir/FsPutChunked/5.054Mi (2.63s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/5.054Mi/5299999 (0.64s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/5.054Mi/5299999/Streamed (0.35s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/5.054Mi/5300000 (0.69s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/5.054Mi/5300000/Streamed (0.34s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/5.054Mi/10600001 (1.31s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/5.054Mi/10600001/Streamed (0.65s)
                    --- PASS: TestIntegration/FsMkdir/FsPutChunked/5.055Mi (2.59s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/5.055Mi/5300052 (0.63s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/5.055Mi/5300052/Streamed (0.34s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/5.055Mi/5300053 (0.67s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/5.055Mi/5300053/Streamed (0.34s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/5.055Mi/10600107 (1.29s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/5.055Mi/10600107/Streamed (0.64s)
                    --- PASS: TestIntegration/FsMkdir/FsPutChunked/6.667Mi (3.44s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/6.667Mi/6990505 (0.83s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/6.667Mi/6990505/Streamed (0.44s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/6.667Mi/6990506 (0.89s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/6.667Mi/6990506/Streamed (0.45s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/6.667Mi/13981013 (1.72s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/6.667Mi/13981013/Streamed (0.86s)
                    --- PASS: TestIntegration/FsMkdir/FsPutChunked/8Mi (4.11s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/8Mi/8388607 (1.00s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/8Mi/8388607/Streamed (0.54s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/8Mi/8388608 (1.08s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/8Mi/8388608/Streamed (0.54s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/8Mi/16777217 (2.04s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/8Mi/16777217/Streamed (1.02s)
                    --- PASS: TestIntegration/FsMkdir/FsPutChunked/10Mi (5.10s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/10Mi/10485759 (1.24s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/10Mi/10485759/Streamed (0.67s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/10Mi/10485760 (1.33s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/10Mi/10485760/Streamed (0.66s)
                        --- PASS: TestIntegration/FsMkdir/FsPutChunked/10Mi/20971521 (2.52s)
                            --- PASS: TestIntegration/FsMkdir/FsPutChunked/10Mi/20971521/Streamed (1.25s)
                --- FAIL: TestIntegration/FsMkdir/FsCopyChunked (441.20s)
                    --- FAIL: TestIntegration/FsMkdir/FsCopyChunked/5242880 (91.19s)
                    --- FAIL: TestIntegration/FsMkdir/FsCopyChunked/5242881 (80.99s)
                    --- FAIL: TestIntegration/FsMkdir/FsCopyChunked/10485759 (90.87s)
                    --- FAIL: TestIntegration/FsMkdir/FsCopyChunked/10485760 (84.96s)
                    --- FAIL: TestIntegration/FsMkdir/FsCopyChunked/10485761 (93.18s)
                --- PASS: TestIntegration/FsMkdir/FsUploadUnknownSize (0.11s)
                    --- PASS: TestIntegration/FsMkdir/FsUploadUnknownSize/FsPutUnknownSize (0.11s)
                    --- PASS: TestIntegration/FsMkdir/FsUploadUnknownSize/FsUpdateUnknownSize (0.00s)
                --- PASS: TestIntegration/FsMkdir/FsRootCollapse (0.00s)
                --- SKIP: TestIntegration/FsMkdir/FsDirSetModTime (0.00s)
                --- SKIP: TestIntegration/FsMkdir/FsMkdirMetadata (0.00s)
                --- SKIP: TestIntegration/FsMkdir/FsDirectory (0.00s)
            --- SKIP: TestIntegration/FsShutdown (0.00s)
        === RUN   TestIntegration2
            s3_test.go:36: skipping as -remote is set
        --- SKIP: TestIntegration2 (0.00s)
        === RUN   TestAWSDualStackOption
        --- PASS: TestAWSDualStackOption (0.00s)
        FAIL
        exit status 1
        FAIL	github.com/rclone/rclone/backend/s3	462.270s
        ----------
    servetest.go:99: 
        	Error Trace:	/mnt/workspace/JT/rclone/cmd/serve/servetest/servetest.go:99
        	            				/mnt/workspace/JT/rclone/cmd/serve/servetest/servetest.go:107
        	Error:      	Received unexpected error:
        	            	exit status 1
        	Test:       	TestS3Integration/Normal
        	Messages:   	Running s3 integration tests
=== RUN   TestS3Integration/AuthProxy
2024/04/16 15:59:20 NOTICE: Starting s3 server on [http://127.0.0.1:40443/]
2024/04/16 15:59:21 ERROR : serve s3: unexpected EOF
2024/04/16 15:59:21 ERROR : rclone-test-zorejeq5duyides9cikesok9/hello? sausage/êé/: Dir.Remove not empty
2024/04/16 15:59:21 ERROR : rclone-test-zorejeq5duyides9cikesok9/hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/: Dir.Remove not empty
2024/04/16 15:59:21 ERROR : rclone-test-zorejeq5duyides9cikesok9/: Dir.Remove not empty
panic: test timed out after 10m0s
running tests:
	TestS3Integration (10m0s)
	TestS3Integration/AuthProxy (2m17s)

goroutine 3250 [running]:
testing.(*M).startAlarm.func1()
	/home/sawjan/.gobrew/current/go/src/testing/testing.go:2366 +0x385
created by time.goFunc
	/home/sawjan/.gobrew/current/go/src/time/sleep.go:177 +0x2d

goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc000131040, {0xd3f7d3?, 0x0?}, 0xdcd438)
	/home/sawjan/.gobrew/current/go/src/testing/testing.go:1750 +0x3ab
testing.runTests.func1(0xc000131040)
	/home/sawjan/.gobrew/current/go/src/testing/testing.go:2161 +0x37
testing.tRunner(0xc000131040, 0xc00029fc70)
	/home/sawjan/.gobrew/current/go/src/testing/testing.go:1689 +0xfb
testing.runTests(0xc000339d70, {0x1443f00, 0x5, 0x5}, {0x1?, 0xc0003b6680?, 0x1453f60?})
	/home/sawjan/.gobrew/current/go/src/testing/testing.go:2159 +0x445
testing.(*M).Run(0xc0003d1720)
	/home/sawjan/.gobrew/current/go/src/testing/testing.go:2027 +0x68b
main.main()
	_testmain.go:55 +0x16c

goroutine 21 [chan receive, 2 minutes]:
testing.(*T).Run(0xc0001311e0, {0xd388e5?, 0xc736a0?}, 0xc002ca4100)
	/home/sawjan/.gobrew/current/go/src/testing/testing.go:1750 +0x3ab
github.com/rclone/rclone/cmd/serve/servetest.Run(0xc0001311e0, {0xd324be, 0x2}, 0xc00034bb60)
	/mnt/workspace/JT/rclone/cmd/serve/servetest/servetest.go:109 +0x105
github.com/rclone/rclone/cmd/serve/s3.TestS3Integration(0xc0001311e0)
	/mnt/workspace/JT/rclone/cmd/serve/s3/s3_test.go:387 +0x65
testing.tRunner(0xc0001311e0, 0xdcd438)
	/home/sawjan/.gobrew/current/go/src/testing/testing.go:1689 +0xfb
created by testing.(*T).Run in goroutine 1
	/home/sawjan/.gobrew/current/go/src/testing/testing.go:1742 +0x390

goroutine 35 [syscall, 10 minutes]:
os/signal.signal_recv()
	/home/sawjan/.gobrew/current/go/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
	/home/sawjan/.gobrew/current/go/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 21
	/home/sawjan/.gobrew/current/go/src/os/signal/signal.go:151 +0x1f

goroutine 36 [chan receive, 10 minutes]:
github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler.func1()
	/mnt/workspace/JT/rclone/fs/accounting/accounting_unix.go:25 +0x27
created by github.com/rclone/rclone/fs/accounting.(*tokenBucket).startSignalHandler in goroutine 21
	/mnt/workspace/JT/rclone/fs/accounting/accounting_unix.go:22 +0xab

goroutine 1798 [syscall, 2 minutes]:
syscall.Syscall6(0xf7, 0x1, 0x2fd3b, 0xc000817ba8, 0x1000004, 0x0, 0x0)
	/home/sawjan/.gobrew/current/go/src/syscall/syscall_linux.go:91 +0x39
os.(*Process).blockUntilWaitable(0xc002a160c0)
	/home/sawjan/.gobrew/current/go/src/os/wait_waitid.go:32 +0x76
os.(*Process).wait(0xc002a160c0)
	/home/sawjan/.gobrew/current/go/src/os/exec_unix.go:22 +0x25
os.(*Process).Wait(...)
	/home/sawjan/.gobrew/current/go/src/os/exec.go:134
os/exec.(*Cmd).Wait(0xc000430420)
	/home/sawjan/.gobrew/current/go/src/os/exec/exec.go:897 +0x45
os/exec.(*Cmd).Run(0xc000430420)
	/home/sawjan/.gobrew/current/go/src/os/exec/exec.go:607 +0x2d
os/exec.(*Cmd).CombinedOutput(0xc000430420)
	/home/sawjan/.gobrew/current/go/src/os/exec/exec.go:1012 +0x85
github.com/rclone/rclone/cmd/serve/servetest.run(0xc000228000, {0xd324be, 0x2}, 0xc00034bb60, 0x1)
	/mnt/workspace/JT/rclone/cmd/serve/servetest/servetest.go:95 +0x9c5
github.com/rclone/rclone/cmd/serve/servetest.Run.func2(0xc000228000?)
	/mnt/workspace/JT/rclone/cmd/serve/servetest/servetest.go:110 +0x25
testing.tRunner(0xc000228000, 0xc002ca4100)
	/home/sawjan/.gobrew/current/go/src/testing/testing.go:1689 +0xfb
created by testing.(*T).Run in goroutine 21
	/home/sawjan/.gobrew/current/go/src/testing/testing.go:1742 +0x390

goroutine 51 [chan receive, 10 minutes]:
github.com/rclone/rclone/lib/atexit.Register.func1.1()
	/mnt/workspace/JT/rclone/lib/atexit/atexit.go:45 +0x29
created by github.com/rclone/rclone/lib/atexit.Register.func1 in goroutine 37
	/mnt/workspace/JT/rclone/lib/atexit/atexit.go:44 +0x68

goroutine 1801 [IO wait]:
internal/poll.runtime_pollWait(0x7fb345f825d8, 0x72)
	/home/sawjan/.gobrew/current/go/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0xc0003c8780?, 0xc002ce0e3f?, 0x1)
	/home/sawjan/.gobrew/current/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/home/sawjan/.gobrew/current/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0003c8780, {0xc002ce0e3f, 0x11c1, 0x11c1})
	/home/sawjan/.gobrew/current/go/src/internal/poll/fd_unix.go:164 +0x27a
os.(*File).read(...)
	/home/sawjan/.gobrew/current/go/src/os/file_posix.go:29
os.(*File).Read(0xc00023c168, {0xc002ce0e3f?, 0xc002afe628?, 0x1e0b?})
	/home/sawjan/.gobrew/current/go/src/os/file.go:118 +0x52
bytes.(*Buffer).ReadFrom(0xc002cc2c30, {0xea04e0, 0xc00023c188})
	/home/sawjan/.gobrew/current/go/src/bytes/buffer.go:211 +0x98
io.copyBuffer({0xea0b20, 0xc002cc2c30}, {0xea04e0, 0xc00023c188}, {0x0, 0x0, 0x0})
	/home/sawjan/.gobrew/current/go/src/io/io.go:415 +0x151
io.Copy(...)
	/home/sawjan/.gobrew/current/go/src/io/io.go:388
os.genericWriteTo(0xc00023c168?, {0xea0b20, 0xc002cc2c30})
	/home/sawjan/.gobrew/current/go/src/os/file.go:269 +0x58
os.(*File).WriteTo(0xc00023c168, {0xea0b20, 0xc002cc2c30})
	/home/sawjan/.gobrew/current/go/src/os/file.go:247 +0x9c
io.copyBuffer({0xea0b20, 0xc002cc2c30}, {0xea0540, 0xc00023c168}, {0x0, 0x0, 0x0})
	/home/sawjan/.gobrew/current/go/src/io/io.go:411 +0x9d
io.Copy(...)
	/home/sawjan/.gobrew/current/go/src/io/io.go:388
os/exec.(*Cmd).writerDescriptor.func1()
	/home/sawjan/.gobrew/current/go/src/os/exec/exec.go:577 +0x34
os/exec.(*Cmd).Start.func2(0xc002afe748?)
	/home/sawjan/.gobrew/current/go/src/os/exec/exec.go:724 +0x2c
created by os/exec.(*Cmd).Start in goroutine 1798
	/home/sawjan/.gobrew/current/go/src/os/exec/exec.go:723 +0x9ab

goroutine 2343 [IO wait]:
internal/poll.runtime_pollWait(0x7fb345f82008, 0x72)
	/home/sawjan/.gobrew/current/go/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0xc0002db800?, 0xc000169000?, 0x0)
	/home/sawjan/.gobrew/current/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/home/sawjan/.gobrew/current/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0002db800, {0xc000169000, 0x1000, 0x1000})
	/home/sawjan/.gobrew/current/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc0002db800, {0xc000169000?, 0xc00004fa98?, 0x4ba3e5?})
	/home/sawjan/.gobrew/current/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc000348318, {0xc000169000?, 0x0?, 0xc002b48038?})
	/home/sawjan/.gobrew/current/go/src/net/net.go:179 +0x45
net/http.(*connReader).Read(0xc002b48030, {0xc000169000, 0x1000, 0x1000})
	/home/sawjan/.gobrew/current/go/src/net/http/server.go:789 +0x14b
bufio.(*Reader).fill(0xc003901560)
	/home/sawjan/.gobrew/current/go/src/bufio/bufio.go:110 +0x103
bufio.(*Reader).Peek(0xc003901560, 0x4)
	/home/sawjan/.gobrew/current/go/src/bufio/bufio.go:148 +0x53
net/http.(*conn).serve(0xc002b36120, {0xea7ea0, 0xc0003ba0f0})
	/home/sawjan/.gobrew/current/go/src/net/http/server.go:2074 +0x749
created by net/http.(*Server).Serve in goroutine 1800
	/home/sawjan/.gobrew/current/go/src/net/http/server.go:3285 +0x4b4

goroutine 1800 [IO wait]:
internal/poll.runtime_pollWait(0x7fb345f826d0, 0x72)
	/home/sawjan/.gobrew/current/go/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0x7?, 0x221a?, 0x0)
	/home/sawjan/.gobrew/current/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/home/sawjan/.gobrew/current/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0002da580)
	/home/sawjan/.gobrew/current/go/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc0002da580)
	/home/sawjan/.gobrew/current/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc002ca4320)
	/home/sawjan/.gobrew/current/go/src/net/tcpsock_posix.go:159 +0x1e
net.(*TCPListener).Accept(0xc002ca4320)
	/home/sawjan/.gobrew/current/go/src/net/tcpsock.go:327 +0x30
net/http.(*Server).Serve(0xc0002881e0, {0xea6bd0, 0xc002ca4320})
	/home/sawjan/.gobrew/current/go/src/net/http/server.go:3255 +0x33e
github.com/rclone/rclone/lib/http.instance.serve({{0xc002acca08, 0x17}, {0xea6bd0, 0xc002ca4320}, 0xc0002881e0}, 0xc0003e6748?)
	/mnt/workspace/JT/rclone/lib/http/server.go:153 +0xa5
created by github.com/rclone/rclone/lib/http.(*Server).Serve in goroutine 1798
	/mnt/workspace/JT/rclone/lib/http/server.go:423 +0x66
FAIL	github.com/rclone/rclone/cmd/serve/s3	600.024s
FAIL
make: *** [Makefile:90: quicktest] Error 1

@nanawel
Copy link

nanawel commented May 6, 2024

You might be interested in taking a look at my comment here #7645 (comment) regarding URL-encoded keys/filenames at least in v1.66.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@saw-jan @ncw @nanawel