[Feature]: Upload large files in chunks

[Eidenz]

Feature detail

Currently, uploading is done in a very simple way: upload the whole file directly to the server. However, some users may have limits on their reverse proxy preventing uploading such files.

Uploading in chunks (like LibrePhotos does for exemple) would allow bypassing this problem.

With immich not being able to scan local folders, I think this feature might be needed.

Platform

Server

[bo0tzz]

It probably can't hurt to add the chunking, but I think issues like this should generally be solved on the proxy side. As for watching a folder, we're planning to add an option for that to the cli.

[TheLonelinessOfHS]

This should be implemented in Immich or otherwise, it will result in 1. Additional configuration on the reverse proxy, which I consider as dirty work 2. You cannot increase the size limit of Cloudflare, which is widely used by many people.

[zackpollard]

I think this has extra benefits besides just the reverse proxy side of things. If we have this, then uploading large files on slow or unstable connections would mean they could be resumed later rather than the entire upload failing. Temporary files could be kept around for a configurable amount of time and if the file continues to be uploaded, negotiation between client and server could be handled so that the client resumes from where it was before.

[fyfrey]

For reliable upload we could implement the tus protocol https://tus.io/

[JVT038]

Additionally, chunked uploading would solve an issue for Cloudflare users.
Cloudflare only allows a max upload size of 100MB. Using chunked uploading, large files can still be uploaded and solve the Cloudflare issue.
Currently, when uploading a large file, it shows this error in the logcat:
flutter : ERROR backupAsset: SocketException: Broken pipe (OS Error: Broken pipe, errno = 32), address = immich.mydomain.com, port = 49742 and the file does not get uploaded.

[perryizgr8]

Version 1.52.0 has my PR: #2101

This solves the CloudFlare tunnel problem for me. I can upload ~ 200 MB files now.

[Thinkscape]

#2101 (comment)

[AdamGoodApp]

Any update if we can get chunking working via web? The current solution for mobile which uses req.headers["Transfer-Encoding"] = "chunked"; does not work for web as Transfer-Encoding header is not allowed to be set in browsers.

[fabiannagel]

Are you sure? Seems like only HTTP/2 has banned it for a better alternative: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding

edit: Alright, I see what you mean: https://developer.mozilla.org/en-US/docs/Glossary/Forbidden_header_name

[anh-chu]

Would love to see proper chunked upload since my reverse proxy server is memory/space limited. This means that without chunking, a 400MB file will simply over-utilize the resources of the server, leading to crashes.

[sveken]

Chunking would also allow the use of cloudflare tunnels

[traktuner]

+1 for chunking.
I have fast connections on mobile and on the backend immich server, so uploads from mobile are very fast - on mobile I take 4k videos which have 100+ MB
I would like to put Cloudflare in front of everything because I see some security benefits with that. But the 100 MB Cloudflare limit and non-chunked uploads are not allowing that currently...

[YapWC]

I have tried with Immich App on IOS, on the phone app I believe there is chunking process that helped me with uploading 100+ MB video file with CloudFlare Tunnel (100MB upload limit) as reverse proxy. As of now uploading it directly via the WebUI would still fail but I guess Immich App can act as a temporary solution.

[Hely0n]

You mean the app (and thus the server) already implements chunking?

[Hely0n]

never mind. I just found out that it's not real chunking and that the method of the mobile app won't work in browsers

[meidlinga]

Eventually this could be implemented by integrating tusd. It would also support pausing and resuming uploads.

[bo0tzz]

Folks, if you don't have anything new to add please just upvote the original request.

[jkjustjoshing]

You can also subscribe to this discussion in the sidebar (or at the bottom of the page on mobile) if you want to be notified when there are updates.

[fabiannagel]

I gave tusd a closer look at it seems pretty straightforward to me (at first glance). We'd probably have the frontend use tus-js-client to talk to tusd in another Docker container. Once uploads are completed, there are event-based hooks we can use to notify Immich's backend about a new file. I'll try to dig some more this weekend.

Bonus: uppy.io natively supports tus so we'd be able to build a rich batch-upload UI for the web.

@alextran1502 What's your take on potential future PR here?

[bo0tzz]

tusd in another Docker container

We've been working on decreasing the number of containers in the stack, so this is probably a no-go. However, tus-node-server can be included in the current server as a library.

Another small caveat with tus is that while there is a dart client, it seems somewhat unmaintained. I don't know enough about flutter/dart or tus to evaluate whether that will give problems though.

I think a PR would be welcome, though I would suggest discussing the design more in-depth first.

[fabiannagel]

Good point regarding Flutter support. I suppose you were talking about jjmutumi/tus_client which is mentioned in the tus docs? There's also a fork with some more recent activity and a Flutter plugin that wraps native iOS/Android implementations.

So maybe the first step could be for someone that actually knows Dart/Flutter to have a look-see at those libraries. Anyone?