Why does overlayfs2 have a limit of 125 layers?

[jerrac]

I just spend several hours today merging RUN commands, adding --squash to build commands, and just trying to find a way around the "max depth exceeded" error message.

Basically, I use a lot of different layers to take advantage of buildx's parallel building and to make the best use of the build cache that I can. End result is that I have a bunch of different layers in my final build. Which means I've run into that "max depth exceeded" error message fairly often.

Since my web searches haven't dug up a very thorough explanation of the limit yet, I thought I'd ask a few questions here. I'm sure there are good reasons, and would like to understand them.

Why is the limit 125 layers? (Yes, the real limit is 128, but Docker reserves a couple of them, right?)

What are the downsides of having more layers?

Why is the limit not configurable (as far as I can find)?

Thanks in advance!

[cpuguy83]

This is due to the size of mount options being limited to the page size in the kernel.
The containerd overlayfs driver does have a workaround for their overlayfs driver but it is not implemented in docker's grpahdrivers.

It is not configurable because that is a hard-coded in your kernel, and it wouldn't be portable even if your kernel supported larger pages.

[jerrac]

This is due to the size of mount options being limited to the page size in the kernel.

Ok, so I'm guessing that each layer is mounted to the running container? And all mounts need to fit in a single (or some set number of pages) page of memory?

A follow up question that I'll research on my own as well: Do you know of any good articles that get into the details of how Docker containers actually work? Something that would cover my initial question, along with other details that would be good to know. I'm using Docker enough now that I really should dig in and learn more about it.

Thanks!

[corhere]

Correction: Docker's overlay2 graph driver implements more or less the same workaround. All the layer directory paths need to be squeezed into the mount options for a single mount(2) system call. (A new kernel interface was added in Linux 5.1 which lifts the one-page limit for mount options but neither Docker's nor containerd's overlayfs-powered storage drivers have been updated to use it at this time.)

[jerrac]

Ah. Interesting. So things might improve down the line? Thanks for the update.