IPv6 enabled by default in Docker engine v26, causes fatal firewall ERROR

[pdfrg]

Is this urgent?

No

Host OS

Ubuntu server 20.04

CPU arch

aarch64

VPN service provider

Mullvad

What are you using to run the container

docker-compose

What is the version of Gluetun

Running version v3.37.0 built on 2024-01-02T00:01:06.245Z (commit c826707)

What's the problem 🤔

Gluetun shuts down immediately on startup.

Had been running it successfully for 1-2 months. After a reboot, did manual apt update/upgrade, then pulled latest gluetun. Had been running version from prior to v3.38 update. Gluetun would not fully startup due to problem with firewall citing ip6tables (see attached logs). I downgraded to gluetun:v3.37 and problem was unchanged.

After some investigation, some docker updates were installed by apt, including this update from the apt logs:

docker-ce:arm64 (5:25.0.3-1~ubuntu.20.04~focal, 5:26.0.0-1~ubuntu.20.04~focal)

From the Docker engine release notes

Always attempt to enable IPv6 on a container's loopback interface, and only include IPv6 in /etc/hosts if successful. [moby/moby#47062](https://github.com/moby/moby/pull/47062)

Note

By default, IPv6 will remain enabled on a container's loopback interface when the container is not connected to an IPv6-enabled network. For example, containers that are only connected to an IPv4-only network now have the ::1 address on their loopback interface.

To disable IPv6 in a container, use option --sysctl net.ipv6.conf.all.disable_ipv6=1 in the create or run command, or the equivalent sysctls option in the service configuration section of a Compose file.

If IPv6 is not available in a container because it has been explicitly disabled for the container, or the host's networking stack does not have IPv6 enabled (or for any other reason) the container's /etc/hosts file will not include IPv6 entries.

From gluetun wiki
(in order to enable ipv6, I wanted to disable it)

Edit your Gluetun docker-compose.yml and add the sysctls section:

services:
  gluetun:
    # ...
    sysctls:
      - net.ipv6.conf.all.disable_ipv6=0

So I changed my docker-compose.yml file to include ( =1 to disable)

services:
  gluetun:
    # ...
    sysctls:
      - net.ipv6.conf.all.disable_ipv6=1

Recreated the container, now everything is working as before.

Gluetun logs suggest upgrading my kernel, but I am not sure how or if I can update it, as I have a Radxa Rockpi-4 and am on their latest release, which is 4.4.194-10-rk3399-rockchip-gf9d08dbd6762

My initial suspicion was the update to gluetun v3.38, but now it looks like the gluetun upgrade is unrelated.

I am submitting this because when others upgrade their docker install, the same problem may be encountered. v26 was just released on 2024-03-20.

Thank you.

Share your logs (at least 10 lines)

Running version v3.37.0 built on 2024-01-02T00:01:06.245Z (commit c826707)

🔧 Need help? https://github.com/qdm12/gluetun/discussions/new
🐛 Bug? https://github.com/qdm12/gluetun/issues/new
✨ New feature? https://github.com/qdm12/gluetun/issues/new
☕ Discussion? https://github.com/qdm12/gluetun/discussions/new
💻 Email? quentin.mcgaw@gmail.com
💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12
2024-04-01T12:34:00-04:00 INFO [routing] default route found: interface eth0, gateway 172.18.0.1, assigned IP 172.18.0.11 and family v4
2024-04-01T12:34:00-04:00 INFO [routing] local ethernet link found: eth0
2024-04-01T12:34:00-04:00 INFO [routing] local ethernet link found: eth1
2024-04-01T12:34:00-04:00 INFO [routing] local ipnet found: 172.18.0.0/16
2024-04-01T12:34:00-04:00 INFO [routing] local ipnet found: 192.168.92.0/24
2024-04-01T12:34:00-04:00 INFO [routing] local ipnet found: fe80::/64
2024-04-01T12:34:00-04:00 INFO [routing] local ipnet found: fe80::/64
2024-04-01T12:34:00-04:00 INFO [firewall] enabling...
2024-04-01T12:34:00-04:00 ERROR enabling firewall: accept output from fe80::42:acff:fe12:b to fe80::/64: ip6tables is required, please upgrade your kernel to support it
2024-04-01T12:34:00-04:00 INFO Shutdown successful

Share your configuration

No response

[github-actions]

@qdm12 is more or less the only maintainer of this project and works on it in his free time.
Please:

• do not ask for updates, be patient
• 👍 the issue to show your support instead of commenting
  @qdm12 usually checks issues at least once a week, if this is a new urgent bug,
  revert to an older tagged container image

[qdm12]

Thanks for the detailed issue and investigation!
Try having ip6tables working in a container, for example does this work:

docker run -it --rm alpine:3.19
apk add ip6tables
ip6tables -L

[pdfrg]

Doesn't look like it. I'm not exactly proficient in alpine, but I appear to be acting as root and still get this error

/ # id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
/ # apk add ip6tables
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/aarch64/APKINDEX.tar.gz
(1/4) Installing libmnl (1.0.5-r2)
(2/4) Installing libnftnl (1.2.6-r0)
(3/4) Installing libxtables (1.8.10-r3)
(4/4) Installing iptables (1.8.10-r3)
Executing busybox-1.36.1-r15.trigger
OK: 16 MiB in 19 packages
/ # ip6tables -L
ip6tables v1.8.10 (nf_tables): Could not fetch rule set generation id: Permission denied (you must be root)
/ # sudo ip6tables -L
/bin/sh: sudo: not found

[qdm12]

How about, the following, which one does work?

Try 1

docker run -it --rm --cap-add=NET_ADMIN alpine:3.19
apk add ip6tables
ip6tables -L
exit

Try 2

docker run -it --rm --privileged --cap-add=NET_ADMIN alpine:3.19
apk add ip6tables
ip6tables -L
exit

Try 3

docker run -it --rm alpine:3.19
apk add iptables-legacy
ip6tables-legacy -L

Try 4

docker run -it --rm --cap-add=NET_ADMIN alpine:3.19
apk add iptables-legacy
ip6tables-legacy -L

[pdfrg]

Doesn't look like any of them do.

~$ docker run -it --rm --cap-add=NET_ADMIN alpine:3.19
/ # apk add ip6tables
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/aarch64/APKINDEX.tar.gz
(1/4) Installing libmnl (1.0.5-r2)
(2/4) Installing libnftnl (1.2.6-r0)
(3/4) Installing libxtables (1.8.10-r3)
(4/4) Installing iptables (1.8.10-r3)
Executing busybox-1.36.1-r15.trigger
OK: 16 MiB in 19 packages
/ # ip6tables -L
ip6tables v1.8.10 (nf_tables): Could not fetch rule set generation id: Invalid argument
/ # exit

~$ docker run -it --rm --privileged --cap-add=NET_ADMIN alpine:3.19
/ # apk add ip6tables
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/aarch64/APKINDEX.tar.gz
(1/4) Installing libmnl (1.0.5-r2)
(2/4) Installing libnftnl (1.2.6-r0)
(3/4) Installing libxtables (1.8.10-r3)
(4/4) Installing iptables (1.8.10-r3)
Executing busybox-1.36.1-r15.trigger
OK: 16 MiB in 19 packages
/ # ip6tables -L
ip6tables v1.8.10 (nf_tables): Could not fetch rule set generation id: Invalid argument
/ # exit

~$ docker run -it --rm alpine:3.19
/ # apk add iptables-legacy
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/aarch64/APKINDEX.tar.gz
(1/4) Installing libip4tc (1.8.10-r3)
(2/4) Installing libip6tc (1.8.10-r3)
(3/4) Installing libxtables (1.8.10-r3)
(4/4) Installing iptables-legacy (1.8.10-r3)
Executing busybox-1.36.1-r15.trigger
OK: 8 MiB in 19 packages
/ # ip6tables-legacy -L
modprobe: can't change directory to '/lib/modules': No such file or directory
ip6tables v1.8.10 (legacy): can't initialize ip6tables table `filter': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.
/ # exit

~$ docker run -it --rm --cap-add=NET_ADMIN alpine:3.19
/ # apk add iptables-legacy
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/aarch64/APKINDEX.tar.gz
(1/4) Installing libip4tc (1.8.10-r3)
(2/4) Installing libip6tc (1.8.10-r3)
(3/4) Installing libxtables (1.8.10-r3)
(4/4) Installing iptables-legacy (1.8.10-r3)
Executing busybox-1.36.1-r15.trigger
OK: 8 MiB in 19 packages
/ # ip6tables-legacy -L
modprobe: can't change directory to '/lib/modules': No such file or directory
ip6tables v1.8.10 (legacy): can't initialize ip6tables table `filter': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.
/ # exit

[qdm12]

The error ip6tables v1.8.10 (nf_tables): Could not fetch rule set generation id: Invalid argument is possibly unsolvable, except using a workaround described at P0cL4bs/wifipumpkin3#140 (comment)

On the other hand, the error ip6tables v1.8.10 (legacy): can't initialize ip6tables table 'filter': Table does not exist (do you need to insmod?) can possibly be solved in a less ugly way, can you try running on your host:

sudo modprobe ip6table_filter

And run the 3rd or 4th command again?

Sorry I can't help more, this Docker/Apple update is pretty bad (2 other issues popped up as well because of its bad IPv6 support).

[qdm12]

Actually reviewing all this, I think it's just your Kernel not supporting nftables, try running on your host as root: modprobe nf_tables?

[qdm12]

Actually this might just be fixed in the latest image (please pull it and try if it works?), see #2256 (comment) on additional explanations.

[qdm12]

Also fd4689e might help to avoid detecting IPv6 as supported when it's only available on the loopback interface.