Disable bridge n/w masquerading for IPv4, IPv6, or both.

[robmry]

- What I did

Make it possible to disable NAT/masquerading for IPv6 but not IPv4 in a bridge network.

- How I did it

New bridge options:

com.docker.network.bridge.enable_ip4_masquerade
com.docker.network.bridge.enable_ip6_masquerade

The option com.docker.network.bridge.enable_ip_masquerade still exists, if it's false masquerading is disabled for IPv4 and IPv6. If true, the individual options are used.

They're stored as three separate flags - the original option must be present in the store for a rollback, because the old code will panic if it's not present, and there's no no need to discard an existing setting.

- How to verify it

New integration test.

- Description for the changelog

Add bridge options to make it possible to disable NAT/masquerading for IPv4 and IPv6
separately:
- `com.docker.network.bridge.enable_ip4_masquerade`
- `com.docker.network.bridge.enable_ip6_masquerade`
Existing option `com.docker.network.bridge.enable_ip_masquerade=false` still disables
masquerading for both IPv4 and IPv6.

[akerouanton]

As discussed earlier today, I'm not convinced these new parameters, as well as the original enable_ip_masquerade, make much sense.

For 'routed-only' networks (ie. networks with no masquerading), there's currently no way to publish a port without mapping it to the host. On one hand you disable NATing for egress connections, and on the other you're forced to rely on NATing to make a port available for ingress connections. I think what people disabling masquerading really want is to disable NATing altogether (ie. both for ingress and egress).

As I see it, port publishing has two distinct goals:

1. Specify what ports can receive ingress connections;
2. Specify what port to map to the host, when you want connections to be NATed;

As the 2nd goal implies the 1st one, there has never been a clear distinction between both. However, I think we should better support 'routed-only' networks, and thus make it possible to achieve goal 1. alone.

Instead, we could introduce a new set of parameters enable_v4_nat / enable_v6_nat. This would both enable / disable masquerading and change the semantics of port publishing.

[corhere]

I agree with @akerouanton. But if the semantics of port publishing can differ depending on the address family, users will need to affordances to configure port publishing independently for IPv4 vs IPv6. Once case which illustrates the need is where IPv4 is NATed but IPv6 is not. Since we won't be allowing port translation when NAT is disabled, -p 80:8080/tcp would conflict with the IPv6 network config. Conversely, if the user wants port 80/tcp6 published on the container's routable IPv6 address, they are forced to also forward port 80/tcp4 from the host to the container. (And while we're extending the port publishing microformat, it'd be useful to also afford publishing ports to a subset of the container's network attachments. Strawman: -p SRC[:DEST][@NETWORK][/PROTOCOL[4|6]]. If the network name contains a / or \ character, it may be escaped by prepending a \.)

Given that with routed-only networks the user has to "bring their own" ingress routes to forward traffic into the containers, some aspects of how the bridge driver behaves with NAT disabled necessarily form part of the interface contract with the user. It would be to both the benefit of our users and to ourselves to formally define and document that contract. Users will be able to configure their systems in a way that remains compatible with future engine upgrades, and we will confidently be able to make "breaking" changes to implementation details which fall outside of the scope of the contract without concern for breaking well-behaved users. The only downside to having such a contract is that we cannot violate our end of it, ever. So I would want us to plan ahead by designing the network parameters which control the masquerading semantics to be extensible, e.g. by using an enum instead of a boolean in the Engine API. ipv4_gateway_mode=(nat|routed|...)?

[corhere]

Speaking of future directions, I wonder if it'd be worth implementing full pseudo-bridge support where, given the name of the physical interface, libnetwork manages the routing table entries and proxy-ARP config on behalf of the user.

[robmry]

Once case which illustrates the need is where IPv4 is NATed but IPv6 is not. Since we won't be allowing port translation when NAT is disabled, -p 80:8080/tcp would conflict with the IPv6 network config.

From internal discussion - the way to specify [/PROTOCOL[4|6]] in the existing -p syntax is to use a v4/v6 address, which may be 0.0.0.0 or [::]. Not specifying an address means v4 and v6. So, rather than -p 80:8080/tcp4, it's -p 0.0.0.0:80:8080/tcp.

We're all agreed that we need to be able to run with no IPv6 NAT-ing or port-forwarding - just "bring your own" direct routing.

With this PR as-is, disabling masquerading for v6 means outgoing packets from a container get the container's address as source-IP. There will also be a forwarding rule (belonging to the port mapping) that makes the mapped port on the container accessible.

That gives the "bring your own" user direct-routing for IPv6 with no NAT, with firewalling.

The issue is that they also get unnecessary/unwanted access to the IPv6'd container via mapped ports on the host.

I'm not sure if there's value in that but, maybe? (Particularly as untangling the port-mapping may not make it into moby 27.0.)