Traefik v3.0 Let's encrypt TLS Challenge failing when behind a traefik TCP Router

[jan-thoma]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

I have multiple servers in my Network and one public static IP. The firewall is forwarding all traffic to Server 1 on ports 80 and 443. Server 1 has multiple TCP Router in place to send the traffic to Server 2 and Server 3 which look live this:

tcp:
    routers:
        server_2:
            entrypoints:
                - websecure
            rule: HostSNIRegexp(`^.+\.server2\.subdomain\.example\.com$`)
            tls:
                passthrough: true
            service: server_2
    services:
        server_2:
            loadBalancer:
                proxyProtocol:
                    version: 2
                servers:
                    - address: 172.16.10.12:443

and

tcp:
    routers:
        server_3:
            entrypoints:
                - websecure
            rule: HostSNIRegexp(`^.+\.server3\.subdomain\.example\.com$`)
            tls:
                passthrough: true
            service: server_3
    services:
        server_3:
            loadBalancer:
                proxyProtocol:
                    version: 2
                servers:
                    - address: 172.16.10.13:443

On Server 2 and Server 3 lets's encrypt is configures like this:

certificatesResolvers:
    letsencrypt-tls:
        acme:
            email: le@example.com
            storage: /letsencrypt/acme.json
            tlsChallenge: true 

If try to create a Certificate on Server 2 or Server 3 for example www.app.server2.subdomain.example.com and app.server2.subdomain.example.com it always fails for the domain with the www prefix. With a error 400 code.

Edit
it looks like all certification request are failing not only the one with the www prefix, which was reported because it was the first to process.

On Server 1 which is facing directly to the internet the problem is not present.

What did you see instead?

---

What version of Traefik are you using?

Version: 3.0.0
Codename: beaufort
Go version: go1.22.2
Built: 2024-04-29T14:25:59Z
OS/Arch: linux/amd64

What is your environment & configuration?

Server 1

log:
    level: INFO

accessLog:
    filePath: /var/log/traefik/access.log
    format: json
    bufferingSize: 100
    fields:
        defaultMode: keep
        headers:
            defaultMode: keep

api:
    dashboard: true

providers:
    docker:
        endpoint: "unix:///var/run/docker.sock"
        exposedByDefault: false
        network: external
    file:
        directory: /etc/traefik/rules
        watch: true

entryPoints:
    web:
        address: :80
        forwardedHeaders:
            trustedIPs:
                - 172.16.10.11
        http:
            middlewares:
                - crowdsec-bouncer@file
            redirections:
                entryPoint:
                    to: websecure
                    scheme: https
    websecure:
        address: :443
        forwardedHeaders:
            trustedIPs:
                - 172.16.10.11
        http:
            middlewares:
                - crowdsec-bouncer@file
        http3:
            advertisedPort: 443

certificatesResolvers:
    letsencrypt-tls:
        acme:
            email: le@example.com
            storage: /letsencrypt/acme.json
            tlsChallenge: true 

metrics:
    influxDB2:
        address: https://influxdb.sg.t-k-f.ch
        token: ***
        org: ***
        bucket: traefik
        addEntryPointsLabels: true
        addRoutersLabels: true
        addServicesLabels: true
        additionalLabels:
            host: traefik.example.com

Server 2

log:
    level: INFO

accessLog:
    filePath: /var/log/traefik/access.log
    format: json
    bufferingSize: 100
    fields:
        defaultMode: keep
        headers:
            defaultMode: keep

api:
    dashboard: true

providers:
    docker:
        endpoint: "unix:///var/run/docker.sock"
        exposedByDefault: false
        network: external
    file:
        directory: /etc/traefik/rules
        watch: true

entryPoints:
    websecure:
        address: :443
        forwardedHeaders:
            trustedIPs:
                - 172.16.10.12
        proxyProtocol:
            trustedIPs:
                - 172.16.10.11
        http:
            middlewares:
                - crowdsec-bouncer@file
        http3: true

certificatesResolvers:
    letsencrypt-tls:
        acme:
            email: le@example.com
            storage: /letsencrypt/acme.json
            tlsChallenge: true 

metrics:
    influxDB2:
        address: https://influxdb.sg.t-k-f.ch
        token: ***
        org: ***
        bucket: traefik
        addEntryPointsLabels: true
        addRoutersLabels: true
        addServicesLabels: true
        additionalLabels:
            host: traefik.example.com

If applicable, please paste the log output in DEBUG level

No response

[jan-thoma]

This Behaviour was introduced with v3.0.0-rc4 downgrading to v3.0.0-rc3 resolves the problem

[nmengin]

Hello @jan-thoma,

Thanks for reaching out.
We will check the merge back we have done from Traefik v2 Traefik v3.