Traefik v3.0.0 breaks video streams

[Janhouse]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

Seems like Traefik v3 breaks some http streams. After upgrading to v3.0.0 certain clients were not able to seek videos in Jellyfin. Weirdly it shows "DownstreamStatus": 0, and "RequestContentSize": 0,. No errors on the Jellyfin server side.

Downgrading traefik to 2.11 fixes the issue.

What did you see instead?

Request on traefik 3.0.0

{
...
  "DownstreamContentSize": 0,
  "DownstreamStatus": 0,
  "Duration": 11988789492,
  "OriginContentSize": 0,
  "OriginDuration": 0,
  "OriginStatus": 0,
  "Overhead": 11988789492,
...
  "RequestContentSize": 0,
  "RequestCount": 53985,
...
  "RequestMethod": "GET",
  "RequestPath": "/Videos/6db0e85d20dadacca25d553cc63815c4/stream.mkv...",
  "RequestPort": "-",
  "RequestProtocol": "HTTP/2.0",
  "RequestScheme": "https",
  "RetryAttempts": 0,
  "RouterName": "jellyfin_pub@docker",
  "ServiceAddr": "192.168.128.10:8096",
  "ServiceName": "jellyfin@docker",
  "ServiceURL": "http://192.168.128.10:8096",
  "StartLocal": "2024-05-02T14:40:26.710272631Z",
  "StartUTC": "2024-05-02T14:40:26.710272631Z",
  "TLSCipher": "TLS_AES_128_GCM_SHA256",
  "TLSVersion": "1.3",
  "entryPointName": "extranet",
  "level": "info",
  "msg": "",
  "time": "2024-05-02T14:40:38Z"
}

Same request on traefik 2.11

{
...
  "ClientUsername": "-",
  "DownstreamContentSize": 47530,
  "DownstreamStatus": 206,
  "Duration": 278278129,
  "OriginContentSize": 47530,
  "OriginDuration": 270501887,
  "OriginStatus": 206,
  "Overhead": 7776242,
...
  "RequestContentSize": 0,
  "RequestCount": 422,
...
  "RequestMethod": "GET",
  "RequestPath": "/Videos/6db0e85d20dadacca25d553cc63815c4/stream.mkv...",
  "RequestPort": "-",
  "RequestProtocol": "HTTP/2.0",
  "RequestScheme": "https",
  "RetryAttempts": 0,
  "RouterName": "jellyfin_pub@docker",
  "ServiceAddr": "192.168.128.10:8096",
  "ServiceName": "jellyfin@docker",
  "ServiceURL": {
    "Scheme": "http",
    "Opaque": "",
    "User": null,
    "Host": "192.168.128.10:8096",
    "Path": "",
    "RawPath": "",
    "OmitHost": false,
    "ForceQuery": false,
    "RawQuery": "",
    "Fragment": "",
    "RawFragment": ""
  },
  "StartLocal": "2024-05-02T15:28:39.216701682Z",
  "StartUTC": "2024-05-02T15:28:39.216701682Z",
  "TLSCipher": "TLS_AES_128_GCM_SHA256",
  "TLSVersion": "1.3",
  "entryPointName": "extranet",
  "level": "info",
  "msg": "",
  "time": "2024-05-02T15:28:39Z"
}

What version of Traefik are you using?

$ docker run traefik version
Version:      3.0.0
Codename:     beaufort
Go version:   go1.22.2
Built:        2024-04-29T14:25:59Z
OS/Arch:      linux/amd64

What is your environment & configuration?

      - "traefik.http.routers.jellyfin.rule=Host(`${DOMAIN}`)"
      - "traefik.http.routers.jellyfin.entrypoints=websecure"
      - "traefik.http.services.jellyfin.loadbalancer.server.port=8096"
      - "traefik.http.routers.jellyfin.service=jellyfin"

If applicable, please paste the log output in DEBUG level

No response

[bluepuma77]

Did you see the release notes about the new shorter timeouts because of some CVEs?

[Janhouse]

I don't think it has anything to do with timeouts, because it happens immediately, not after some time.

[nmengin]

Hey @Janhouse,

Thanks for reaching out.

Could you provide a minimal reproducible case (for instance, full Kubernetes manifest to reproduce the issue)?
In the meantime, if any community member can help us find verified steps to reproduce and fix the issue if possible, we would love the help.

[springhack]

Did you use compress middleware?

I have same problem while using compress middleware with filebrowser inside my k3s cluster, even I set includedContentTypes only contain mime types like this:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: auth-body-size
spec:
  compress:
    includedContentTypes:
      - text/xml
      - text/css
      - text/html
      - text/plain
      - text/javascript
      - image/tiff
      - image/svg+xml
      - application/xml
      - application/json
      - application/rss+xml
      - application/javascript
      - application/x-javascript

It will cause status code rewrited from 206 to 200, which cause seeking media progress not work in web browser

So I have to remove the compress middleware entirely to make it work

[springhack]

I write a k8s yaml to reproduce this problem, a simple html and mp4 video, can't seek progress, unless remove the compress middleware:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: test-server
data:
  index.html: |
    <html>
      <body>
        <video controls>
          <source src="video.mp4" type="video/mp4">
        </video>
      </body>
    </html>
  Caddyfile: |
    {
      admin off
      auto_https off
    }
    :8080 {
      handle_path /video.mp4 {
        rewrite * /html/mov_bbb.mp4
        reverse_proxy * https://www.w3schools.com {
          header_up Host {upstream_hostport}
          header_down -*
        }
      }
      file_server * {
        root /etc/caddy
      }
    }

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
  labels:
    app: test
spec:
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      app: test
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
        - image: caddy:latest
          imagePullPolicy: IfNotPresent
          name: test
          ports:
            - containerPort: 8080
              name: test
          volumeMounts:
            - name: config
              mountPath: /etc/caddy
      volumes:
        - name: config
          configMap:
            name: test-server

---
apiVersion: v1
kind: Service
metadata:
  name: test
  labels:
    app: test
spec:
  ports:
    - port: 8080
  selector:
    app: test

---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: test-compress
spec:
  compress:
    includedContentTypes:
      - text/xml
      - text/css
      - text/html
      - text/plain
      - text/javascript
      - image/tiff
      - image/svg+xml
      - application/xml
      - application/json
      - application/rss+xml
      - application/javascript
      - application/x-javascript

---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: test
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`test.dosk.host`)
      kind: Rule
      services:
        - name: test
          port: 8080
      middlewares:
        - name: test-compress

[nmengin]

Hello @springhack,

Thank you for the full test case.

We think the error may be due to the deletion of the automatic content type detection in v3.

Could you give a try to the proposed solution and provide feedback please?

[Janhouse]

@nmengin that did not help, issue persists when using contenttype middleware.

[springhack]

@nmengin add a contenttype middleware won't solve this problem, I read the code and I guess maybe here and here, request with range header go into brotli handler, which contenttype not include in includedContentTypes will write directly to rw without r.rw.WriteHeader(r.statusCode), then the statusCode will be 200 not 206