Reloading haproxy causes endpoints to incorrectly respond with a version requirement error. #289

fatchan · 2023-02-19T04:03:09Z

Hi,

After reloading haproxy e.g. sudo systemctl reload haproxy, a POST to v2/services/haproxy/runtime/servers?backend=whatever endpoint will return {"code":500,"message":"this operation requires HAProxy 2.6 or later"}.

However, I am on haproxy 2.7. This api call works correctly before reloading, or after restarting rather than reloading haproxy.

This is an issue for me because after I add a certificate with dataplaneapi and haproxy reloads, some future API calls get broken with the "requires haproxy 2.6 or later" error.

For now, I have worked around the issue by ~~setting the dataplaneapi.hcl reload_cmd to actually restart haproxy instead~~ running my own build of dataplaneapi with the version check from client-native removed, which is not ideal.

Any ideas?

Kind regards.

haproxy -vv

HAProxy version 2.7.2-1~bpo11+1 2023/01/20 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2024.
Known bugs: http://www.haproxy.org/bugs/bugs-2.7.2.html
Running on: Linux 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_SYSTEMD=1 USE_PROMEX=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY +LUA -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC +RT +SHM_OPEN +SLZ -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=1).
Built with OpenSSL version : OpenSSL 1.1.1n  15 Mar 2022
Running on OpenSSL version : OpenSSL 1.1.1n  15 Mar 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Support for malloc_trim() is enabled.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.36 2020-12-04
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 10.2.1 20210110

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE]

dataplaneapi -v

HAProxy Data Plane API v2.7.2 3d8f41f

Build from: https://github.com/haproxytech/dataplaneapi
Build date: 2023-02-03T13:31:57Z

Update: did some grep'ing and realised that error is actually from inside client-native here: https://github.com/haproxytech/client-native/blob/master/runtime/runtime_client.go#L229 but nevertheless it shouldn't be returning the error because i am on haproxy 2.7. Should I open an issue there instead?

The text was updated successfully, but these errors were encountered:

oliwer · 2023-02-20T14:35:00Z

Hi @fatchan and thanks for the report.

I could not reproduce this behavior on my machine. Could you please provide:

the "haproxy" section of your dataplaneapi.hcl
the systemd unit file that you use for haproxy

For me it works when using reload_strategy: systemd and not providing any custom commands.

fatchan · 2023-02-21T10:40:12Z

Thanks Olivier,

as requested:

haproxy section of dataplaneapi.hcl

haproxy {
  config_file = "/etc/haproxy/haproxy.cfg"
  haproxy_bin = "/usr/sbin/haproxy"
  reload {
    reload_delay    = 5
    reload_cmd      = "service haproxy reload"
    restart_cmd     = "service haproxy restart"
    reload_strategy = "custom"
  }
}

systemd unit file

[Unit]
Description=HAProxy Load Balancer
Documentation=man:haproxy(1)
Documentation=file:/usr/share/doc/haproxy/configuration.txt.gz
After=network-online.target rsyslog.service
Wants=network-online.target

[Service]
EnvironmentFile=-/etc/default/haproxy
EnvironmentFile=-/etc/sysconfig/haproxy
BindReadOnlyPaths=/dev/log:/var/lib/haproxy/dev/log
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid" "EXTRAOPTS="
ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS
ExecReload=/usr/sbin/haproxy -Ws -f $CONFIG -c -q $EXTRAOPTS
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
SuccessExitStatus=143
Type=notify

[Install]
WantedBy=multi-user.target

Update: I just build dataplaneapi myself with a forked client-native with the version check removed. Works fine for now.

mjuraga · 2023-03-17T12:03:33Z

Hi @fatchan there seems to be an error when fetching the version from the HAProxy's stats socket. Can you paste the output of show info command in the stats socket when this is happening (after a reload)?

fatchan · 2023-03-19T03:52:28Z

Thanks @mjuraga

Here is the output of show info in the stats socket while the behaviour is being exhibited:

`show info` output

Name: HAProxy
Version: 2.7.3-1~bpo11+1
Release_date: 2023/02/15
Nbthread: 2
Nbproc: 1
Process_num: 1
Pid: 141229
Uptime: 0d 0h00m38s
Uptime_sec: 38
Memmax_MB: 0
PoolAlloc_MB: 0
PoolUsed_MB: 0
PoolFailed: 0
Ulimit-n: 200032
Maxsock: 200032
Maxconn: 100000
Hard_maxconn: 100000
CurrConns: 0
CumConns: 78
CumReq: 71
MaxSslConns: 0
CurrSslConns: 0
CumSslConns: 21
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 0
ConnRateLimit: 0
MaxConnRate: 2
SessRate: 0
SessRateLimit: 0
MaxSessRate: 2
SslRate: 0
SslRateLimit: 0
MaxSslRate: 1
SslFrontendKeyRate: 0
SslFrontendMaxKeyRate: 1
SslFrontendSessionReuse_pct: 0
SslBackendKeyRate: 0
SslBackendMaxKeyRate: 0
SslCacheLookups: 0
SslCacheMisses: 0
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
Tasks: 42
Run_queue: 0
Idle_pct: 99
node: fe-sg-1.bfcdn.host
Stopping: 0
Jobs: 7
Unstoppable Jobs: 1
Listeners: 6
ActivePeers: 0
ConnectedPeers: 0
DroppedLogs: 0
BusyPolling: 0
FailedResolutions: 0
TotalBytesOut: 174706
TotalSplicdedBytesOut: 0
BytesOutRate: 32
DebugCommandsIssued: 0
CumRecvLogs: 0
Build info: 2.7.3-1~bpo11+1
Memmax_bytes: 0
PoolAlloc_bytes: 593944
PoolUsed_bytes: 593944
Start_time_sec: 1679197528
Tainted: 0x40

In this case, it was after a reload from adding an ssl certificate through dataplaneapi. Later, when sending a DELETE to /v2/services/haproxy/runtime/servers/example?backend=example, I get a response like:

{
  "code": 500,
  "message": "this operation requires HAProxy 2.6 or later"
}

And again, just in case you need:

haproxy -vv

HAProxy version 2.7.3-1~bpo11+1 2023/02/15 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2024.
Known bugs: http://www.haproxy.org/bugs/bugs-2.7.3.html
Running on: Linux 5.10.0-12-amd64 #1 SMP Debian 5.10.103-1 (2022-03-07) x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_SYSTEMD=1 USE_PROMEX=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY +LUA -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC +RT +SHM_OPEN +SLZ -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=2).
Built with OpenSSL version : OpenSSL 1.1.1n  15 Mar 2022
Running on OpenSSL version : OpenSSL 1.1.1k  25 Mar 2021
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Support for malloc_trim() is enabled.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.36 2020-12-04
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 10.2.1 20210110

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace

dataplaneapi -v

HAProxy Data Plane API v2.7.3 763edd3

Build from: https://github.com/haproxytech/dataplaneapi
Build date: 2023-03-10T13:00:43Z

deskjet · 2024-02-07T20:45:15Z

Hello! It appears Docker images are affected by this issue. Specifically, I am using haproxytech/haproxy-alpine-quic:2.9. Using reload_strategy: systemd is not an option as systemd is not available inside the container.

I have Datplane configured by adding stats socket /var/run/haproxy-runtime.sock mode 666 level admin to the haproxy.cfg global section and appending

userlist haproxy-dataplaneapi
user admin insecure-password mypassword

program api
  command /usr/bin/dataplaneapi --host 0.0.0.0 --port 5555 --haproxy-bin /usr/sbin/haproxy --config-file /etc/haproxy/haproxy.cfg --reload-cmd "kill -SIGUSR2 1" --restart-cmd "kill -SIGUSR2 1" --reload-delay 5 --userlist haproxy-dataplaneapi --master-runtime=/var/run/haproxy-runtime.sock
  no option start-on-reload

To reproduce the issue, create a new backend called "test" using Dateplane API by POSTing to /v2/services/haproxy/configuration/backends, which works fine. Then, send a POST /v2/services/haproxy/runtime/servers?backend=test, which triggers the issue.

Just in case it's relevant:

haproxy -vv

/ # haproxy -vv
HAProxy version 2.9.4-4e071ad 2024/01/31 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.9.4.html
Running on: Linux 6.5.9-201.fsync.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Oct 31 20:31:40 UTC 2023 x86_64
Build options :
  TARGET  = linux-musl
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_PTHREAD_EMULATION=1 USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 -BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX +PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=16).
Built with OpenSSL version : OpenSSL 3.1.4+quic 24 Oct 2023
Running on OpenSSL version : OpenSSL 3.1.4+quic 24 Oct 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with Lua version : Lua 5.4.6
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.42 2022-12-11
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 12.2.1 20220924

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace

dataplaneapi -v

HAProxy Data Plane API v2.9.0 91da11d4

Build from: https://github.com/haproxytech/dataplaneapi.git
Build date: 2024-01-31T16:05:38Z

matthisholleville · 2024-02-26T21:57:57Z

Hello, do you have any updates on whether a potential fix is underway? We are completely stuck on transitioning our HAProxy to containers.

If needed, I am interested in contributing.

mjuraga added the bug Something isn't working label Mar 17, 2023

matthisholleville mentioned this issue Feb 26, 2024

Error while using runtime: option is not avaiable #324

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reloading haproxy causes endpoints to incorrectly respond with a version requirement error. #289

Reloading haproxy causes endpoints to incorrectly respond with a version requirement error. #289

fatchan commented Feb 19, 2023 •

edited

oliwer commented Feb 20, 2023

fatchan commented Feb 21, 2023 •

edited

mjuraga commented Mar 17, 2023

fatchan commented Mar 19, 2023

deskjet commented Feb 7, 2024 •

edited

matthisholleville commented Feb 26, 2024

Reloading haproxy causes endpoints to incorrectly respond with a version requirement error. #289

Reloading haproxy causes endpoints to incorrectly respond with a version requirement error. #289

Comments

fatchan commented Feb 19, 2023 • edited

oliwer commented Feb 20, 2023

fatchan commented Feb 21, 2023 • edited

mjuraga commented Mar 17, 2023

fatchan commented Mar 19, 2023

deskjet commented Feb 7, 2024 • edited

matthisholleville commented Feb 26, 2024

fatchan commented Feb 19, 2023 •

edited

fatchan commented Feb 21, 2023 •

edited

deskjet commented Feb 7, 2024 •

edited