Packets block happens on writing to wireguard device by udp protocol exceeds 30 packets/sec

[bigboyq]

Package version

1.0.20211208-1

Firmware version

v2.0.9-hotfix.2

Device

EdgeRouter X (SFP) - e50

Issue description

My network connection looks like
'''
Server[192.168.100.18]------Wireguard-------ERX[192.168.100.19/192.168.0.1]-------LAN-------Mac[192.168.0.110]
'''
192.168.100.18/192.168.100.19 is wireguard devices
192.168.0.1/192.168.0.110 is Lan devices
if I run on Server

iperf3 -s

run on Mac

iperf3 -u -c 192.168.100.18 -l 99

I got log on Server

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.0.110, port 50429
[ 5] local 192.168.100.18 port 5201 connected to 192.168.0.110 port 58909
[ ID] Interval      Transfer   Bitrate     Jitter  Lost/Total Datagrams
[ 5]  0.00-1.00  sec 2.90 KBytes 23.7 Kbits/sec 15644245684.792 ms 0/30 (0%)  
[ 5]  1.00-2.00  sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
[ 5]  2.00-3.00  sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
[ 5]  3.00-4.00  sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
[ 5]  4.00-5.00  sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
[ 5]  5.00-6.00  sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
[ 5]  6.00-7.00  sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
[ 5]  7.00-8.00  sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
[ 5]  8.00-9.00  sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
[ 5]  9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 15644245684.792 ms 0/0 (0%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval      Transfer   Bitrate     Jitter  Lost/Total Datagrams
[ 5]  0.00-10.01 sec 2.90 KBytes 2.37 Kbits/sec 15644245684.792 ms 0/30 (0%) receiver

Obviously, while udp packets writing to wireguard device exceeds 30p/s, the device blocked writing.
I think something might happens on packet queue located inside wireguard driver.

Other tests been done

1. If sending from Server to Mac by udp over wireguard , No blocking
2. If NAT enable/disable on ERX, No difference
3. If iperf3 using TCP, No blocking

Configuration and log output

No response

[bigboyq]

Similar discuss
https://community.ui.com/questions/Paket-loss-with-hwnat-enabled-to-Wireguard-device/e0112b17-ff0b-4348-893c-73b413873209

[bigboyq]

More tests has been done

1. If run iperf3 -c directly on Er-x, no packet loss
2. If tune the -l from 99 to 64, Server got 0/103, then 0/0
3. if tune the -l from 99 to 32, Server got 0/183, then 0/0
   If change server from ping 9ms to 45ms, test results keep the same

[yukinobu]

Hi.

I'm also using EdgeRouter X and faced a similar issue. From a client computer which connected to EdgeRouter's WireGuard server, I could browse web sites such as YouTube, but the video playback is very slow.

When I inspect the packet flow with Wireshark, it seems that YouTube first sends me the video data over UDP. But EdgeRouter dropped these UDP packets. After I waited a while it seemed to fall back to TCP and and I was able to play the video.

Note: the link above points to the hwnat issue. When I disabled my EdgeRouter's hwnat, the issue no longer occurred. However, the EdgeRouter's performance will drop significantly...

[jbeez]

@yukinobu Was it UDP over port 443(aka QUIC)? You might be able to just block QUIC outright so it never uses that and see if it improves the experience?

[yukinobu]

@jbeez Yes, it seemed QUIC. Now, I blocking all egress UDP packets and it improves my experience definitely!

Below is a snippet of my NAT rules. For packets from WireGuard tunnels, only TCP and ICMP are handled by NAT.

rule 5030 {
    description "masquerade from WireGuard to WAN (exclude udp to avoid hwnat bug)"
    log disable
    outbound-interface eth0
    protocol tcp
    source {
        address 10.xxx.yyy.0/24
    }
    type masquerade
}
rule 5031 {
    description "masquerade from WireGuard to WAN (exclude udp to avoid hwnat bug)"
    log disable
    outbound-interface eth0
    protocol icmp
    source {
        address 10.xxx.yyy.0/24
    }
    type masquerade
}