Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

国产龙蜥操作系统运行DPVS时出现段错误 #896

Open
bjwxy opened this issue Jun 22, 2023 · 7 comments
Open

国产龙蜥操作系统运行DPVS时出现段错误 #896

bjwxy opened this issue Jun 22, 2023 · 7 comments
Assignees
@ywc689

Comments

@bjwxy
Copy link

bjwxy commented Jun 22, 2023

2块万兆做bond4，2个numa节点，启动DPVS时出现段错误。段错误的位置出现在neigh_add_table函数中的链表初始化INIT_LIST_HEAD. coredump信息如下，请问下大家有没有什么思路，谢谢

image
@ywc689
Copy link
Collaborator

ywc689 commented Jul 10, 2023

用的是哪个版本？能否稳定复现?可以打开 src/config.mk 中的 CONFIG_DPVS_NEIGH_DEBUG 调试配置选项，看看能否得到更有用的信息。

@zhangjs
Copy link

zhangjs commented Nov 7, 2023

用的是哪个版本？能否稳定复现?可以打开 src/config.mk 中的 CONFIG_DPVS_NEIGH_DEBUG 调试配置选项，看看能否得到更有用的信息。

在centos7，intel cpu下遇到同样的问题，版本1.9.4， dpdk版本dpdk-stable-20.11.1，更新neigh_add_table时出错，在INIT_LIST_HEAD(&new_neighbour->queue_list);后读取new_neighbour->flag出错，中间加一个调试打印就不会出错，不知道是不是跟编译环境有关

@zhangjs
Copy link

zhangjs commented Nov 7, 2023

改成-g -O0后也是在INIT_LIST_HEAD(&new_neighbour->queue_list)中SIGSEGV，是内存对齐的问题吗

@zhangjs
Copy link

zhangjs commented Nov 7, 2023

看着是O3的问题，改成O2，或者Makefile开debug就没问题

@yfming
Copy link
Contributor

yfming commented Nov 7, 2023 •
edited

内核 5.14 gcc-11.3.1
我们也遇到这个问题了，因为neighbour_entry里面的list不能保证16字节对齐，但是用了__rte_cache_aligned，编译器会以为是16字节对齐的，编译后list的那个初始化操作用了movaps指令，这个指令操作的地址必须要16字节对齐，否则就崩溃了。
可以去掉neighbour_entry结构体的__rte_cache_aligned。编译器就不会以为结构体里面的list是16字节对齐，就不会编译的时候优化这个list的操作了。

@bjwxy
Copy link
Author

bjwxy commented Nov 21, 2023

内核 5.14 gcc-11.3.1
我们也遇到这个问题了，因为neighbour_entry里面的list不能保证16字节对齐，但是用了__rte_cache_aligned，编译器会以为是16字节对齐的，编译后list的那个初始化操作用了movaps指令，这个指令操作的地址必须要16字节对齐，否则就崩溃了。
可以去掉neighbour_entry结构体的__rte_cache_aligned。编译器就不会以为结构体里面的list是16字节对齐，就不会编译的时候优化这个list的操作了。

这个问题隐藏的很深啊。。。不是大牛很难发现。膜拜一下。

另外想请教下，这个段错误为什么不是必现的，如果neighbour_entry不是16字节对齐会导致段错误，那么难道这个结构体有的时候是对齐的，有的时候是不能对齐的吗？是什么决定它能不能对齐的呢？

@ywc689 ywc689 self-assigned this Nov 24, 2023
@ywc689 ywc689 mentioned this issue Dec 19, 2023
Open
@benmutlu
Copy link

I have same problem which is #929
I enabled CONFIG_DPVS_NEIGH_DEBUG for more details
while dpvs working, I binded IP addresses to kni interfaces, and whenever I start keepalived, I got this error.
( IP addresses are example )
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.85 tha 00:00:00:00:00:00 tip 1.1.1.85
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.86 tha 00:00:00:00:00:00 tip 1.1.1.86
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.89 tha 00:00:00:00:00:00 tip 1.1.1.89
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.91 tha 00:00:00:00:00:00 tip 1.1.1.91
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.70 tha 00:00:00:00:00:00 tip 1.1.1.70
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.71 tha 00:00:00:00:00:00 tip 1.1.1.71
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.72 tha 00:00:00:00:00:00 tip 1.1.1.72
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.73 tha 00:00:00:00:00:00 tip 1.1.1.73
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.74 tha 00:00:00:00:00:00 tip 1.1.1.74
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.75 tha 00:00:00:00:00:00 tip 1.1.1.75
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.76 tha 00:00:00:00:00:00 tip 1.1.1.76
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.77 tha 00:00:00:00:00:00 tip 1.1.1.77
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.78 tha 00:00:00:00:00:00 tip 1.1.1.78
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.79 tha 00:00:00:00:00:00 tip 1.1.1.79
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.81 tha 00:00:00:00:00:00 tip 1.1.1.81
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.83 tha 00:00:00:00:00:00 tip 1.1.1.83
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.58 tha 00:00:00:00:00:00 tip 1.1.1.58
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.59 tha 00:00:00:00:00:00 tip 1.1.1.59
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.54 tha 00:00:00:00:00:00 tip 1.1.1.54
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.55 tha 00:00:00:00:00:00 tip 1.1.1.55
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.85 tha 00:00:00:00:00:00 tip 1.1.1.85
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.86 tha 00:00:00:00:00:00 tip 1.1.1.86
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.89 tha 00:00:00:00:00:00 tip 1.1.1.89
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.91 tha 00:00:00:00:00:00 tip 1.1.1.91
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.70 tha 00:00:00:00:00:00 tip 1.1.1.70
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.71 tha 00:00:00:00:00:00 tip 1.1.1.71
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.72 tha 00:00:00:00:00:00 tip 1.1.1.72
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.73 tha 00:00:00:00:00:00 tip 1.1.1.73
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.74 tha 00:00:00:00:00:00 tip 1.1.1.74
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.75 tha 00:00:00:00:00:00 tip 1.1.1.75
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.76 tha 00:00:00:00:00:00 tip 1.1.1.76
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.77 tha 00:00:00:00:00:00 tip 1.1.1.77
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.78 tha 00:00:00:00:00:00 tip 1.1.1.78
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.79 tha 00:00:00:00:00:00 tip 1.1.1.79
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.81 tha 00:00:00:00:00:00 tip 1.1.1.81
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.83 tha 00:00:00:00:00:00 tip 1.1.1.83
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.58 tha 00:00:00:00:00:00 tip 1.1.1.58
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.59 tha 00:00:00:00:00:00 tip 1.1.1.59
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.54 tha 00:00:00:00:00:00 tip 1.1.1.54
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.55 tha 00:00:00:00:00:00 tip 1.1.1.55
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.85 tha 00:00:00:00:00:00 tip 1.1.1.85
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.86 tha 00:00:00:00:00:00 tip 1.1.1.86
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.89 tha 00:00:00:00:00:00 tip 1.1.1.89
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.91 tha 00:00:00:00:00:00 tip 1.1.1.91
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.70 tha 00:00:00:00:00:00 tip 1.1.1.70
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.71 tha 00:00:00:00:00:00 tip 1.1.1.71
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.72 tha 00:00:00:00:00:00 tip 1.1.1.72
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.73 tha 00:00:00:00:00:00 tip 1.1.1.73
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.74 tha 00:00:00:00:00:00 tip 1.1.1.74
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.75 tha 00:00:00:00:00:00 tip 1.1.1.75
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.76 tha 00:00:00:00:00:00 tip 1.1.1.76
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.77 tha 00:00:00:00:00:00 tip 1.1.1.77
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.78 tha 00:00:00:00:00:00 tip 1.1.1.78
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.79 tha 00:00:00:00:00:00 tip 1.1.1.79
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.81 tha 00:00:00:00:00:00 tip 1.1.1.81
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.83 tha 00:00:00:00:00:00 tip 1.1.1.83
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.58 tha 00:00:00:00:00:00 tip 1.1.1.58
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.59 tha 00:00:00:00:00:00 tip 1.1.1.59
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.54 tha 00:00:00:00:00:00 tip 1.1.1.54
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.55 tha 00:00:00:00:00:00 tip 1.1.1.55
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.85 tha 00:00:00:00:00:00 tip 1.1.1.85
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.86 tha 00:00:00:00:00:00 tip 1.1.1.86
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.89 tha 00:00:00:00:00:00 tip 1.1.1.89
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.91 tha 00:00:00:00:00:00 tip 1.1.1.91
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.70 tha 00:00:00:00:00:00 tip 1.1.1.70
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.71 tha 00:00:00:00:00:00 tip 1.1.1.71
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.72 tha 00:00:00:00:00:00 tip 1.1.1.72
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.73 tha 00:00:00:00:00:00 tip 1.1.1.73
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.74 tha 00:00:00:00:00:00 tip 1.1.1.74
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.75 tha 00:00:00:00:00:00 tip 1.1.1.75
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.76 tha 00:00:00:00:00:00 tip 1.1.1.76
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.77 tha 00:00:00:00:00:00 tip 1.1.1.77
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.78 tha 00:00:00:00:00:00 tip 1.1.1.78
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.79 tha 00:00:00:00:00:00 tip 1.1.1.79
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.81 tha 00:00:00:00:00:00 tip 1.1.1.81
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.83 tha 00:00:00:00:00:00 tip 1.1.1.83
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.58 tha 00:00:00:00:00:00 tip 1.1.1.58
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.59 tha 00:00:00:00:00:00 tip 1.1.1.59
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.54 tha 00:00:00:00:00:00 tip 1.1.1.54
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.55 tha 00:00:00:00:00:00 tip 1.1.1.55
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.85 tha 00:00:00:00:00:00 tip 1.1.1.85
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.86 tha 00:00:00:00:00:00 tip 1.1.1.86
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.89 tha 00:00:00:00:00:00 tip 1.1.1.89
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.91 tha 00:00:00:00:00:00 tip 1.1.1.91
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.70 tha 00:00:00:00:00:00 tip 1.1.1.70
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.71 tha 00:00:00:00:00:00 tip 1.1.1.71
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.72 tha 00:00:00:00:00:00 tip 1.1.1.72
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.73 tha 00:00:00:00:00:00 tip 1.1.1.73
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.74 tha 00:00:00:00:00:00 tip 1.1.1.74
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.75 tha 00:00:00:00:00:00 tip 1.1.1.75
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.76 tha 00:00:00:00:00:00 tip 1.1.1.76
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.77 tha 00:00:00:00:00:00 tip 1.1.1.77
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.78 tha 00:00:00:00:00:00 tip 1.1.1.78
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.79 tha 00:00:00:00:00:00 tip 1.1.1.79
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.81 tha 00:00:00:00:00:00 tip 1.1.1.81
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.83 tha 00:00:00:00:00:00 tip 1.1.1.83
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.58 tha 00:00:00:00:00:00 tip 1.1.1.58
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.59 tha 00:00:00:00:00:00 tip 1.1.1.59
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.54 tha 00:00:00:00:00:00 tip 1.1.1.54
send lcore 0 port0 arp hlen 6 plen 4 op 1 sha 0c:c4:7a:1d:91:38 sip 1.1.1.55 tha 00:00:00:00:00:00 tip 1.1.1.55
Segmentation fault (core dumped)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ywc689 ywc689

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@zhangjs @ywc689 @benmutlu @yfming @bjwxy