Ioniq 6 CAN Bus Disconnected Message

[limatango79]

Describe the bug

I drive the Hyundai Ioniq 6 2023 with HDA II and I've tested Comma 3X with OP 0.9.6, 0.9.7 master-ci, SP release, staging and dev. Regardless of the software, I get the same error.
Every now and then I get the message "CAN bus disconnected". I can't reproduce it either.
I also get the message "Controls Lagging".

I have checked all the cables several times. The device is new and undamaged.

Which car does this affect?

Hyundai Ioniq 6 2023 HDA II

Provide a route where the issue occurs

78e8c2388cf1ce49/00000000--09dfcd3706/0

openpilot version

0.9.6 and 0.9.7 master-ci

Additional info

No response

[sshane]

@briskspirit do you have any time to check this out? Looks like this was hit after boardd stopped sending can messages for 200 ms:

openpilot/selfdrive/boardd/panda.cc

Lines 248 to 253 in d56f188

	if (calculate_checksum(&data[pos], sizeof(can_header) + data_len) != 0) {
	// TODO: also reset CAN comms?
	LOGE("Panda CAN checksum failed");
	size = 0;
	return false;
	}

[adeebshihadeh]

I'll check it out before 0.9.7.

[royjr]

I have the same exact messages daily on a 2024 Kona, may be related to..

• can/pandaStates skips on 2024 Hyundai ports #31563
• Controls Lagging - 2023 Hyundai Santa Cruz #31822
• Hyundai Ioniq 5: Intermittent "Can Bus Disconnected" #30815

[limatango79]

I get the message "CAN Bus Disconnected" about once an hour. The message "Controls Lagging" appears about 3 to 4 times an hour. If desired, I can load more routes in which this occurs.

[thftgr]

It also happens once or twice a week in KIA K8 HEV 2024 HDA II
I drive this vehicle about 20km 6 days a week on average

[adeebshihadeh]

Working on this now. I'll have a branch for you guys to try soon!

[adeebshihadeh]

There's a few things going on here:

• subtle boardd bugs that made edge cases worse boardd: fix SPI return code on some transfer failures #32401, boardd: SPI corruption test #32404
• out of sync slave/master, with slave trying to TX a large buffer but master is clocking out a few bytes at a time trying to get an ACK
• in some cases, boardd + spi0 kernel thread can saturate a core pandad: move to core 3 #32402

[royjr]

Can't specifically remember, but does it make sense that I see these errors less frequently now?

[limatango79]

Unfortunately, that doesn't apply to me.
On a 4-hour drive two days ago, I had “Controls Lagging” about every 15 minutes and “CAN Bus Disconnected” every 25 minutes.
And just yesterday, on a 10-minute drive, I had “Controls Lagging” exactly 4 times.
Somehow you get used to it, but it is annoying. And you become numb to “real” error messages.
I also rarely get the message “Communication Issue between Processes” and “Locationd Error”. But I don't think that has anything to do with the other messages.

[adeebshihadeh]

We're blocking the next release on this, and it's one of the last TODOs. I should have a fully fixed branch for you guys to try in a day or two.

Can't specifically remember, but does it make sense that I see these errors less frequently now?

Yes, I shipped one fix already, but there's still a couple things.

[adeebshihadeh]

@royjr @limatango79 @JJeonbear @MarcoTheDingo @CruiseBrantley @crwayman7

Can you guys try this branch: installer.comma.ai/commaai/pandad-max-mode

[limatango79]

@royjr @limatango79 @JJeonbear @MarcoTheDingo @CruiseBrantley @crwayman7

Can you guys try this branch: installer.comma.ai/commaai/pandad-max-mode

I've just been on a one-hour test drive with this branch. I did not get any error messages such as “CAN bus disconnected” or “controls lagging”.
Here is the corresponding route: 78e8c2388cf1ce49/00000000--e86f98e678

I will take an even longer drive tomorrow, but I think the problem seems to be solved.

Thanks for the hard work.

[JJeonbear]

@royjr @limatango79 @JJeonbear @MarcoTheDingo @CruiseBrantley @crwayman7

Can you guys try this branch: installer.comma.ai/commaai/pandad-max-mode

I will install it today and test drive it.