APL/SAND no audio after running setup-audio script using "sof"

[RodBarnes]

Describe the bug
No audio after reboot. Sound still shows only "Dummy Output". setup-audio script completed without reporting any errors. Ran it a second time and rebooted but no change. This is running under Linuxmint 21 install over Mr ChromeBox UEFI. For context: When I was previously working with Breath/Ubuntu, I had audio after running that audio setup script. (I think you know that project was terminated.)

Boardname
SAND

Logs
Attempted to run debugging.sh script but it fails with: lspci not found. Please install pciutils.
Attempted sudo apt install pciutils and it reports pciutils is already the newest version (1:3.7.0-6)
Ran lspci and it identifies a "Multimedia audio controller" (output attached).
Ran hwinfo and it identifies a "Multimedia audio controller" (output attached).
lspci.txt
hwinfo.txt

[WeirdTreeThing]

Can you send dmesg, please?

[RodBarnes]

output of dmesg attached.
dmesg.txt

Additional info from the "System Information" area:

Audio:
  Device-1: Intel Celeron N3350/Pentium N4200/Atom E3900 Series Audio Cluster
    driver: sof-audio-pci-intel-apl bus-ID: 00:0e.0 chip-ID: 8086:5a98
  Sound Server-1: ALSA v: k6.6.0-chrultrabook running: yes
  Sound Server-2: PulseAudio v: 15.99.1 running: yes
  Sound Server-3: PipeWire v: 0.3.48 running: yes

[RodBarnes]

Any other suggestions of things I can try? Or things I can check and provide to you for analysis? This is a clean system -- I can easily rebuild it as needed -- so wondering if installing Ubuntu and trying that would be worth the effort. As mentioned earlier, I had audio 95% working under Breath/Ubuntu. That was using PulseAudio; the PipeWire efforts (at that time) weren't working well enough. Or maybe try AVS just to see if it works? I think if SOF works, that would be sufficient for me. I don't care about headphone audio, just speaker audio, microphone. Not sure if audio over HDMI plays into this or not.

[WeirdTreeThing]

This looks like a kernel regression based on dmesg. AVS will likely work just fine, just be careful if you override the speaker protection.

[RodBarnes]

Could you clarify what actually happens with the speakers under AVS with the max98357a? I'd like to better understand what options I have to manage the risk if I chose to enable the speakers. Like, do I ensure I never go above 50% volume or something like that? Or is the risk more invisible to the user so that it is not really controllable?

Quoting you from this thread :
"There is something wrong with the AVS (and older SKL) driver that makes the left channel heat up faster than the right channel, causing damage."

• By "heat up", I understand you to mean that the volume level on the left channel increases faster than the right channel and can then blow your left speaker? Or are you actually referring to physical heat that damages the module?

"software is used to limit the maximum audio output."

• By "limit" I understand it means "disables" since the prompt during setup of AVS states the speakers are "disabled".

Lastly, is there a way to "revert to stock" if I choose to no longer use your setup-audio? Is it just a matter of reverting to the stock kernel?

[WeirdTreeThing]

By "heat up", I understand you to mean that the volume level on the left channel increases faster than the right channel and can then blow your left speaker? Or are you actually referring to physical heat that damages the module?

Physical heat which causes speakers to melt.

By "limit" I understand it means "disables" since the prompt during setup of AVS states the speakers are "disabled".

ChromeOS audio configuration limits the maximum playback db level.

Lastly, is there a way to "revert to stock" if I choose to no longer use your setup-audio? Is it just a matter of reverting to the stock kernel?

What exactly do you mean by stock?

[RodBarnes]

What exactly do you mean by stock?

Sorry, I was unclear. I was referring to reverting to an earlier kernel; i.e., the 5.15 kernel which LinuxMint 21.1 installs by default. But, again, my question is really, "If I choose to no longer use your setup-audio, what steps do I take to "undo" what it does?

[WeirdTreeThing]

The kernel doesn't really matter. Idk why you'd want to remove audio but you can just delete the modprobe config in /etc/modprobe.d/, either sof or avs. That will make the kernel use the default driver, which is the non-functional SKL driver. UCM doesn't really matter when the driver does nothing, same with firmware and topology.

[RodBarnes]

The kernel doesn't really matter. Idk why you'd want to remove audio but you can just delete the modprobe config in /etc/modprobe.d/, either sof or avs. That will make the kernel use the default driver, which is the non-functional SKL driver. UCM doesn't really matter when the driver does nothing, same with firmware and topology.

Thanks for clarifying. I assumed that it used an updated kernel because uname -r returns 6.6.0-chrultrabook and you wrote in this thread that "the minimum supported kernel version is now 6.4". I just began going through the setup-audio code but I am not a Python programmer. I became very familiar with the setup-audio script used under Breath so I do know some of what is happening.

My goal is to have: 1) speaker audio, 2) audio over BT headphones, 3) built-in microphone, and 4) audio over HDMI -- in that order. But neither of the SOF or AVS options seem to provide a driver that does this. AVS did provide headphone audio on my system but I don't care about that. SOF is supposed to provide speaker audio but, so far, that hasn't worked on my system. BT audio works under both (maybe not affected by this driver?). Neither provide HDMI audio.

If you're willing to keep working on this issue, I'm willing to continue. Yet, given that SOF isn't doing what you indicate it does on other APL systems, I wonder if I've got a one-off problem. My reference to "undoing" the changes made by setup-audio is based on my experience and thinking I might need to try going back to what was being done under Breath since that was working on my system at one point.

[WeirdTreeThing]

Thanks for clarifying. I assumed that it used an updated kernel because uname -r returns 6.6.0-chrultrabook and you wrote in this #60 (comment) that "the minimum supported kernel version is now 6.4". I just began going through the setup-audio code but I am not a Python programmer. I became very familiar with the setup-audio script used under Breath so I do know some of what is happening.

Just note that while this script was originally a rewrite of the breath script, it has now evolved into something completely different.

My goal is to have: 1) speaker audio, 2) audio over BT headphones, 3) built-in microphone, and 4) audio over HDMI -- in that order. But neither of the SOF or AVS options seem to provide a driver that does this. AVS did provide headphone audio on my system but I don't care about that. SOF is supposed to provide speaker audio but, so far, that hasn't worked on my system. BT audio works under both (maybe not affected by this driver?). Neither provide HDMI audio.

That is doable when the SOF issues get resolved. Bluetooth is completely unrelated to the internal audio DSP which is why it works just fine, same goes for usb audio. Not sure why HDMI audio didn't work with the AVS driver, since that should work just fine.

If you're willing to keep working on this issue, I'm willing to continue. Yet, given that SOF isn't doing what you indicate it does on other APL systems, I wonder if I've got a one-off problem. My reference to "undoing" the changes made by setup-audio is based on my experience and thinking I might need to try going back to what was being done under Breath since that was working on my system at one point.

APL SOF may just be broken on that kernel version, ideally we can get someone else to test it. Breath used the SKL driver on APL machines which requires old chromeOS kernels with various patches applied on top, which is less than ideal.

[RodBarnes]

That is doable when the SOF issues get resolved.

APL SOF may just be broken on that kernel version, ideally we can get someone else to test it.

SOF was working fully for APL devices previously? Given that, if you have other things for me to try, let me know; I have time to put into this.

I assume the SOF driver is maintained somewhere? Is there somewhere I can monitor for updates? Or is that just going to be here? Also, I had the impression that AVS was the "new thing" and going to replace SOF -- maybe that's incorrect?

[WeirdTreeThing]

SOF was working fully for APL devices previously? Given that, if you have other things for me to try, let me know; I have time to put into this.

It has always been working with the exception of headphones, which are broken for some reason.

I assume the SOF driver is maintained somewhere? Is there somewhere I can monitor for updates? Or is that just going to be here? Also, I had the impression that AVS was the "new thing" and going to replace SOF -- maybe that's incorrect?

SOF development happens on the lkml and the sof github org (https://github.com/thesofproject). AVS is replacing the deprecated SKL driver, but sof is separate, it's weird. The old SKL driver covers Skylake (obviously), Kabylake, and Apollolake. Starting with Geminilake, google started using SOF on ChromeOS, which is signed with the community key so that anyone can modify and recompile the audio firmware. APL was caught right in the middle of the transition so it has support for both SKL (now replaced by AVS) and SOF, but SOF was never officially supported which is probably why headphones don't work. AVS is essentially a rewritten updated version of the SKL driver, it even uses the same proprietary firmware signed with intel's private key. SKL, KBL, and APL will all only boot audio firmware signed by intel.

[RodBarnes]

I confirmed (by rebuilding the laptop) that this doesn't work under Ubuntu 22.04 either. Not unexpected but wanted to eliminate that possibility.

I noticed that for LM and Ubuntu, setup-audio attempts to get an updated libasound2 module from the Pop!_OS distro. Yet it doesn't appear that has happened after running the program for 'sof':

$ ldconfig -p | grep libasound2*
	libasound.so.2 (libc6,x86-64) => /lib/x86_64-linux-gnu/libasound.so.2
$ ldconfig -p | grep libasound*
	libasound.so.2 (libc6,x86-64) => /lib/x86_64-linux-gnu/libasound.so.2

But I see a libasound2 in the standard libraries:

$ sudo apt list libasound*
Listing... Done
libasound2-data/now 1.2.8-1pop1~1669155503~22.04~a04c9a1 all [installed,local]
libasound2-dev/jammy 1.2.6.1-1ubuntu1 amd64
libasound2-dev/jammy 1.2.6.1-1ubuntu1 i386
libasound2-doc/jammy,jammy 1.2.6.1-1ubuntu1 all
libasound2-plugin-bluez/jammy 3.0.0-2 amd64
libasound2-plugin-equal/jammy 0.6-8 amd64
libasound2-plugin-smixer/jammy 1.2.6.1-1ubuntu1 amd64
libasound2-plugin-smixer/jammy 1.2.6.1-1ubuntu1 i386
libasound2-plugins-extra/jammy 1.1.0-1ubuntu2 amd64
libasound2-plugins/jammy,now 1.2.6-1 amd64 [installed]
libasound2-plugins/jammy 1.2.6-1 i386
libasound2/now 1.2.8-1pop1~1669155503~22.04~a04c9a1 amd64 [installed,local]
libasound2/jammy 1.2.6.1-1ubuntu1 i386

Thoughts?

[WeirdTreeThing]

libasound2-data/now 1.2.8-1pop1~1669155503~22.04~a04c9a1 all [installed,local]
libasound2/now 1.2.8-1pop1~1669155503~22.04~a04c9a1 amd64 [installed,local]

Those are the packages from PopOS

[RodBarnes]

Oh, right. Missed seeing the "installed" marker in the apt list. That they didn't appear with ldconfig threw me.

So, what do you recommend as the next step?

Is there merit in submitting an issue to SOF Project?

[WeirdTreeThing]

SOF will likely work on an older kernel, maybe try uninstalling the chrultrabook kernel. apt list --installed | grep linux-image- will get you a list of kernels on your system.

[RodBarnes]

What I did:

• Via grub, I reverted to the original 5.15.0 kernel installed with LM 21.2. I rebooted and still only "Dummy Output" sound device.
• Via Synaptics, I removed the 6.6.0-chrultrabook kernel -- yeah, I realize this shouldn't be necessary, but trying anything -- and rebooted. No change.
• Via update manager, I installed the 6.2.0 kernel, rebooted, made it active via grub. No change.

Are there other steps I need to take besides switching the kernel?

[WeirdTreeThing]

Send dmesg from 5.15.

[RodBarnes]

Send dmesg from 5.15.

dmesg.txt

It looks like its trying to load firmware/topology for SKL not APL?

[WeirdTreeThing]

Very likely that apl sof is not enabled on that kernel.

[RodBarnes]

This may be helpful. I rebuilt the system and then did the following:

1. Clean install of LM 21.2
2. dmesg > dmesg_5.15.0-76-generic_clean.txt
3. Install git, download setup-audio and then run with "sof".
4. Reboot.
5. dmesg > dmesg_6.6.0-chrultrabook.txt
6. Reboot and select 5.15. kernel.
7. dmesg > dmesg_5.15.0-76-generic_after.txt
8. Install 6.2 kernel via Synaptics.
9. Reboot and select 6.2 kernel.
10. dmesg > dmesg_6.2.0-37-generic.txt

I agree that it doesn't look like this will work with the 5.15 kernel. Yet I'm not seeing any errors or other indications as to why I see no output device under the 6.2 and later kernel.

[WeirdTreeThing]

On 6.2:

[   10.221840] bxt_da7219_max98357a bxt_da7219_mx98357a: ASoC: sink widget DMic overwritten
[   10.221859] bxt_da7219_max98357a bxt_da7219_mx98357a: ASoC: Failed to add route iDisp3 Tx(*) -> hifi3
[   10.221868] bxt_da7219_max98357a bxt_da7219_mx98357a: ASoC: Failed to add route iDisp3_out -> iDisp3 Tx(*)
[   10.221874] bxt_da7219_max98357a bxt_da7219_mx98357a: ASoC: Failed to add route iDisp2 Tx(*) -> hifi2
[   10.221879] bxt_da7219_max98357a bxt_da7219_mx98357a: ASoC: Failed to add route iDisp2_out -> iDisp2 Tx(*)
[   10.221884] bxt_da7219_max98357a bxt_da7219_mx98357a: ASoC: Failed to add route iDisp1 Tx(*) -> hifi1
[   10.221889] bxt_da7219_max98357a bxt_da7219_mx98357a: ASoC: Failed to add route iDisp1_out -> iDisp1 Tx(*)
[   10.221899] bxt_da7219_max98357a bxt_da7219_mx98357a: snd_soc_bind_card: snd_soc_dapm_add_routes failed: -19

On 6.6:

[    9.165511] bxt_da7219_max98357a bxt_da7219_mx98357a: ASoC: sink widget DMic overwritten
[    9.165526] bxt_da7219_max98357a bxt_da7219_mx98357a: ASoC: Failed to add route iDisp3 Tx(*) -> hifi3
[    9.165535] bxt_da7219_max98357a bxt_da7219_mx98357a: ASoC: Failed to add route iDisp3_out -> iDisp3 Tx(*)
[    9.165540] bxt_da7219_max98357a bxt_da7219_mx98357a: ASoC: Failed to add route iDisp2 Tx(*) -> hifi2
[    9.165545] bxt_da7219_max98357a bxt_da7219_mx98357a: ASoC: Failed to add route iDisp2_out -> iDisp2 Tx(*)
[    9.165550] bxt_da7219_max98357a bxt_da7219_mx98357a: ASoC: Failed to add route iDisp1 Tx(*) -> hifi1
[    9.165555] bxt_da7219_max98357a bxt_da7219_mx98357a: ASoC: Failed to add route iDisp1_out -> iDisp1 Tx(*)
[    9.165563] bxt_da7219_max98357a bxt_da7219_mx98357a: snd_soc_bind_card: snd_soc_dapm_add_routes failed: -19

These are the exact same errors, which leads me to believe the topology is the problem. I need to do some testing on my APL, I'll get back to you when I find out what's happening.

[WeirdTreeThing]

One thing I thought of is the sof binaries in LM might be too old. Using a newer distro may work better, or you could manually update sof fw/tplg.

[RodBarnes]

I did try this with the latest Ubuntu 22.04 and had the same results; i.e., only a dummy output appears.

Were you able to replicate this issue on your APL? Or, were you successful using a different distro on your device?

Do you have instructions on updating the sof fw/tplg? Or, can you point me to something on the SOF Project site?

[WeirdTreeThing]

Ubuntu 22.04 is the exact same as Linux Mint, you would want to test 23.10. I haven't had the time to test on my APL yet, hopefully soon.

[RodBarnes]

Ubuntu 22.04 is the exact same as Linux Mint, you would want to test 23.10. I haven't had the time to test on my APL yet, hopefully soon.

I can give that a try. However, I had understood that the steps taken in setup-audiodid the necessary to apply the sof fw/tplg to the kernel and that was used after the reboot. Reading through the SOF materials on their site, that's what I gathered.

[WeirdTreeThing]

The script will install the distro package for sof (with the exception of jasperlake devices) so if your distro has old packages then sof firmware/topology will also be old.

[WeirdTreeThing]

Sof audio is working on my APL running Fedora.

[RodBarnes]

Sof audio is working on my APL running Fedora.

Which version? Have you tried it on any Ubuntu variant or downstream distro?

[WeirdTreeThing]

Fedora 39. I have not tried Ubuntu.

[RodBarnes]

I just installed Fedora-Cinnamon 39 and ran your setup-audio. I now have sound through the speakers. I do not have any microphone. Does your microphone work?

[WeirdTreeThing]

As per usual on SOF, the mic is pretty quiet. Using commands to increase the volume to 200% does help.

[RodBarnes]

Thanks, I'll look at increasing the mic volume. I also have a really small USB mic that was working previously under Breath so I'll try that, too.

Given that it's working under Fedora, I'm going to try the Ubuntu 23.10 to see if it works there. I'm much more familiar with the Debian/Ubuntu system model and would prefer not having to learn to do things all over again. I'll report back.

[RodBarnes]

I confirmed that SOF is working under both Fedora-Cinnamon 39 and Ubuntu 23.10 and all audio devices are present except the headphone jack.

Thank you for your work on this and helping me to find a resolution. It seems the current options are:

1. Switch to Fedora 39 and learn new ways of doing things -- like dnf instead of apt.
2. Switch to Ubuntu 23.10 -- and live with its walled garden of snaps (👎) or maybe Ubuntu Cinnamon with snap disabled.
3. Stay on LinuxMint and wait for the release based upon 23.10.

NOTE1: Initially, the microphone was not available as an input device under either OS. But, for both, I went to mictests.com (which is part of my test list for validation) and was able to select each of the listed audio inputs. One of the listed audio selections would be the actual microphone (the others were monitor devices) and after selecting it, the test would succeed and then the microphone would appear as a device. Sort of like some configuration was completed in this selection process to make it available.

NOTE2: Just for completeness, I tried installing the 6.5 mainline kernel (How to Install Linux Kernel 6.5 on Linux Mint 21 or 20) then ran setup-audio but this was not successful; still only Dummy Output appears in the sound app.

[WeirdTreeThing]

Switch to Fedora 39 and learn new ways of doing things -- like dnf instead of apt.

Luckily, the dnf command syntax is similar to apt. So the switch should be relatively easy.

Stay on LinuxMint and wait for the release based upon 23.10.

LM only ships LTS releases of upstream Ubuntu, so the next LM release will be 24.04, which is over 4 months away.

Initially, the microphone was not available as an input device under either OS.

This seems to be a strange issue with GNOME, as I have not been able to reproduce that issue on any other DE.

NOTE2: Just for completeness, I tried installing the 6.5 mainline kernel (How to Install Linux Kernel 6.5 on Linux Mint 21 or 20) then ran setup-audio but this was not successful; still only Dummy Output appears in the sound app.

The kernel isn't the issue, old SOF firmware/topology is the issue. You can manually upgrade SOF by downloading the latest release of sof-bin and following the instructions in the tarball to upgrade. I've never done this before so I don't exactly know how manually upgrading it works.

[RodBarnes]

Back again :-) I'm trying to get this to work on LinuxMint Edge (which uses the 6.2 kernel). I tried just using the instructions from the SOF site and running the install.sh script then rebooted, but that didn't fix the audio.

One thing I noticed between the install on fedora (after running your setup-audio script) and the LM default install is that lspci shows fedora Kernel driver in use: sof-audio-pci-intel-aplwhile LM shows Kernel driver in use: snd_soc_skl. So, there must be something else I need to do to cause it to use the correct driver? But, so far, I haven't found anything that helps me know what to do to switch the driver. Looking at the old script I have from Breath, it appears that a rebuild of the kernel is required to tell it to use the other driver as it is setting some kernel flags regarding DSP before it then rebuilds the kernel.

I'll walk through your setup-audio and try to determine the steps it is taking.

[RodBarnes]

UPDATE: I installed Pop!_OS to test this and the sound doesn't work there after cloning and running the setup-audio. (It seemed reasonable to test it given that the topologies for ubuntu and LM are pulled from that distro's repositories.) Given that it doesn't work on Pop!_OS it makes sense that it doesn't work on LM, either. sudo dmesg shows the same errors as shown previously.

$ uname -r
6.6.6-76060606-generic
$ python3 --version
Python 3.10.12
$ git --version
git version 2.34.1

While I was able to get sound on Fedora 39 Cinnamon, I don''t prefer the constant updates of being on the bleeding edge.

[RodBarnes]

I installed Ubuntu 23.10 and, as before, after running setup-audio, the sound is working. I'm going to try grabbing the libraries from the 23.10 repositories and see if that gets things working on LM.

[WeirdTreeThing]

One thing I noticed between the install on fedora (after running your setup-audio script) and the LM default install is that lspci shows fedora Kernel driver in use: sof-audio-pci-intel-aplwhile LM shows Kernel driver in use: snd_soc_skl

That would suggest that the SOF driver for APL isn't enabled for whatever reason.

I installed Ubuntu 23.10 and, as before, after running setup-audio, the sound is working. I'm going to try grabbing the libraries from the 23.10 repositories and see if that gets things working on LM.

So 23.10 ships with a new enough kernel and new enough sof.