AirPlay 2 support / research

[lime pewter]

Thread to discuss the investigation around AirPlay 2 support for Music Assistant.

Music Assistant currently only support RAOP over AirPlay. RAOP is what was invented by Apple (maybe even before AirPlay was created on top of that) to do audio streaming. I believe the alternative name for RAOP is "AirTunes v2". AirPlay was invented later on and is focused on video + audio, screen sharing etc. So not only audio.

Most AirPlay 2 device still support RAOP which is why it works in most cases.
But some devices have been cheap in their implementation and didn't implement the classic RAOP streaming but only the AirPlay 2 protocol, at least that is my working theory.

So far we know about Samsung devbices that have the problem that RAOP streaming is not working (audio is just silent), as well as the latest generation Sonos speakers (ERA etc.) with a recent firmware update. Also it is apparently broken in the Unif Poweramp, according to the discussion here: https://github.com/orgs/music-assistant/discussions/4077

For streaming airplay/RAOP, we use a slightly modified version of libraop, hosted here:
https://github.com/music-assistant/libraop

We will either need to adjust libraop to also support AirPlay (2) or maybe create a whole new library dedicated to AirPlay streaming without RAOP. Maybe it is as simple as adding a bit of missing command's/handshakes but it could also be much more complicated where we have to implement the airplay protocol from scratch.

For reference, I have been looking at OwnTone (https://owntone.github.io/owntone-server/), which has both classic RAOP and AirPlay 2 support and also discovered if you disable RAOP in OwnTone (so it is only uses AirPlay) it does work so there is some hope.

Now, I am pretty sure that AirPlay 2 without RAOP only supports lossy codecs but I think that is fine for most people as they wont be able to tell the difference between the lossless RAOP stream and the lossy AP2. I could be mistaken and AP2 still supports ALAC for lossy streaming but its all one big pile of mystery.

@raw crag let's use this thread here to discuss any progress on AirPlay 2

[raw crag]

I am happy to work on the changes necessary to support AirPlay 2. I have a PowerAmp and HASS/MASS setup in my holiday house, which I am remote from until September. I can develop and debug in that environment remotely, but that’s about it.

I only have AirPlay 1 devices in my normal house, so learning about AirPlay 2 protocol is difficult for me.

If anyone has an AirPlay 2 device and has the ability to packet capture the RTSP exchanges from a working AirPlay stream, I would really appreciate if you could supply me a copy.

[lime pewter]

@winter reef FYI

[winter reef]

Awesome

You can check the implementation on Owntone they’re supporting AirPlay 2

Thank you for all the love which is put into MA

[raw crag]

See https://github.com/orgs/music-assistant/discussions/4077 for a very recent learning about RTSP and the PwerAmp. I suspect other devices will be similar.

[lime pewter]

@raw crag you can also use libraop CLI in a standalone manner - just generate a pcm audio file on your disk and use the command line syntax to start libraop with all parameters to connect to the speaker

`maybe that is easier for debugging

[winter reef]

@lime pewter Are We talking about sending or receiving or both?

[raw crag]

I’d love to. But I am running HASSOS, so it’s a bit tricky. If I can build an add-on that gives me an alpine build environment that I can ssh into, then that would improve my productivity enormously

[lime pewter]

Yeah, you can actually - with the advanced SSH terminal addon

[lime pewter]

sending - we are talking about sending audio from MA to AirPlay 2 (only) targets

receiving is no problem

[winter reef]

Ah okay then I would look into the owntone implementation

Because that was the only thing which worked with my cheap Xiaomi speaker

[winter reef]

AirPlay 2 only device

[lime pewter]

Yeah, that is why I pinge dyou becaus eyou have one of those flawed devices. Doe sit also only work if you hard disable RAOP support in OwnTone ?

[winter reef]

@raw crag did you add your device to HomeKit

[lime pewter]

Because that is what I found out - you need to really disable RAOP support because it will always try that first

[raw crag]

@winter reef can you do a tcpdump of the initial exchanges between OwnTone and the speaker please?

[winter reef]

I need to check if I can. Because I actually re-used the speaker case for an open source player

I can re-assemble that maybe

[winter reef]

Could you check that?

[raw crag]

No

[winter reef]

That was my solution:

If the speaker becomes unselected when you start playback, and you in the log see "ANNOUNCE request failed in session startup: 400 Bad Request", then try the Apple Home app > Allow Speakers & TV Access > Anyone On the Same Network (or Everyone).

Directly from the HomeKit app written in the docs from owntone

[winter reef]

@raw crag could you try that?

[raw crag]

OK. Thank you. I will see if I can do that remotely. I assume it should be possible.

I have never used homekit.

[lime pewter]

homekit is not the correct term (as that is a protocol) - the correct term is Apple Home

Sometimes you indeed need to add a AirPlay device to Apple Home for authentication purposes. I never fully grasped why that is

[winter reef]

If this is working we should add it to the docs

But yeah sending to AirPlay 2 devices works good with the owntone but only after changing that setting

[winter reef]

You need to be at home .. Apple Home is not tunneled through VPN. That stuff is really bad

@lime pewter it would be crazy good if we can get AirPlay streaming as endpoint working

[raw crag]

That’s a bummer. Earliest I can do that is mid-August.

[winter reef]

You can try but my experience was not that great.

If you buy me an power amp I can try for you 😛

@raw crag here is the log from owntone with my cheap Xiaomi speaker:

#lounge message

Check also the messages from Marcel we had some discussions about that topic

[lime pewter]

this is not the place for such feature requests. that will come in time, just be patient. This is about debugging airplay (2) protocol to fix the silence issue

[winter reef]

My fault.

[raw crag]

Well, I have some very exciting news. I was able to find a way to get mdns reflection happening between my holiday house in Romania and my house in Australia and I installed a local copy of owntones on a raspberry pi back here in Australia. The fact I could get mdns reflection working, mean't that I could also packet capture exchanges between client devices (like my phone and owntones) and the UniFi PowerAmp.

I first tried streaming to the PowerAmp using apple Music on my iPhone and captured those packets. The music ended up not playing, but the RTSP exchanges that I captured were nothing like RAOP exchanges, so I assume that they were AirPlay 2 exchanges. That is a good start.

I then tried streaming from owntones locally to the PowerAmp and it tried using AirPlay 1 and failed with the same "403 Forbidden" RAOP exchange that I had been experiencing with libraop (including my mods to it).

I then reconfigured OwnTones to disable RAOP for the PowerAmp and you would never guess the outcome!!

It worked. And, even better, I was playing music to an AirPlay 1 airport express in my home in Australia and to the PowerAmp using AirPlay 2 simultaneously ..... I actually can't say how well synchonised they are, but I don't think that matters.

[winter reef]

Did you add it to Apple Home before?

[raw crag]

No

So, we have a pathway forward to get AirPlay 2 support into Music Assistant. OwnTones looks like the go!

[winter reef]

That is really interesting!

[raw crag]

It is very possible that the Apple Music streaming from my phone will work after I do the Apple Home config you suggested, but I can't see the PowerAmp in Apple Home at the moment - perhaps it doesn't use mdns to discover devices - or perhaps I don't know how to use Apple Home 😆

[winter reef]

Does the streaming work from your device ?

iOS device

[raw crag]

I don't understand your question fully sorry. Can you elaborate?

[winter reef]

This sounds like AirPlay is not working from your iPhone.

So my question does it work or not 😉

[raw crag]

You are correct that AirPlay from my iphone in Aus does not work to my PowerAmp in Romania. But, it works to local AirPlay devices.

[lime pewter]

Lets start by reverse engineering OwnTone then and see if we can extract that into a libairplay or whatever

[raw crag]

I am not sure what the reason is, but it may be that Authorisation thing you mentioned, but that doesn't account for why owntones does work.

[raw crag]

I think owntones has a library already, so hopefully we can use it.

[winter reef]

If you want to use Siri to control your PowerAmp like play Adele in the living Room then you need to Add it to Apple home. So maybe we can try to add to Apple home and try again with Raop?

[raw crag]

My bad - I was too hasty about owntones. Their library is a music library .... of course. I agree with your suggestion @lime pewter

I am heading overseas to ride motrobikes in the Himalayas for a couple of weeks from tomorrow, so I will make a start on this when I get back.

[lime pewter]

We can also look into pyatv, which also supports raop, airplay and airplay 2 - I'm pretty sure it has the same support and if you disable raop support, you can actually have working audio. But pyatv has no support for audio sync as far as I know but most of all is fully python which simply has too much overhead

[raw crag]

You might be right, but until I can see the PowerAmp in Apple Home, then I can't move forward. It will be mid-August until I get physically get onto the same network as the PowerAmp to try this with Apple Home.

[raw crag]

I reckon we stick with a C library to ensure we have the performance we need.

[winter reef]

The owntone repo is great they separated AirPlay 1/2 separated libraries. Interesting would be if the AirPlay 2 library can play to AirPlay 1 devices.

[winter reef]

Apple Home is not working through vpn yes that is a mess.

[lime pewter]

No, they have seperate Raop vs AirPlay support.
The issue is not AirPlay 1 vs AirPlay 2... Problem is that these devices lack RAOP support

You could argue that RAOP == AirPlay 1 but its strictly not the same, these protocols live next to eachother in the apple world

[winter reef]

Crazy only things Apple can do 😛😂

[winter reef]

But then can we mix AirPlay 2 + Raop + AirPlay 1 Devices with supported sync?

[lime pewter]

So technically AirPlay (1 or 2) devices should still be backwards compatible with RAOP but as we found out that is not the case. That is also why you need to hard disable RAOP support in OwnTone because it will try RAOP first. This is also the case in e.g. pyatv but also products like Roon have users reporting these issues so this still is a device issue.

If we are to make a new/better implementation, I would envision we would need a toggle to switch between legacy/raop mode and airplay 2 mode

So that as a user you can enforce AP2 only mode fo rdevices that have this raop issue

[lime pewter]

Yes, as long as the start NTP time is shared between them

[winter reef]

Or maybe starting playback and ask the user initially do you hear sound? Yes use Raop otherwise use the newer implementation

And we could maybe save that in a separated DB so we could auto-decide later on if we have that device in a MA Device Database so that I have Device x it will autoselect the AirPlay Mode.

[raw crag]

I think we can leverage from the same type of construct you created for broken raop models in the MASS code @lime pewter and for those devices that we know have borken raop models, we use AirPlay 2 - or perhaps even better, we extract the AirTunes version from the mdns info and then select our airplay version from there. I think we can still do ALAC with the airplay 2 devices, but I also suspect you are correct in that no one will know the difference if its not ALAC.

Ciao. Chat again in a couple of weeks.

[winter reef]

Have a great summer 😅

[raw crag]

I am making progress on AirPlay2 support. I've been down several rabbit holes and hopefully I am converging on a solution. I need some advice though please. In order to get the cliraop binary to work with AirPlay2 devices, I need to pass some of the decoded mdns properties from the _airplay service. Currently, the Airplay player only contains decoded properties from the _raop service. Can anybody advise how I can add the _airplay decoded properties to the _raop ones in the airplay provider? At a minimum, I need the "deviceid" property, but I may need more as I progress with testing.

[raw crag]

Actually, I have found a more elegant solution. I can uset RTSP GET /info request as the 1st exchange with the device and obtain the required information.

[lime pewter]

Oh nice! Great to hear you’re making progress!

[raw crag]

Thanks @lime pewter It is progress at a snail's pace though. I've been on a steep learning curve about MASS, owntones and Airplay, but I think I'm close to some breakthroughs and when that happens, we should have a working version soon after.

[lime pewter]

Amazing to hear! I'm on holiday currently so a little bit less online but feel free to ping here when you are stuck. Also be aware that we (slowly) working on a major refactor of the players controller and its models but I'll help you with incorporating your changes into that

[raw crag]

Enjoy your holiday

[raw crag]

I have been making more progress and am getting close to getting a successfully negotiated session established with my UniFi PowerAmp. I've ported across the pairing logic used in owntones and this supports multiple pairing capabilities. I have only been focused on Homekit Transient pairing for the moment, but will expand to include others once I have a working prototype and have other devices to test against. My next step is to get ciphering working for the RTSP session and then I think I'll be able to dive into getting the music streaming to work - hopefully that will be fairly straight forward, but nothing so far has been - so perhaps I shouldn't expect it to get easier now! 😆

The code is far from optimised at the moment and contains a mash of libraop constructs and owntones airplay constructs. I will seek to clean it up once I get a working prototype and now what I need and what is redundant or obsolete. If anyone is interested I am keeping my work updated on a daily basis at https://github.com/bradkeifer/libraop in the AirPlay-2 branch

[lime pewter]

Amazing work!

[raw crag]

I’ve made some decent progress and am ready to test and debug streaming to airplay2 devices.
I would like to use a test file containing audio data in the correct format to get the cliraop program to read from.
Do we have any such test files available in a repo somewhere or is there a standard file format that you would recommend please?

[winter reef]

@raw crag checkout this website: https://www.audiocheck.net/audiofrequencysignalgenerator_sinetone.php

this is my go to website for tone generator with specific settings

I have an AirPlay 2 only device here so I can also support in testing if needed. GREAT WORK!

[raw crag]

Thank you. I definitely would appreciate you doing some testing as soon as I have it working. I don’t really like the complexity of the solution at the moment, but I am sure i can fix that once we get the current approach working.

I think a cleaner solution will be to have a totally separate airplay 2 binary that is built entirely in top of the owntones code and is not grafted into the libraop solution. But, that can wait until we get an initial solution working.

[winter reef]

Yes absolutely. Is that AirPlay 2 solution AirPlay 2 only? Because my not using a simple switch inside the device settings (AirPlay 1 mode or 2 mode). Ideally the user starts the playback on that device and asks the user: do you hear sound? Yes => uses AirPlay 1 mode otherwise it will tried the AirPlay 2 mode

Additionally we send those settings back to MA repo so we can have a device list repo

[raw crag]

I’m sure we will find an elegant solution. I suspect that a seperate provider will be the go, because airplay2 works best from the airplay mdns discovery data and not from the raop mdns discovery data. I suspect we will find that there is a version number in the mdns discovery data that we will find tells us whether raop or airplay is the way to go. But I have done no testing for this yet.

[lime pewter]

Lets avoid a separate provider and just extend the existing provider. Also this should not become a problem of the user so if we can auto detect AP2 support, we should just pick the right stream. In the current AP provider you can already see that I splitted up the streaming part to account for this to happen.

[raw crag]

Hi @lime pewter - sorry for my hiatus, I have been busy building a home, but I am back on the job for airplay 2. I don't really like the code I've developed so far, but I want to get it working and then try a 2nd approach of having a seperate executable for airplay2 support, instead of having cliraop support both versions - I think that will be cleaner. Anyway, I have a question for you. I would like to get devices like Apple TV's working (devices which require a PIN on first connect). Have you given thought previously about how to obtain the PIN from the user? I was thinking that we might be able to do it in with an airplay-specific config entry - would that be ok with you, or do you have an alternative method that you'd like to see used?

[lime pewter]

Nice to hear back from you! I was about to ask if you were still hunting this one down 🙂
Yeah, for getting the user pin I wanted to use some config entry actions and maybe even leverage pyatv in between only for the pairing part but it should also be doable to do with the cliraop, I believe some early code to support pairing is already there but I just did not yet implement it.

Yeah, 2 binaries sounds a better approach. one for airplay2 and one for plain old raop

we can still sync raop and ap2 targets together by using the same ntp start time

[raw crag]

ok, perfect - thank you. Sorry I have been so slow on this - it's a combination of a lot of learning and also a shortage of time, but my time availability has now shifted for the next month or so.

[lime pewter]

Let me know if you need help on implementing the pairing part so I can have a look at that, it was on my todo list anyways. For now I think it would be a very cool first step to have AP2 streaming working next to RAOP en then on top of that we can start supporting devices that require pairing (= ATV)

[raw crag]

@lime pewter I have abandonded my work on adding ap2 support into cliraop - end result of that work is at https://github.com/bradkeifer/libraop/tree/AirPlay-2 It connects to the PowerAmp, but streaming is not working and I have not investigated why.

I have commenced work on a dedicated cliap2 codebase that leverages as much as possible from owntones. It is at https://github.com/bradkeifer/cliairplay

At this stage, I have been able to get it to compile, but it doesn't do anything useful yet. I am leveraging and aligning with owntones overall architecture design, because I think that makes sense if we want to minimise the codebase we need to support. Owntones uses the GNU autoconf family of tools for generation of configs and Makefiles etc. This is all new to me, but I am learning. I have not done anything yet with respect to cross-compilation for the various architectures that we need to support, but from my reading, this should be possible using the autoconf family of tools. If you have any feedback on my approach, please let me know.

[winter reef]

To be honest the documentation of the owntone is also not 100% accurate I can feel your pain 😉

[raw crag]

I think the ultimate solution is as you have outlined, where we use pyatv python code for pairing and C code for streaming - but I am pretty sure we will need to use cython, because the pairing creates sockets that we need to leverage in the C codebase - at least I think that is the case.

My plan is to get a cliap2 that has equivalent capability as cliraop, and then we can decide on whether another refactor of the approach is required after that.

[winter reef]

Agree and highly appreciate that you’re spending your time on it

[lime pewter]

Amazing work! I’ll have a play with the pairing part this weekend, shouldn’t be too hard. If that is working, we can apply it to both raop and ap2

[winter reef]

I can’t wait to test this on my Samsung soundbar and speaker 😉

[raw crag]

Question for you @lime pewter please. I have a partially working cliap2 and am ready to get it integrated with mass to continue development and testing. My question is about how to extend the existing provider to add support for cliap2. Do I create an ap2 equivalent of raop.py with classes AirPlay2StreamSession and AirPlay2Stream?

[lime pewter]

Yeah, that is what I envisioned when I created that airplay provider back then. I already separated the raop streaming part. And I think you can leave the streamsession generic so you will be able to mix and match raop and AP2 together (as long as they get the same NTP timestamp)

[raw crag]

OK, thanks. I've already created an airplay2.py and the two classes. I will stick with this for the moment, and when I get it working, I will see if I can collapse to a single streamsession.

I've also removed the raop terminology where it existed in player.py/provider.py to make it generic to handle both both raop and ap2. Once I get something that semi-works, I would appreciate if you could have quick look at the .py files I've changed and let me know if the approach is ok - will probably be tomorrow, but if I'm lucky, maybe today.

[lime pewter]

OK sounds good! I can have a look if I can do the auth part with the pyatv lib

[raw crag]

I don't think it's worth looking at pyatv yet, because I am pretty sure that the pairing and streaming must exist in the same process.

The process of pairing in airplay2 creates some of the sockets used for streaming.

[lime pewter]

Are you sure ? I had a brief look into it before and discovered that pairing is a one time process and then you have a key to connect with on all subsequent connections. It even partially exists in cliraop

[raw crag]

It's a couple of months since I had my head deep down in it, but that is what I recall. There are definitely one or more encryption keys that are derived during pairing and they are required for streaming. The pairing/connection handshakes for ap2 are quite different to raop

I could be wrong. It's been a massive learning curve for me to date.

[lime pewter]

Ah yeah, I never looked into it for the AP2 part, only for RAOP. Maybe RAOP is more simple.
What i was hoping for is that we can use pyatv to connect to official apple devices. So that way you can keep track of the actual volume level, what is playing oustide of MA etc. Then for actual playback we leverage the ap2/raop streaming. For generic airplay devices, we only leverage what we have now, by only connecting to the speaker/player at playback time.

Anyways, that is for later, let's not make it even more complicated than it already is 🙂

[lime pewter]

I can fully understand. Its amazing what progress you already made by biting fully into this, kudos!

[raw crag]

We have some improvements for the AirPlay provider in dev now. @lime pewter has done some awesome work to implement PIN-based pairing for devices suck Apple TV's, Macs etc using the RAOP prototcol and we also have initial support for AirPlay2 protocol, which will be helpful for those devices which were problematic under RAOP. We'd really appreciate it if people could give these new capabilities a good test please. There are some improvements to be delivered for AirPlay2 in the future, such as PIN-based pairing and possibly support for higher quality streams. But first priority is to get it tested across several devices to confirm if there are higher priority improvements required.

[winter reef]

If it’s in beta i can test with several Samsung AirPlay speakers

[raw crag]

Absolutely.

[winter reef]

Just ping me

Whenever it’s available

[fierce jetty]

It will be in the next beta

[winter reef]

Will report then!

[near rover]

I can also test on my Samsung sound bar (hw s61b) if that is needed. It would be so great to make it work! I was only able to make audio notifications working there through the smartthings API.

[lime pewter]

No need to test - we already know its not working.
It might not even be fixable at all as these devices behave different than others.
The only way that would be helpful is proving wireshark traces with some man in the middle approach to look in the encrypted data

[near rover]

Thanks. Unfortunately, I cannot help with this as I do not own any Apple device and for my Linux+ Android combo, I did not succeed in casting through Airplay :/

[winter reef]

I will add you to a chat group we already did that.

[lime pewter]

No need, I have enough other stuff on my plate and I do not have time to debug these edge-case airplay issues, sorry

[winter reef]

Okay then it’s fine 😉

We’re on it with @raw crag

[soft token]

I tried pairing my Apple TV using AirPlay 2 and pairing failed (didn’t even show the code on my TV). Looked at the logs at it looks like MA is using the wrong syntax when call cliap2, only using a single hyphen for command flags instead of two: https://github.com/music-assistant/support/issues/4484

[soft token]

Ah, looking closer at the code I think I see what is happening. It’s using the RAOP pairing function, but since the protocol is set as AirPlay 2 get_cli_binary(self.player.protocol) tells it to use cliap2 instead of cliraop.