Communication between Unity and Python

[summer elk]

Hello, I'm trying to implement a functionality for my thesis. Basically, in ML-Agents, the observations made in the Unity environment are sent to a neural network through a communication bridge. What I want to do is implement a hook that intercepts these observations and sends them to an LLM instead, but I can’t quite figure out how the communication between Unity and Python works, or how I can redirect the observations to another module instead of the network. Do you have any idea?

[untold nebula]

The most common way is sockets.

Aside from that, web requests, named pipes.

Depends on the OS too. Pipes are on windows only. Other OS might have a similar feature.

[summer elk]

but i have to redirect the traffic from unity to python, right? i remember for sockets you should specify the port number, or something like that. so i have to modify unity ml agents code in order to do that, right?

[untold nebula]

but i have to redirect the traffic from unity to python, right?
I don't know. Isn't that what you're asking..?

Ah, I see, so you want to intercept the observations in unity before they are sent to the unity ml-agent model. I thought you're talking about some third party stuff.

The easiest way would probably be to look at the C# side and see where the observations are consolidated and sent to the python side. Then hook there and redirect them wherever you want.

[summer elk]

yes basically i think there is some IPC between unity and python that lets the observation vector to go from unity to python, what i want to do is inject an hook that lets unity send the observation to another module, but i cant quite get how the communication work (the documentation is not so clear on this, there are some classes that do the communication but its not so clear how it works...)

[untold nebula]

You'll need to look at the source code.

[summer elk]

yeah i made the thread to ask if someone knows how it works 😭

[untold nebula]

I really doubt someone ever had to deal with it, aside from unity devs.
Youll need to research on your own or hire someone to do it for you. Granted, it's not that complicated and most mid level devs should be able to figure it out.
You can ask the community specific narrow questions, but we won't do the whole job for you.

[slow marsh]

Find method ExchangeData() in Packages/com.unity.ml-agents/Runtime/Communicator/GrpcCommunicator.cs. Before you send AgentInfoProto, forward the observation buffer to your own pipe.

[summer elk]

i wondered it could be a better idea using the plugin system that is raccomented on their documentation (i think it's the the way extensions could be made) so that you can make your custom trainter and your custom policy. but i'm having a little bit of trouble figuring the role of the trainer and the policy.