Camera feed to python

[dire galleon]

Lets make a thread, this is getting to confusing 😄

So, again. You get a camera feed from a system. And you want to pass in that feed from unity to the python part. And then get a updated feed back?

[leaden orchid]

btw, for testing purposes it might be useful to use a simple camera texture and use the laptop's webcam

[leaden orchid]

yes, tho there is maybe a faster way, because I don't really need an entire frame back, but only object detection, so I can send the frame, do object detection and let python, not send me an entire frame, but a list of coordinates and I do the object detection myself

that, or simply let python do the rendering on top of the frame i sent it, and get it back

I have read about
Inter Process Communication (for windows)
so I do have an idea of how those stuff should work, but could not find that much info about how to use them in unity, so anything would be useful
thanks in advance

[dire galleon]

Yeah. its hard to help when you are not sure, how you gonna do it. Can you maybe write down, what exactly needs to be processed where? As you said, you do not need the detection back as an image array but rather just an array of object positions and maybe ids so you can instantiate your object according to the cameras view, right?

[leaden orchid]

so what is needed is simple
get a frame from a feed. //already done.
pass it to the python program that does object detection (which can either return an image, or coordinates with the frame number and object type and 2 points (boxes around the object) )

render the frame with object detection on it

[dire galleon]

Okay, so the obejct detection, is that like a visual line on top of the camera frame?

[leaden orchid]

yes, with some text like what is this, a cat, a dog, human etc (python also returns the name of the object)

[dire galleon]

And the object of that visual line is just an array of coordinates?

[leaden orchid]

yes
specifically 2 coordinates,
so what python produces is something like this:

1, x1,y1, x2,y2, object_type, confidence_level

where 1 is the frame number, x1,y1 x2,y2 are the box border
type is a string of predefined types.
confidence level is just how confident it is

[dire galleon]

Okay, so just throw a nice callback to your Unity thing, maybe as JSON because its the easiest to work with and then just create or update your objects based on the type or id.

[leaden orchid]

sounds interesting, having a callback means I will not worry that much about syncing
but how do I register the python application to get the output into a callback in unity?

[dire galleon]

I guess you have to use some networking / websocket to throw back a string storing your json update. You might need to keep it in sync if you use the camera on Unity but update your detection with python. Either do some math in predecting movement or up the sync cycles from python

Am I predecting right that you are on a local environment all the time? Like unity and python are on the same machine?

[leaden orchid]

Yes is true.

[leaden orchid]

I don't think I need the prediction to be done on unity, because that is being sent to the original sender of the frame, and it does the prediction (tho it is not perfect)

[dire galleon]

Maybe thats just something to test against. But yeah, I would go with that simple update of a list compared to your json values as you have all the visual authoring be done in your unity environment and only data is being sent from outside.

[cosmic meteor]

if this only needs to run in the editor / with the editor on a desktop, you can use the official unity python ml package, which has a remoting (i.e. two way communication) solution between a python process and the editor

[leaden orchid]

what do you mean when you say only in the editor?

[cosmic meteor]

do you need to distribute an application

that people use

or is this for something experimental

where you will be well served by hitting "Play" in the unity editor

[leaden orchid]

distributed on a windows10 machine

[dire galleon]

Ye he was talking about a client, so I assumed it is being build for a platform