We could put the RGB values of that pixel in a variable and manipulate that number and use the value to: change the color of lights, rotation of spotlights, movements of spotlights or something else to be controlled (sky is the limit) in sync with the video stream. A small area of the to be streamed video could be dedicated to communicate "data" in the form of the color of specific pixels, It could be our equivalent on how MIDI and DMX is used to author the lights/stage sets in a rock concert.
#Get the color from a specific pixel from current frame of the stream used in the media surface
sleek trail
midnight marsh
Yes, the video stream is probably the best channel to transmit signals synchronized with audio at the moment. A single pixel may be too small, though. Due to video stream compression, you should rather rely on a small area.
Being able to use the color on the client would be awesome indeed. But it would require client side scripts to be really powerful. If the motion is decoded on the sansar server, the synchronicity is lost on the way to the client.
silent zenith
I thought about this some time ago. We could use blocks of 8x8 or 16x16 (to get the color without compression errors), and then sample a point or several (averaging between them to really be sure) from the video "texture", and transfer it to the lights parameters. This would happen client-side most likely, to keep the sync intact.
sleek trail
Yes, the video stream is probably the best channel to transmit signals synchroni...
Yes we need a bigger block 9x9 would be great yes, bud we only need to read the middle pixel in mine opinion. Because (hopefully) the compression artefacts will not be for big in the middle .
sleek trail
I thought about this some time ago. We could use blocks of 8x8 or 16x16 (to get ...
YEs those sizes would be fine i guess, I still hope tht reading one pixel from the middle of the block is sufficient, because of the load on the system to do around 64 reads (8x8) and calculate the average
silent zenith
that's why I said 8x8 at least 🙂 because h264 encoding uses 8x8 blocks, just like the jpegs 😉 1 sample ber macroblock makes sense though, yes.
sleek trail
Yes client side would be SWEET, but isn't it true that the scripts that have to do the calculations for converting the RGB (8 bit 256x256x256 right?) to the values we need and than pluging them intothe lights run on server side?
sleek trail
that's why I said 8x8 at least 🙂 because h264 encoding uses 8x8 blocks, just li...
8x8 .... did not know that, thnx
I already did some testing on my client using autohotkey to read the pixeld, but there is something strange going on with the render of the client, the measured value is never the same and the measurement of the color differ 30 increments sometimes. I have tried to change the quality settings, turn off all the lights in the experience , but the measurements are the same. Is it because of the "non linear" renderer of the client?
silent zenith
Good question. If the darker elements differ much more than bright ones, then it's most likely due to gamma difference.
sleek trail
the colors change like 10 times a second, never the same
silent zenith
you could create a specifically made test video 🙂 I remember doing this for testing.
sleek trail
this is the problem
"problem" is a big word, for now i soved it by dividing the channel measured pixel value by 30, take the ABS number (drop the value behind the decimal) and multiply by 30. Than the measurements are stable
but this means that i can only use 256/30=8,3 values for my light per channel R,G or B
silent zenith
I see very slight color noise, I see value changes no more than 3-4 out of 255 for each channel 🙂
sleek trail
Yes, i know it seems not so much, but i have tried it with dividing by 5 and still the numbers would "shift" sometimes
aroud 10 it looked stable, but i have decided to use 30 in the final script, do not know why (anymore) it has been a while sins I worked on this
silent zenith
remember, you're screenshotting the render, with the light, which does change a bit now and then
if we divide 4 (our error) by 256 (full range of color for one channel) we'll get 1,6 percent deviation. A perfectly good value, I'd say
Also, when setting up the lights, 1.5% error is not too much
sleek trail
I have even made a box inside the experience that blocks out all the other lights, and look at a emmissive shader, but still the same problem
silent zenith
Well sometimes the color shift will be baked into the video itself. Especially if it's a gradient and not a solid color.
sleek trail
if we divide 4 (our error) by 256 (full range of color for one channel) we'll ge...
That is what i thought also , but nonetheless, i could not get is stable at that value
silent zenith
Usually it has to do with digital or film grain. When compressed, it presents as this "noise", slight change in values of the same pixel over time.
sleek trail
I had transmitted a solid color, thru the media surface , even just put an albido standard material on there...
just to see if i can get is stable...
silent zenith
Unless you use the 8x8 or larger squares filled with solid colors, you're bound to get such results, i'm afraid 🙂
Also, does your scene have any bloom enabled? That could also modify the colors.
sleek trail
desert isle
brazen tapir
Also, does your scene have any bloom enabled? That could also modify the colors.
@sleek trail When doing this type of analysis your smoothing/averaging can not be just spatial, it needs to be temporal as-well. OpenCV is an open source computer vision library has lots of handy tools for this.
silent zenith
Indeed, but with any temporal smoothing 8nevitably adds the risk of losing high frequency data. It's a delicate balance 😅
brazen tapir
Indeed, but with any temporal smoothing 8nevitably adds the risk of losing high ...
Yes, finding the right smoothing values is an art 😉 it’s basically about finding a match between our actual perception and the real data. Our brain does a lot of smoothing all the time. Ex: we usually don’t see the flicker of fluorescent lights, above 60hz
silent zenith
Or even 50hz in other parts of the world 🙂
sleek trail
UPDATE: so actually there is a tool to communicate DMX in a media stream called VRSL https://github.com/AcChosen/VR-Stage-Lighting that create 256 grey scale colored 16x16 pixel blocks to communicate a DMX value. From the site: "This system is powered 95% by shaders, including the actual method of reading the pixels from the screen. The other 5% is to enable GPU instancing for the shaders and certain properties. There is also basic "RAW" Udon support for when DMX Support is disabled for these shaders."
This method is used in VRCHAT for events. The actual program that create s the 256 greyscale blocks is running standalone, so we could totally use this for sansar based events also.
I am convinced that making a shader that can read pixels from a texture and place the grayscale value to a (global) variable for script to use will solve many things for this "nut to crack".
Yes , there will still be the latency of the values being communicate back and forth to the server and the client, But I think (hope) that this will be "tenth" of milliseconds.
sleek trail
preview of the gridnode in action
The gridnode program also has the possibility to communicate DMX over audio. The DMX data is dend through extra surround sound audio channels in a 5.1-enabled stream.This page discribed how to set up OBS studio to do this. https://github.com/AcChosen/VR-Stage-Lighting/wiki/VRSL-DMX:-DMX-via-Audio
silent zenith
Thinking, if it would be more efficient or less efficient to use a spare audio channel to do this 🙂
sleek trail
Perhaps this is a smarter way to communicate DMX data to the client, But perhaps decoding the extra surround audio channel will use more resources than using the shader solution.
silent zenith
yup, some tests are definitely needed