Issues with integrating for native applications

[remote fog]

I'm trying to follow the directions on both of these pages:
https://docs.lookingglassfactory.com/core/looking-glass-bridge-sdk/integrating-native-applications
https://docs.lookingglassfactory.com/core/looking-glass-bridge-sdk/native-function-reference
But I'm running into an issue when trying to build my visual studio project. Since including bridge.h in one of my files the project fails to compile . I think the issue is with the _WIN32 ifdef section of bridge.h file. I haven't had to deal with an issue like this before so I'm not sure on what to do and would appreciate any help. Thanks

I'm using Looking Glass Bridge 2.5. Visual Studio 2022 v143. Windows 10 - 64 bit. And C++ 20

[wooden scaffold]

what it looks like is you have two different libraries both including windows.h but are conflicting. it's probably better that you build libraries for the 3rd party dependencies (e.g. .lib or .dll files) so that it's all not mixed up in the project.

[remote fog]

Thanks for the info. I tried removing any code that uses #include <Windows.h> (turns out only imgui used it) but I'm still running into the same issue. I tried making a .lib and .dll for both imgui and the looking glass code found here: C:\Program Files\Looking Glass\Looking Glass Bridge 2.5.0\runtime. I tried getting rid of all of my imgui code and any reference to it and it still gives the errors. I also tried (without imgui) not using a .lib or .dll for the LKG code and instead using the Additional Include/Library Directories on the paths provided on the website but it still doesn't work. I must be missing something/making a mistake but I just can't see it.

[ember locust]

I believe bridge needs to use vs 2019

At least that’s what we use to build it internally, not sure if that requirement extends to dev builds or not

@compact valve would know more

[compact valve]

We are releasing updates to this very soon which should help. Visual studio 2022 or 2019 works fine for the samples.

[remote fog]

As of right now I am stuck and don't know what to do. @ember locust you mentioned that the example code here: https://github.com/Looking-Glass/LookingGlassCoreSDK/tree/master is out of date and that we should follow the examples from here: https://docs.lookingglassfactory.com/core/looking-glass-bridge-sdk/integrating-native-applications. However, all I can find are just code snippets. Is there a complete example project that I can take a look at to help me with mine? Thanks

[wooden scaffold]

I haven't got a chance to try the SDK yet, but I hope to put together a simple example project (something that works with modern software, Vulkan/Metal/etc.), Just haven't got around to it, perhaps in the next couple weeks.

[compact valve]

We are planning to release sample projects very soon, likely this week.

I've been working on them for a week or so

[wooden scaffold]

sweet

[compact valve]

https://github.com/Looking-Glass/bridge-sdk-samples

The examples are finally public! This requires the latest version of bridge and provides two very simple opengl examples where I render a simple cube mesh. I am working on an .obj viewer that will be a more complex example.

feel free to open issues on github for any bugs or questions you have!

[remote fog]

Turns out the issue with redefinition was due to vulkan including windows.h. I fixed the issue by changing my code to this

//#define VK_USE_PLATFORM_WIN32_KHR <-- commented this out and included the below
#define WIN32_LEAN_AND_MEAN 1
#include <windows.h>
#include <vulkan/vulkan_win32.h>

I'm now running into an issue where when I run this line of code

if (!displays.empty() && controller->InstanceWindowGL(&wnd, displays[0].display_id)) {

I put breakpoints in the bridge_utils.h file and it isn't exiting early because the func variable is invalid/couldn't load the func from the dll, so it appears that the result of func(wnd, display_index) is returning false which is confusing. I updated to the latest bridge sdk version, have my looking glass portrait connected to my PC and what not.

[compact valve]

interesting bridge is connecting to your program and you are seeing the logs as I would expect, as you can see in the log bridge can see your portrait. does the example work for you?

[remote fog]

I just tried running the BridgeSDKSampleNative app on my machine and it does show up on my looking glass portrait, however it is very blurry and doesn't look good. I'm guessing its not working as expected. I'll try the BridgeSDKSampleNativeInteractive rn

[ember locust]

can you share a photo/video of what it looks like on the display?

just to double check, are your display settings correct in windows? 100% scaling, correct res?

[remote fog]

It was a 200% scaling. Just changed it to 100%, still looks the same. Resolution is 1536x2048.

That was BridgeSDKSampleNative. This is BridgeSDKSampleNativeInteractive

Definitely not as sharp as any of the videos that come on the device. Kind of has a 3d effect but it feels like as I move my head horizontally left to right, the cube move diagonally upwards to the right

For reference, here’s what one of the videos looks like on it

[compact valve]

it looks unfocused, if you click in the 2d window and scroll it will change the focus in and out, I have it focused for my go and landscape 16 but it might be different on portrait

[remote fog]

Didn't realize that you could do that. Scrolling fixed the focus issue

[remote fog]

For the issue where my application won't create an window instance. I can't really think of anything right now. I copied the code from bridge-sdk-samples\BridgeRuntime and created my own .lib from it in a separate project under the project's solution in Visual Studio. I'll try just referencing the code on its own and see if that does anything

[compact valve]

yeah, I should call out the controls in the readme better.

I have only ever used cmake to build the bridge inproc code, but I know the visual studio project generated by the cmake works, maybe it would be helpful to look at that?

[remote fog]

Just tried referencing the bridge-sdk-samples\BridgeRuntime and not my own lib. Still doesn't work.

[remote fog]

I just tried creating a blank visual studio project, and only including the bridge-runtime files. Giving me the same issue. I'm guessing there is something special in the cmake ... ? Or it needs to have the other included files (open gl, glm, etc)

[summer wren]

Looks like these samples only work with OpenGL. Is it possible to just send a quilt image to the display without OpenGL?

[wooden scaffold]

what's wrong with using OpenGL? You are going to need some library or API to display graphics.

[remote fog]

I figured out why I was running into issues with the InstanceWindowGL function even in the blank project. I've found that if you do not have this line of code:

glfwMakeContextCurrent(window);

then

controller->InstanceWindowGL(&wnd, displays[0].display_id)

will fail every time. This poses a problem for vulkan support: https://www.glfw.org/docs/latest/vulkan_guide.html

Vulkan requires that windows used for surface creation do not have a context. Since the InstanceWindowGL function call seems to require that the window have a context, it appears that as things stand right now, Vulkan cannot be supported.

[wooden scaffold]

you can definitely use GLFW with Vulkan. It may be an issue with how the app interfaces with the LKG software. I'd have to look at it to know the specifics, but there should certainly be a way to make this work

[remote fog]

The LKG .dll function call for InstanceWindowGL seems to always fail if you haven't called glfwMakeContextCurrent. But every time I've tried doing that in my code, when I try to call glfwCreateWindowSurface, it fails. If there is a way to initially create a glfwWindow without context, use that window to create a vulkan surface, then ... give that window context so we can call glfwMakeContextCurrent(window) then it should work

[wooden scaffold]

You can do that, but if the DLL is expecting a GL surface, then Vulkan won't likely work

[remote fog]

Assuming we only care about running our program on our own LKG display and that we won't frequently mess with it or it's settings, we can skip the controller->InstanceWindowGL(...) call all together. By debugging the example code, you can get the BridgeWindowData struct and just copy it over to a different application. We'll still have to correctly create the window ourselves but I was able to do so in my application. I got as far as successfully creating the LKGCamera and a windowed fullscreen application on my LKG Portrait.

However when I was looking at the rest of the example code, I noticed these functions being called in the main loop:

controller->DrawInteropQuiltTextureGL(...)
controller->GetOffscreenWindowTextureGL(...)

I'm guessing that I'll run into similar issues with both of those calls. There might be a work around like the one for controller->InstanceWindowGL(...) but I'm doubtful

[wooden scaffold]

I'd assume that they are using GL to do some sort of internal rendering (e.g. converting your render into some format for the display or processing it somehow) so those functions are likely necessary

[summer wren]

Hehehe, no I don't need OpenGL, I just want to display a quilt that I rendered out with Davinci Resolve

[wooden scaffold]

right, but in order to display something on the screen, you need some sort of API that works with graphics, such as OpenGL

[summer wren]

I see what you mean, though win32 should be sufficient to do that

[wooden scaffold]

I doubt for the kind of high resolution processing that is needed

this is not just showing a bitmap

when you take a quilt image to put it on a LKG, the software has to process the pixels (or perhaps sub-pixels) to the unique calibration of the pattern on the display. For the LKG Go, it's a 1440 x 2560 screen, and then the quilt could be 4K res at least, or even up to 16K. So we are talking in the tens or hundreds of millions of pixels that need processing, not something you can do with Win32/GDI/etc unless you want to wait 10 minutes

[ember locust]

You should use the rest api for that

[compact valve]

The issue here comes down to memory namespaces, bridge needs to do the shader work it does using opengl, we do support other api's in the sdk (opengl, dx11, dx12, and metal) by using interop api's to pass the quilt memory around between the different api's. I would assume you could do something like that with vulkan but I have not implemented this specifially.

You can see an example here: https://eleni.mutantstargoat.com/hikiko/vk-gl-interop-2/

[summer wren]

Oh yeah, or course, completely forgot about that. Thanks for clarifying

[summer wren]

Thanks for the pointer. I think this is the best sample on using the rest api from python to display a quilt, right ?
https://github.com/Looking-Glass/LKG-ComfyUI/blob/main/DisplayInBridge.py

[summer wren]

I'm stuck, is there maybe some sample code that uses the rest api from python ?
As a non-web coder there's a bit too many unknowns here :\

[wooden scaffold]

the code is here, it doesn't look that bad, but honestly you should be somewhat familiar with web dev: https://github.com/looking-Glass/bridge.js#readme

[remote fog]

Thanks for the article. If I am to understand correctly, until vulkan support is implemented on your end, because I do not know what explicitly happens for

• controller->DrawInteropQuiltTextureGL(...)
• controller->GetOffscreenWindowTextureGL(...)
  I should:

1. Create a vulkan application that does headless rendering
2. Create an OpenGL application that is able to draw to the LKG
3. Using the article you posted, have vulkan and opengl both have the ability to manipulate/control shared image/textures

That should give me the ability to keep my current code that does the rendering with vulkan, but also give me the ability to call the .dll functions

Does that sound about right?

[wooden scaffold]

I didn't check that article, but the keyword is "interop" e.g. "vulkan/opengl interop". I've never done it, but from what I understand it can be non-trivial depending on the application

But yeah, it's something like that. You have a Vulkan app drawing to an off-screen render target and then transfer the texture to some shared OpenGL context

[summer wren]

That was the first thing I tried, but as a non-web dev... trying to use python instead of js and it's dependencies trip me up
Something simple like this should work, but no
import requests response = requests.get(f"http://localhost:33334/bridge_version") print (response.text)

[ember locust]

You’d still need to include a request body, I’ll try to get a basic example up but might be pressed for time before thanksgiving break

[summer wren]

Much appreciated, no rush 👍

[compact valve]

Sorry I just saw this response.

You would be surprised how easy it is, I have never done vulkan <-> opengl, but I have done alot of cuda <-> opengl, and dx11 and dx12 and metal. The data is in the GPU memory, and simple textures (without compression or mips or anything more complex) all use the same memory layout. In this case vulkan has its own memory space that it manages, what you need to do is tell vulkan you are going to share the memory, then take that pointer you get and have opengl use it as a texture.

Reading through that article a bit. You need to allocate the texture with this magic stuff: VkExternalMemoryImageCreateInfo and then you use glCreateMemoryObjectsEXT to tell opengl you are going to create an object that uses memory external to opengl, then you just tell opengl the texture metadata, like format and size and the like and it can access it!

[summer wren]

@ember locust, @compact valve can you help me with an example of how to use the rest api from python ?

[compact valve]

I can get you something more tutorial like in the next few days however this comfyui node does everything you would need:
https://github.com/Looking-Glass/LKG-ComfyUI/blob/main/DisplayInBridge.py#L305

There are a few parts, the BridgeConnectionHTTP clsss maintains the api connection to bridge, then the playlist and playlist item classes are there to help you manage the playback, If you have specific questions about the python code I can help

[summer wren]

Indeed that looks right up my alley, I tried to get that to work but I got lost in the weeds
I'm strugling with the fundamentals. Something simple like this should work.
import requests response = requests.get("http://localhost:33334/bridge_version") print (response.text)
But no... What am I missing ?

[compact valve]

what happens when you call that code? does it timeout or just print nothing

bridge might not respond with any text if you dont setup an orchestration

try extracting out this function that initializes bridge and running it:
https://github.com/Looking-Glass/LKG-ComfyUI/blob/31c870769e4fc3bdad4e43802f9074a20cf0ba5f/DisplayInBridge.py#L290

[summer wren]

It exits after a couple of seconds with "Process finished with exit code 0"

I tried copying DisplayInBridge.py to my project and using this

from DisplayInBridge import BridgePreview bp = BridgePreview() bp.connect_to_bridge()

The result:
`Exception in thread Thread-1 (_web_socket_listener):
Traceback (most recent call last):
File "C:\Users\patri\AppData\Local\Programs\Python\Python312\Lib\threading.py", line 1073, in _bootstrap_inner
self.run()
File "C:\Users\patri\AppData\Local\Programs\Python\Python312\Lib\threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "D:\Projects- Looking Glass holograms- Coding -\Looking-Glass-batch-tools\Davinci Resolve pipeline\Development\DisplayInBridge.py", line 158, in _web_socket_listener
websocket.enableTrace(True)
^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'websocket' has no attribute 'enableTrace'
Entered orchestration 'default'.
{"name":"value","orchestration":{"name":"orchestration","type":"WSTRING","value":"{05DB4C84-BFE4-44BF-AB36-EDA2F74F3FD8}"},"payload":{"name":"payload","type":"UNSIGNED_INT","value":0},"status":{"name":"status","type":"WSTRING","value":"Completion"}}
Exited orchestration and cleaned up the bridge connection.

Process finished with exit code 0`

[compact valve]

it seems like it connected, that websocket.enableTrace() error might just be a chatgpt hallucination, I wrote the comfyui stuff very quickly and might have left that in, I think if you remove that line it should be fine

[compact valve]

I checked and that error also comes up in comfyui even when the bridge casting works so it shouldnt hold you back

[summer wren]

Yep, it's alive 😀
removing all lines dealing with websocket (158-164) produces this
Attempting to connect to bridge... Connected to bridge successfully. Entered orchestration 'default'. {"name":"value","orchestration":{"name":"orchestration","type":"WSTRING","value":"{F3CE37F9-CD08-4FFE-BA20-1BB7FBB73552}"},"payload":{"name":"payload","type":"UNSIGNED_INT","value":2},"status":{"name":"status","type":"WSTRING","value":"Completion"}} Exited orchestration and cleaned up the bridge connection.

[compact valve]

perfect!

[summer wren]

Thanks for getting me unstuck, I can build on this

[compact valve]

awesome, let me know if you need anything else. I think you may be one of the first people to use this api outside of the company so let me know what you make with it, it would be cool to see!

[summer wren]

Hehehe, bumpy road ahead, got it. I plan to add quilt preview to my Davinci Resolve post-render script for quicker iteration

I'll certainly share my findings

[compact valve]

LOL trust me I know how that goes