ComfyUI Style Transfer Quilts | R&D | Looking Glass | Page 1

cerulean flume Nov 26, 2024, 12:29 PM

#

I'm launching a new R&D on Style Transfers (IpAdapter) on already finished quilts. I'm looking for a “simple” way to make visually interesting spitial images - especially painterly images and those whose matter cannot be easily reproduced with RGBD or 3D renders. As we know generating AI images that are consistent among themselves is still difficult (locally) So I took mainly my 3DGS scans which are a good base for compositing. They have full depth and detail so they are good to look at. The current workflow in ComfyUI is not very friendly. Long way to go before I can find a way that won't flicker and won't require a GPU worth an apartment 😄 I'll describe the whole project in detail when I manage to get to good results. I am currently testing different models and different methods. The results are mixed but I can see the potential.
Small Playlist: https://blocks.glass/playlists/0bda9319-a18c-47a7-8dfb-1e181fd9de27 (two examples)

Looking Glass Blocks

Style Transfer R&D | Looking Glass Blocks

The first holographic sharing platform built for 3D creators.

#

shut sphinx Nov 26, 2024, 12:32 PM

#

:O

cerulean flume Nov 26, 2024, 12:35 PM

#

For now workflow is:
Quilt -> Extract every 4th view -> 17 views -> Unsampling -> Ipadapter+Controlnet -> batch generative -> 17views (new style) -> Frame Interpolation with FILM VFI -> Combine Video od 66frames -> Extract Quilt from Video.

#

17 views because VRAM limitation (for now I have only 8GB). I'll test with 16GB better workflow

#

ComfyUI Style Transfer Quilts | R&D

restive bear Nov 26, 2024, 12:39 PM

#

Looks sick!!!

restive bear Nov 26, 2024, 4:46 PM

#

I have to check these on the device. Sadly the LKG is broken for Linux, so I have to boot into another Windows machine just to see any content (or upload them to Blocks and sync, which takes forever). Let me try on my other computer

cerulean flume Nov 26, 2024, 6:30 PM

#

Now I working on my first custom nodes to Comfy to make straitforward workflow. I make some test with better GPU and... consistancy is awesome. Nailed it. I think this will allow us to go wild with visuals

restive bear Nov 26, 2024, 7:26 PM

#

Where does the original video come from, is that just shot with a camera?

cerulean flume Nov 26, 2024, 7:37 PM

#

this is original - quilt from Gaussian Splatting scan (made with https://webxr-model-explorer.vercel.app). With small python script (made with Cloude) I convert quilt to images them to video.

#

Now I convert this script in Comfy nodes to managing Quilts <-> Images <-> Videos in Comfy.
Videos is only for showcasing, only fast preview of change / style transfer effect. Quilts are hard to inspect sometimes.

cerulean flume Nov 26, 2024, 8:19 PM

#

This is two simple custom nodes (for now) to split Quilts for Batch images and reverse - this is needed so that other nodes and models see the quilt not as a whole but as a sequence of images to make operation on this - unsampling for example.

cerulean flume Nov 26, 2024, 8:37 PM

#

Video to Quilt is possible too

#

restive bear Nov 26, 2024, 10:44 PM

#

Interesting thanks

cerulean flume Nov 27, 2024, 6:33 PM

#

Update for first full comfyui workflow! Share files and tutorial soon.
(SDXL / AnimateDiff / 2xControlnet / LORA / Ipadapter)

Changlelog:
Simplify process:
Input - Quilt image
Output - Quilt image
Video is optional.
Support to different sized quilts.

Gen times (4070 TI Super 16GB)
6:30min - for first gen
5:50min - style image and prompt change
4:30min - style image change

cerulean flume Nov 27, 2024, 8:28 PM

#

Crazy. This is addicted. Love this reflections.
Holo: https://blocks.glass/kotlesny/133295

teal stump Nov 27, 2024, 11:24 PM

#

Pretty cool stuff!

frail geode Dec 1, 2024, 12:54 PM

#

want to share the comfyui workflow?

placid quail Dec 3, 2024, 8:40 PM

#

I know I'm not supposed to pick favourites but c'mon

#ComfyUI Style Transfer Quilts | R&D