dusty estuary
Now With Much easier setup, batch processing and 2 second inference time
Install instructions can be found here
https://github.com/Night1099/PBRHeightBatch
Big improvement from last version, still can produce random artifacts or completely fail on some images. Images may require light editing. I am already working on even better versions ๐
Still has some color bias to dark and light
Showcase
https://imgsli.com/MjY0MjY3
crystal patrol
Curious why a max size of 1024?
dusty estuary
proccessing time, and res of diffusion model but while im at it ill make an arg you can choose output res
just fair warning, sd1.5 is based on 512x512 so going up to 2k or 4k may do weird things
crystal patrol
Is this going to be a image.resize?
dusty estuary
ill provide both options
crystal patrol
For now if I hacked in a image.resize((control_image.size[0], control_image.size[1]))
you think this would do?
still pasture
pretty damn impressive. I bet that brick heightmap could look awesome with a good normal to match
dusty estuary
tyty! ๐
dusty estuary
For now if I hacked in a image.resize((control_image.size[0], control_image.size...
im almost done
crystal patrol
I should really learn the value of patience ๐คฃ
dusty estuary
ok done new file is on github
crystal patrol
This is awesome regardless the image (which are all 2048 either height or width or both) its taking average of 1.3-1.7 seconds.
Also attached image, really shouldnt have been in my batch but ended up producing the following error:
Traceback (most recent call last):
File "Z:\SteamLibrary\steamapps\common\Star Wars Battlefront II Classic\GameData\OctoTex\textures\processing\height.py", line 55, in <module>
image = pipe(
File "C:\Users\NZXT\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "C:\Users\NZXT\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\pipelines\controlnet\pipeline_controlnet.py", line 1117, in __call__
image = self.prepare_image(
File "C:\Users\NZXT\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\pipelines\controlnet\pipeline_controlnet.py", line 790, in prepare_image
image = self.control_image_processor.preprocess(image, height=height, width=width).to(dtype=torch.float32)
File "C:\Users\NZXT\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\image_processor.py", line 507, in preprocess
image = [self.resize(i, height, width, resize_mode=resize_mode) for i in image]
File "C:\Users\NZXT\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\image_processor.py", line 507, in <listcomp>
image = [self.resize(i, height, width, resize_mode=resize_mode) for i in image]
File "C:\Users\NZXT\AppData\Local\Programs\Python\Python310\lib\site-packages\diffusers\image_processor.py", line 364, in resize
image = image.resize((width, height), resample=PIL_INTERPOLATION[self.config.resample])
File "C:\Users\NZXT\AppData\Local\Programs\Python\Python310\lib\site-packages\PIL\Image.py", line 2222, in resize
return self._new(self.im.resize(size, resample, box))
ValueError: height and width must be > 0
(image attached is literally the sliver above this message)
I need to run but will be right back and begin ingesting these, here is the output so far
still pasture
at a glance the upscales of the original textures look great. I'd love to see screenshots with the heightmaps added
dusty estuary
This is awesome regardless the image (which are all 2048 either height or width ...
i just pushed another update to handle this
one of my next models will be texture superres, this will hopefully be easier then height maps ๐
@still pasture could you pin my first message to this please โค๏ธ
still pasture
Now With Much easier setup, batch processing and 2 second inference time
Instal...
you got it boss 
dusty estuary
tyty
crystal patrol
what gpu do you have?
3090
crystal patrol
Funny when I actually started the script something started making a funny noise. Got real worried, then figured out it was the fridge xD
crystal patrol
Literally because at that point the sound started the script was stuck loading, on the first part. Man I did get worried it bricked my PC
def adjustPath(Org_Path:str):
'''
Adjust the given path to correctly save the new file.
'''
path = Org_Path.split("\\")
file = path[-1]
filename = file.split(".")[0]
fileext = file.split(".")[-1]
newfilename = filename + "_albedo." + fileext
path.pop(-1)
path.append(newfilename)
newpath = '\\'.join(path)
return newpath
adjustPath(image)
Also since the target audience is presumably Remix users. Consider adding a auto renamer to ensure images output with height in their name. The above function is a method, credit to Mehdi-Antoine (https://github.com/Mehdi-Antoine/NormalMapGenerator)
crystal patrol
sigh I apologize I took the quickest glance at the code, saw this line + args.append + '.png' and it still didnt click with me
dusty estuary
all good lol
crystal patrol
Curious are you still working on a roughness map model? If there is anywhere for improvement in my pipeline Id say its roughness map generation. Right now since I dont nesscarily agree with any of the AI models out there, Ive been simply grayscaling and inverting my diffuses
dusty estuary
Honestly now that the dataset is made up it wouldn't be that hard to start training, but no guarantee it will be a good result
I can do normal and roughness pretty quick, I want to do a superres catered toward textures (which shouldn't be hard either)
crystal patrol
Well in the meanwhile, Ive ingested around 47 height maps for anything really that didnt have an emissive map. Here are some before and afters
Brighter image is after, darker is before height maps
Aisde from those here are just a few more random images. Thats all, this is fnatastic great work much thanks!!
dusty estuary
That's great! ๐
dusty estuary
Ty Kim! Thanks for the help and input
ember palm
Release build when?
ember palm
dusty estuary
I dont plan on building this, i can package into an exe if you would like pretty easily, it would still be about 8 gb, might be able to prune size of model and stable diffusion files
dusty estuary
ok i was able to prune the base diffusion model by 6 GB so overall i could package it into 5.5gb for python dependenies + 2 gb base model +1.5Gb controlnet
for 9gb total
haha this is def just a lucky run but look at what it did here
i mean the background is wrong color
but its like
still pasture
Yeah it's literally perfect outside of the background. Unfortunately the background is extremely important lol
dusty estuary
wouldnt be hard to photoshop rq tho
ember palm
ok i was able to prune the base diffusion model by 6 GB so overall i could packa...
As long as nothing needs to be on the C drive that is fine
ember palm
haha this is def just a lucky run but look at what it did here
But the edges of the rocks
patent kettle
i mean, compared to other solutions for height map generation this is still drastically better
ember palm
Oh, definitely
The only problem is installation is just annoying for me because of pytorch and the cuda toolkit which install on the c drive no matter what
And I only have a 120gb ssd as a c drive
dusty estuary
@ember palm I have released a portable version for ya bud, bottom of readme in github
when ran first time it will download model stuff but it gets saved within portable directory
lemme know if you have any issues, i tested it without cuda installed
ember palm
Thank you so much
ember palm
How are the directories supposed to work by default?
There are no directory args for the exe by default in the portable
I've got no clue
dusty estuary
Root (height.exe)
โโโ Input
โ โโโ Input 1
โ
โ
โโโ Output
โโโ Output 1
Just need folder called input and a folder called output next To exe
Put an image in input
Then use example command in readme
The args also allow to put in custom paths
ember palm
nvm I didn't see the command
ember palm
Also "models--h94--IP-Adapter" is still downloaded on the C drive
ember palm
It is all good it is only 2.4gb a lot better than the 8 or so from before
still pasture
Wow this does an amazing job
@patent kettle check this out!
The heightmap there is super impressive
patent kettle
that's very impressive
and i hate that i didn't think to use a height map to simulate those planks, lol
that makes so much sense
still pasture
To be fair I didn't think of it either when we talked about it 
patent kettle
even he's impressed
still pasture
Barnyard was such a great game. It has no right being that good given it was ultimately a licensed game for a cartoon show
patent kettle
yeah definitely. that's why i want to Remix it ๐
it's so good for what it is. and the humor can be pretty good at points
still pasture
Have you looked into using normals to smooth out the character models
patent kettle
wish i had more time to work on it
patent kettle
Have you looked into using normals to smooth out the character models
that'd need vertex normals, and i have absolutely no idea how to implement those
if it involves mesh replacements, that's broken in Barnyard
still pasture
that'd need vertex normals, and i have absolutely no idea how to implement those...
I was under the impression that if you baked a higher poly version of that model (which was more rounded) onto the old model, then the normal map output would retain a good deal of the curvature in a way that might overcome the blockiness
I've found this to be the case when using a bake for the clone mesh in my project. A 3K tri mesh looked almost as good as the 20K tri
ember palm
I'll fix that later today srry
But for now I've just created a junction and moved it to the portable folder
patent kettle
I was under the impression that if you baked a higher poly version of that model...
i'll look into it at some point. right now i just have no capacity to even try ๐ข
pretty burnt out after the adventure in #1214800744000266250 and some IRL stuff
still pasture
i'll look into it at some point. right now i just have no capacity to even try ๏ฟฝ...
If you wanna send me the .USD for the character mesh I can try it and send the result sometime
It would be very easy to smooth them out in blender for baking
patent kettle
that'd be great, thank you
btw is there an easy way to preview meshes without opening them all in blender?
i can't tell what is what, and i have thousands ๐ข
still pasture
i can't tell what is what, and i have thousands ๐ข
Open the capture USD itself in blender. It includes everything inside it
patent kettle
hmm last time i did that, ๐ฅ
still pasture
Mod.usda or capture?
patent kettle
capture
still pasture
patent kettle
Barnyard splits things into like a million meshes with nonsensical divisions
still pasture
Yep, same with RC. It's a huge pain
patent kettle
i'll ping you in #1106420949257044061
dusty estuary
But for now I've just created a junction and moved it to the portable folder
๐
dusty estuary
it not great, way to much stuff going on
i think like maybe 1000 images were atlas in training of over 1 million pairs
an earlier version right after convergence did better on atlas but had huge color bias
lemme try that
nvm that wasnt better
still pasture
I mean the older model does seem more accurate, barring that I forgot what the original texture is supposed to be
dusty estuary
id be interested to know as well what this texture is of in halo @naive siren ?
looks like a cockpit?
naive siren
yeah
dusty estuary
OMG OMG OMG
I JUST HAD AN AMAZING IDEA
ya know how a big problem with these old textures in games has baked lighting on them
naive siren
a diffuse un-baker?
naive siren
i can imagine how you'd go about making the textures, but it'd be an extremely time consuming project
dusty estuary
na na na see ready
since i have the dataset
i render all the maps in like a blender path tracing scene take a photo,
make the basecolor the ground truth
and the render the condition
train
profit
๐
@patent kettle do you think that would work?
patent kettle
yes, that'll work. i was planning on making a model for it actually
i started to get it set up a while ago but just ran out of time
essentially:
this can definitely just be a normal model rather than diffusion, but having it at all would be good
patent kettle
diffusion? that'd be cool
dusty estuary
@crystal patrol are you around?
@patent kettle i did the same model on roughness and normal map, normal was a bust but roughness came out ok i think, has huge color bias like previous height before i finetuned it
what do you think?
this one is Color, top right Man made roughness, bottom right model made roughness
same thing here
im not in love with it
patent kettle
most of these would actually be better inverted
patent kettle
yeah i dunno what happened
one of my roughness models turned into a height map model lol
i'l thinking still btw. it's hard for me to visualize roughness from the maps for whatever reason
dusty estuary
not gonna be release version thats for sure, im working on other avenues
patent kettle
hmmmmmm yeah i'd say that there's a ton of problem areas ๐ฆ
i had the same issue where inverting some maps provided better results, but there's really no way to tell which ones should be inverted
an idea i had a while ago that you may be able to implement: adding an image recognition AI and using it to select different diffusion models based on which texture type it sees
or using it to determine if a result needs to be inverted based on mismatched roughness areas
dusty estuary
oh boi, thatd be a last resort thing for me
patent kettle
unfortunately as far as i can tell that'd be by far the best way to ensure accurate results
but it'd be a major headache to get running, let alone working for other people without issue
dusty estuary
yeh
crystal patrol
Let em get a hold of this roughness model ya legend!
dusty estuary
I'll get it to ya soon
Ok CC0 texture super res diffusion model is in the oven training on 4 v100s should be done in 75 hrs
dusty estuary
@crystal patrol https://github.com/Night1099/PBRRoughnessBatch
you dont need to install anything else new manually, it uses same stuff as last script
its not great but its something
crystal patrol
Dude the fact these are all python scripts ๐คฃ ๐คฃ ๐คฃ
This is my folder of scripts now
dusty estuary
lol
better normal and roughness models are coming soon hopefully and super res is training as we speak
crystal patrol
Amazing please keep it coming
naive siren
im not in love with it
the problem with diffusion based stuff like this is that for the most part the ai doesnt really know WHAT its looking at
for an example, looking at the image
i can say with certainty, that these are a bunch of square, tiled rocks with dirt & moss between the gaps
but the diffusion ai doesnt know this
it takes a quick look and thinks "hmm... similar to X" and does its best
infact, i dont think the AI even KNOWS what a heightmap even IS
dusty estuary
i agree thats what kim was saying earlier where an identifier model that would then prompt the model with more context
which now that i think about it
is already implemented in huggingfaces diffusers
so it might actually be easy to implement
patent kettle
that's a bit different than what i was suggesting, but it'd still be helpful
naive siren
another thing about this level of generation, is that even if a human were to remake them perfectly, they'd still not be entirely accurate
i also just noticed there are cigarettes in the image lol
the ai would have no idea what that is and just assume its rock
but here's a good example
how would the AI interpolate this?
patent kettle
all it does is learn what colors on the albedo are typically associated with a certain level of brightness or darkness on the height map
then it attempts to recreate it on input textures
it's difficult to get good results
naive siren
yup, that's because that's not a good way to do it
you'd get similar results just using filters in photoshop
but this is a touch better
patent kettle
these are definitely better than just basic filters
even my model was better when i tried to compare it
naive siren
indeed, but there is a caveat
patent kettle
but it's still not ideal, i agree. the problem is to improve it, you'd need a custom architecture which is pretty difficult
naive siren
i don't think i've ever seen sharp results from diffusion works like this
i dont think the AI would even know what a sharp height would look like
patent kettle
sharp height maps are generally bad
it creates sudden jumps in detail
the earliest version of this model did indeed have sharp edges. it just doesn't look good in practice
naive siren
it creates sudden jumps in detail
this can be good depending on what it is
for an example, a texture with both sand and sharp rocks would be optimal if the sand was smooth and the rocks sharp
however, it would also only look good if it had no sharpness artifacts, this is diffucult to do even by hand
patent kettle
yes, but what's better?
having sharp detail on all textures and it benefiting few of them
or having generally softer height maps that look good on the majority, but bad on some
it's pretty much impossible to do both unless you have separate models for the task
dusty estuary
that's a bit different than what i was suggesting, but it'd still be helpful
Sorry yeh you were saying a way to initiate different models, that jogged my brain to think of taking advantage of the prompt guidance part of stable diffusion
The models were trained with a caption to every pair so the clip does have some context
I just don't know how well the prompt generation models will do on 2d textures
dusty estuary
welp this first caption model is um
"Generated Caption for E9DE519158653959_diffuse_1_1_1_1.png: a close up of a stone floor with a cell phone"
patent kettle
lol
dusty estuary
gpt 2 model
Generated Caption for E9DE519158653959_diffuse_1_1_1_1.png: a stone wall with a stone wall and a rock wall
i may have to finetune my own
if im gonna do it
patent kettle
even if the prompts are partially wrong, are they better than nothing?
patent kettle
๐ฆ
ember palm
which now that i think about it
Yeah moondream does a pretty good job at identifying the image
dusty estuary
ill try it!
dusty estuary
Last 2 diffusion upscalers Ive tried training did not turn out well lol
naive siren
this is about as advanced as it gets when it comes to AI training
dusty estuary
this is about as advanced as it gets when it comes to AI training
?
still pasture
for an example, a texture with both sand and sharp rocks would be optimal if the...
I can confirm this isn't true. Even rocks look bad with sharp heightmaps. I know from experience, making my own textures
