yolov8 questions

[golden dew]

1. how do i know if a yolov8 file will work with swarm? i have found some that do, and some that don't, with no clear indication why y/n (edited to add: _seg in filename not being an indication of what works)

2. with those that do work, as in detect the right segment, i get not-so-great results, because the yolo process seems to not be aware of the nature of the picture it's editing (?) - i get colour mixed into b/w faces on eye correction (noticeably so the nose) and

3. REALLY visible leftovers from the masks, especially with finger correction, which leaves a darker-than-the-rest splotch, see pic. what am i doing wrong there?

edit: results below done with the segment syntax
<segment:yolo-yolov8x.pt,0.6,1>perfect detailed eyes <segment:yolo-hand_yolov8s.pt,0.6,1> perfect feminine hands

[silver dew]

I am no expert but I can tell you what I know.

Yolo models will produce three informations for you, say you have this picture, this is the information the Yolo model will find and give back

here is an extract :

{
"name": "bowl",
"class": 45,
"confidence": 0.42298,
"box": {
"x1": 319.21866,
"y1": 285.01895,
"x2": 413.40848,
"y2": 323.51129
}
},
{
"name": "cup",
"class": 41,
"confidence": 0.37787,
"box": {
"x1": 492.77896,
"y1": 255.08443,
"x2": 530.28473,
"y2": 312.61847
}
},
{
"name": "dining table",
"class": 60,
"confidence": 0.36008,
"box": {
"x1": 280.54449,
"y1": 315.3139,
"x2": 811.18756,
"y2": 493.58661
}
}

here's a render of that information

what matters for us here is the type of object and the confidence

when you use segment:texthere,creativity,threshold
or segment:yolo-MyYoloModel.pt,creativity,threshold

the texthere is for the type (or another yolo model) and the threshold is for that confidence

So, if the model doesn't seem to detect/work in your case, it could be that the threshold value is too high

Now for the leftover marks, it could be related to your segmentation settings in the UI and/or to the creativity setting that would be too high and redraw that portion of the picture (determined by the coordinates that the yolo model gives back, see the .json)

If you use different models, you get different data and also different confidence depending on the model

If I use a specific yolo model to detect nails in that picture, then I get 0 matches.

[golden dew]

i should have added that i'm doing all of that within the prompt, with the segment syntax
<segment:yolo-yolov8x.pt,0.6,1>perfect detailed eyes <segment:yolo-hand_yolov8s.pt,0.6,1> perfect feminine hands

[silver dew]

Yes, but you are requiring perfect confidence (threshold = 1)

And that will not always give results

Try to lower the creativity to 0.4 see if you still get the blurry box

Maybe as you suggest, there is something to do about black and white pictures, maybe you should add back that information to the segment ?

eg: segment:yolo-yolov8x.pt,0.6,1black and white picture, perfect detailed eyes segment:yolo-hand_yolov8s.pt,0.6,1 black and white picture, perfect feminine hands ?

I'm not certain but I believe a segment has it's own prompt, so you maybe need to add sufficient context in it

[golden dew]

i tried that, not very successfully though 🙂
one of the reasons being that this prompt produces b/w and colour pics rather randomly
will have to play some more, or wait for developments yet to come

and thank you for your input!

oh, and the creativity is the middle number, the 1 is required for no obvious reasons

To control the creativity with a yolo model just append ,<creativity>,1, for example segment:yolo-face_yolov8m-seg_60.pt-1,0.8,1 sets a 0.8 creativity.

[deft hatch]

The 1s are the threshold (of confidence) required to say x is an x. If you lower that number, the model will be more likely to identify things as x. Too low and it will identify things that clearly are not x as x.

A 1 value there is likely what is causing issues.

[silver dew]

"You can also do yolo-modelnamehere-1 to grab exactly match #1, and -2 for match #2, and etc" cf. https://github.com/mcmonkeyprojects/SwarmUI/blob/master/docs/Features/Prompt Syntax.md#automatic-segmentation-and-refining

My understanding is that if you have two people (two matches) yolo-MyPersonModel.pt-1 will apply on 1st person, and yolo-MyPersonModel.pt-2 will apply on 2nd person

Then the -1 suffix is absolutely not mandatory but could help target specific area

In the end you can use segment:yolo-face_yolov8m-seg_60.pt-1,creativity,threshold or segment:yolo-face_yolov8m-seg_60.pt,creativity,threshold if you only have one face in your picture.

[deft hatch]

sorry, i meant the one after the creativity.

i probably misunderstood what you wrote earlier.

[silver dew]

but yes the threshold must be high enough to not mix hands and spoons like brendan said

[polar dew]

Yolov8 is a suite of multiple models, and any segment/mask/boundingbox models are supported. World models would need a new syntax to support but could potentially be added if there are good n useful ones.

if you have a model that logically should work (ie it's designed to detect a thing and region it) but doesn't, gimme specific info and can probably make it work

[golden dew]

detection is not really an issue, it detects faces or fingers all right, it's what it then does, which is mixing colour into b/w pics, and leaving highly visible mask remnants everywhere it applies itself

and if i put the stuff that makes the pic b/w in the regional face prompt, that seems to confuse the model greatly

but it's a bit moot, because my expectations were outsized, as in i thought it might fix stuff that went south in generation, which it doesn't really - like additional fingers 😉

[polar dew]

oh the example in your op post is mostly starting with, bad model that gave you a big square blob mask

but also: you can try mucking with Advanced->Regional Prompting parameters, or the segment creativity value

or different models/prompts

[golden dew]

i tried with the ones you link in the documentation too

https://huggingface.co/jags/yolov8_model_segmentation-set/tree/main
are those compatible?

[polar dew]

at a glance, should be

"yolov8 segmentation" sounds right

[golden dew]

changed to be comfy compatible, they say

i did not see clipseg-rd64-refined-fp16 being used anywhere in the comfy workflow, is that a worry?

[polar dew]

that's only for non-yolo segments, which you'll see as a SwarmClipSeg node

[deep pasture]

3 is just because you put too much denoise

[golden dew]

denoise where?

[deep pasture]

in your syntax <segment:yolo-hand_yolov8s.pt,0.6,1>
the first number 0.6 is the model has to think that it's a hand with that confidence
the second number 1 is the img2img denoise value

if you do that much of course it'll lose every info about the image

all it'll remember at 1 denoise is a bit of the color so it gave you a darker shade of grey because it doesn't know the actual shade of grey you had

do 0.6 denoise and see how it manages

[golden dew]

segment:texthere,creativity,threshold
other way round?

[deep pasture]

🤔

[golden dew]

To control the creativity with a yolo model just append ,<creativity>,1, for example segment:yolo-face_yolov8m-seg_60.pt-1,0.8,1 sets a 0.8 creativity

from the swarm docs

[deep pasture]

oh yeah

[golden dew]

i did play with the creativity values

[deep pasture]

odd that it ends up giving you such a different color

[golden dew]

not the certainty 1 though, but that was not an issue as such

[deep pasture]

0.6 should only denoise enough to alter the image but not change the color

[golden dew]

i tried 0.4, which barely changed anything, 0.8 really buggers it up
between those, it quite suddenly changes from too little too too much 🙂

[deep pasture]

what is the model you're using?

[golden dew]

ermm, let me look what that was

[deep pasture]

I think it might have been badly trained if it ends up doing that

[golden dew]

https://civitai.com/models/415107

those guys generally know what they're doing

[deep pasture]

doesn't look like a well trained model

[golden dew]

i get good results otherwise 🙂

[deep pasture]

seems extremely overfitted on specific images

[golden dew]

fwiw, i tried with a colossus model too

same same

[deep pasture]

🤔

[drowsy oxide]

how we were setting segment creativity*