segment anything causes out of memory on 6gb card, when it only needs 4.5gb max vram

[high verge]

Ive been trying for a long while to get an object selected with sam_vit_large ... It should have enough ram, with or without partial loading but crashes every time with OOM.

                          Node   Calls   Seconds  VRAM Used
              segment_anything       1    1.313s     4.606G
TOTAL GRAPH EXECUTION TIME:   1.313s
TOTAL GRAPH WALL TIME:   1.313s
RAM used by InvokeAI process: 1.22G (+0.289G)
RAM used to load models: 2.39G
VRAM in use: 2.459G
RAM cache statistics:
   Model cache hits: 1
   Model cache misses: 1
   Models cached: 1
   Models cleared from cache: 0
   Cache high water mark: 2.39/0.00G

I tried with/without partial loading in yaml
set working mem to 1gb all the way up to 5gb, no change in output

system: ubuntu 22.04

Trying to enable pytorch_cuda_alloc_conf: "backend:cudaMallocAsync" as stated in the low vram page just crashes it, so I havent tried that

File "/home/stubkan/DriveTwo/Diffusion/InvokeAI/.venv/lib/python3.10/site-packages/invokeai/app/services/config/config_default.py", line 480, in load_and_migrate_config
    raise RuntimeError(f"Failed to load config file {config_path}: {e}") from e
RuntimeError: Failed to load config file /home/stubkan/DriveTwo/Diffusion/InvokeAI/invokeai.yaml: 1 validation error for InvokeAIAppConfig
pytorch_cuda_alloc_conf
  Extra inputs are not permitted [type=extra_forbidden, input_value='backend:cudaMallocAsync', input_type=str]
    For further information visit https://errors.pydantic.dev/2.7/v/extra_forbidden