VMWare Workstation - Post install & first boot up error "Xsession: unable to launch"

[silent elk]

Hey there. I have been trying to get Vanilla OS to run in VMWare Workstation 17.6.3 but keep encountering the same error, when loading into the OS for first time after install:

"Xsession: unable to launch "env GNOME_SHELL_SESSION_MODE=firstsetup gnome-session -- session=firstsetup" X session --- "env GNOME_SHELL_SESSION_MODE=firstsetup gnome-session -- session=firstsetup" not found; falling back to default session."

The install ran & completed just fine, prompted me to reboot, load into "Current Stats - A" but gives this error when starts to load desktop.
Any help please?

VM Specs:
Normal VanillaOS install option
4GB memory
2 Processors
100GB VHD
Hardware acceleration - off

[silent elk]

They there. I am still looking for help with this is anybody can lend a hand please. Many thanks 🙏

[mortal marsh]

I have vmware workstation 17, I will give this a try and update you here in a bit.

[patent valley]

Which image did you install? vanilla-os/vm or vanilla-os/desktop

[silent elk]

I downloaded the Vanilla OS 2 Orchid iso (VanillaOS-2-stable.20250220) from official website.

I am checking the site again but not seeing a VM download option. Only the iso

The only thing related to VM install is this guide , which I followed anyways -
https://docs.vanillaos.org/handbook/en/install-in-vm

[patent valley]

This guide is absolutely useless as it is for Kinetic, the previous major version of Vanilla.

[patent valley]

image and .iso are mean completely different things in context of Vanilla OS.

[silent elk]

Ah ok. I thought it was for Gnome Boxes section

[silent elk]

Where can I find the VM version?

I can"t see it

[patent valley]

You're gonna need to boot the installer again, and select Install Custom Image (Advanced).
When it asks for the image, specify ghcr.io/vanilla-os/vm:main (edit: corrected x2)

[patent valley]

You won't see it there, it's here: https://github.com/Vanilla-OS/vm-image

[silent elk]

great thanks 🙏 I will try this

[patent valley]

One sec, I have a mistake in the URL and Hyper-V is giving me some troubles...

Seems to be installing now. The URL is ghcr.io/vanilla-os:vm

[coral oasis]

I don't think that will fix the issue.

[patent valley]

Why not? open-vm-tools works with VMware in my experience

[coral oasis]

Yeah, but I don't see how it would fix the issue that the firstsetup X session doesn't work.

[patent valley]

Good point actually. I looked at this and thought "well they're trying to install in a vm, there's a vm image so something in the desktop image is probably broken enough to have required this vm image"

[silent elk]

I am currently trying this but seems to just hang on SETUP 21/21 in the logs

[patent valley]

I can't confirm that it worked for me because my VM is still busy installing

[silent elk]

Oki doki 👌 Maybe it takes longer to install via the custom install with URL method

[patent valley]

same actually. I might still have the url wrong 🫠

[silent elk]

no worries. If it doesn't work it doesn't work . Thanks anyways with the advice 🙏

[patent valley]

I don't think so. It would either way just pull from ghcr.io

Did the installer show you the 'Do you want to install VM Tools' screen at all? @silent elk

[silent elk]

I gave up on the install - Didn't progress any further .
I had the yellow bar at bottom when I first booted the VM with "Install your OS as you usually would ...(blah blah blah) I didn't remember if that was for the VMware tools

[patent valley]

No it is technically not, it essentially installs open-vm-tools and those work with VMware too. Afaik VMware ships a couple of .isos that have VMware tools for Linux on them, but you don't necessarily need those, open-vm-tools work too.

[patent valley]

You might be right here. vm vs desktop did not create a single difference for me, but neither did I run into the original issue. Wasn't this an issue in older installers/installer media or so, I remember having seen an image of this before here on the Discord?

[coral oasis]

Not sure, I think the issue is with the graphics driver for the hardware accelleration of the VM being used.

It's trying to use the X session instead of the default wayland session for some reason.

Since we got rid of the custom session in the new first setup it should work just fine in the next upgrade.

[patent valley]

Alright, first:
Did some more trail and error. I figured out the reason for the hang is indeed a wrong custom-image url, so the correct url for the vm image is ghcr.io/vanilla-os/vm:main (<registry>/<org>/<package>:<tag>). Massive skill issue on my end. Since I didn't know of the VM-check in installer, this info is rather useless now because it will change to that image for you, if you choose to install VM Tools.
Second:
I spent waaay too much time to install VMware Workstation... to specifically install Vanilla... 🤣
Anyways, I installed Workstation because I was using Hyper-V previously so I could eliminate the variable of the Hypervisor being the issue. I attached my .vmx config file if someone wants to recreate the VM, but I don't have anything to report other than 'It just worked'. I got through all installer steps, on both desktop and vm, and Wayland just works.
One thing was giving me troubles: booting the .iso without UEFI.
Anyways for graphics configs I...

• enabled Accelerate 3D graphics
• set VRAM to 512MB

Edit: I should add:

• I was using the latest Workstation version 17.6.3-24583834 (and the latest VM Compatibility version)
• The latest available Vanilla .iso (20250220)
• No VMware Tools, just open-vm-tools in case of vanilla-os/vm
• Debian 12 64-bit as my OS target
• Used UEFI boot (I tried BIOS for science and it just hangs everywhere)
• Was forced to run with side-channel migrations
• Didn't try the Xorg session
• Windows 11 as the Host OS