System freeze after update

[toxic maple]

I hadn’t installed any updates for about 3 weeks. Today, after updating and rebooting the system, I opened Trivalent. Then the display started flickering a bit on the browser's profile selection screen. Once I selected a profile and launched, the system froze, and I had to pull the plug and rollback.
What should I do now?

[bold dragon]

are you using nvidia?

[toxic maple]

No, AMD. silverblue-main-hardened

[bold dragon]

strange. is this only affecting trivalent?

[toxic maple]

I'm not sure... From what I've seen...

[bold dragon]

can you get the output of chrome://gpu in trivalent or does it crash too quickly

[toxic maple]

I can't

[bold dragon]

can you start trivalent from the terminal and gather the logs

[toxic maple]

I couldn't. Everything froze again.

[bold dragon]

oh the entire system is freezing? wow.

[toxic maple]

Yeah

[bold dragon]

can you boot into the working system and run rpm-ostree db diff?

[bold dragon]

okay, yeah there are a lot of potential culprits

[bold dragon]

one idea to try to narrow it down could be to try to gradually update

since you're going from three weeks ago to now

let's try going one week at a time

every day we push an image, and they are tagged with specific dates

so for example

you're currently on ostree-image-signed:docker://ghcr.io/secureblue/silverblue-main-hardened:latest

the broken deployment that is

but we can start by going to 2 weeks ago

then 1 week ago

and see which package breaks it

are you sure it was exactly three weeks ago?

in any case, you can pick a date like so:

rpm-ostree rebase ostree-image-signed:docker://ghcr.io/secureblue/silverblue-main-hardened:20250810

and keep iterating the date closer to today

until it breaks again

then rpm-ostree db diff again and we'll get a smaller set of potential culprits

@toxic maple does that make sense?

[toxic maple]

The last boot date, August 4

[bold dragon]

yeah so start with the 10th ish

if it works great

then keep increasing the date

until it breaks

the interval is up to you

this kind of thing is why atomic is great 😄

time machine

[toxic maple]

I guess only the last part will change for each rebase?

[bold dragon]

yup exactly

we keep 90 days of images

so you can teleport around 😛

[toxic maple]

😄

OK. Thank you.

[bold dragon]

https://tenor.com/view/doctor-who-tardis-time-vortex-peter-capaldi-gif-26069395

[toxic maple]

I'll try tomorrow

[toxic maple]

I did this, but the problem is still there

[bold dragon]

when did it start

[toxic maple]

when I opened Trivalent right after the rebase

[bold dragon]

did you try any earlier date tags

the goal was to find the exact image where the issue started

not just try one 😅

[toxic maple]

Right 😅

But rebasing is too slow, is it normal

[bold dragon]

it should be kinda slow

not very

should take a few minutes

[toxic maple]

15-20 minutes?

[bold dragon]

not unless your internet is very slow

[toxic maple]

Not very slow actually

[bold dragon]

hm

is this an unplugged laptop?

[toxic maple]

What is unplugged

[bold dragon]

as in the power cable is not plugged in

which would downcycle your cpu

[toxic maple]

No, it is plugged

[bold dragon]

no clue then

[drifting ingot]

At this rate, I'd just reinstall ig there is no clue

[toxic maple]

I went until the first day, 20250805, and still there

[bold dragon]

rpm-ostree db diff please

[bold dragon]

and rpm-ostree status please

[bold dragon]

reinstalling is generally pointless on atomic

unless there's evidence of critical system corruption

because /usr is binary identical

reinstalling doesn't change anything

[bold dragon]

very very interesting

so of these

  gnome-shell 48.3-1.fc42 -> 48.4-1.fc42
  gnome-shell-common 48.3-1.fc42 -> 48.4-1.fc42
  gnome-software 48.3-1.fc42 -> 48.4-1.fc42
  libinput 1.28.903-1.fc42 -> 1.29.0-1.fc42
  libsane-airscan 0.99.35-1.fc42 -> 0.99.36-1.fc42
  libsss_certmap 2.11.0-1.fc42 -> 2.11.1-1.fc42
  libsss_idmap 2.11.0-1.fc42 -> 2.11.1-1.fc42
  libsss_nss_idmap 2.11.0-1.fc42 -> 2.11.1-1.fc42
  libsss_sudo 2.11.0-1.fc42 -> 2.11.1-1.fc42
  python3-boto3 1.39.15-1.fc42 -> 1.39.16-1.fc42
  python3-botocore 1.39.15-1.fc42 -> 1.39.16-1.fc42
  sane-airscan 0.99.35-1.fc42 -> 0.99.36-1.fc42
  sssd-client 2.11.0-1.fc42 -> 2.11.1-1.fc42
  sssd-common 2.11.0-1.fc42 -> 2.11.1-1.fc42
  sssd-kcm 2.11.0-1.fc42 -> 2.11.1-1.fc42
  sssd-krb5-common 2.11.0-1.fc42 -> 2.11.1-1.fc42
  sssd-nfs-idmap 2.11.0-1.fc42 -> 2.11.1-1.fc42
  trivalent 138.0.7204.183-438460 -> 139.0.7258.66-438607
  trivalent-subresource-filter 138.0.7204.183-1754280 -> 139.0.7258.66-1754423
  vim-common 2:9.1.1552-1.fc42 -> 2:9.1.1591-1.fc42
  vim-data 2:9.1.1552-1.fc42 -> 2:9.1.1591-1.fc42
  vim-enhanced 2:9.1.1552-1.fc42 -> 2:9.1.1591-1.fc42
  vim-minimal 2:9.1.1552-1.fc42 -> 2:9.1.1591-1.fc42

it's either:

• something that changed in gnome 48.4

or

• something that changed in trivalent/upstream chromium in 139

it might be worth checking if you can reproduce this on kde @toxic maple

to narrow down whether this is a gnome bug

rpm-ostree rebase ostree-image-signed:docker://ghcr.io/secureblue/kinoite-main-hardened:latest

actually, given the nature of the bug

it seems likely that it's trivalent related

@empty folio any ideas? 🤔

we didn't really change anything on our end for 139

[toxic maple]

That's how I feel, too. Just intuition...

[bold dragon]

on this, can you pipe the output to a file

it should be there on reboot

run trivalent > trivalent.log

on the broken image

er wait

hm

nvm

[bold dragon]

you could stream it on discord and then i can screen record the terminal output? 😛

[toxic maple]

Um, I don't use it on laptop. It's banned in my country... I need to setup a VPN for that.

[bold dragon]

oh

stream on twitch? 😄

https://tenor.com/view/obama-cookies-gif-13897680

[toxic maple]

But if necessary, I can

[bold dragon]

there's gotta be a better way tbh

it crashes instantly?

like you can't even read the logs?

that show up in the console?

[toxic maple]

First, the display is flickering. Idk the word is right here?

[bold dragon]

actually

can you film it

like with your phone

[toxic maple]

OK

I'll be back in 1 hour

[empty folio]

This wasnt present in 138?

[toxic maple]

@bold dragon

[bold dragon]

according to the db diff

[toxic maple]

And yes, I can't read the logs when I start from the terminal either

[bold dragon]

oh wow

can you do this but from the terminal

[toxic maple]

This or just trivalent

[bold dragon]

just trivalent

[toxic maple]

I just caught this

[bold dragon]

hm

nothing of particular note

im still curious if you see this on kinoite

[toxic maple]

Idk if these are same

[toxic maple]

Should I try?

[bold dragon]

if you have time

[toxic maple]

Then I'll try tomorrow

[empty folio]

At it was present for all 139 builds?

[bold dragon]

yes

but not for 138

[empty folio]

... wtf?

We didnt change anything

[bold dragon]

yeah

the only other relevant packages that changed were gnome was upgraded to *.4

hence i recommended kinoite as a test

[empty folio]

That seems like a more likely culprit

Maybe kernel updates?

Or graphics drivers?

[bold dragon]

@empty folio

@toxic maple you're absolutely certain you made no karg changes too, right?

[empty folio]

Was it tested without kargs btw? Im not caught up

[bold dragon]

rpm-ostree cleanup -bprm may be in order, if not @toxic maple

[bold dragon]

they have a machine that was three weeks out of date

upgraded

and it broke

then we isolated it to the 0805 build

which happened to be the build the day after the current working build

[empty folio]

Odd

I dont think its trivalent, it shouldnt be freezing the system. Unless this is just a mutter issue

[bold dragon]

yeah idk

[toxic maple]

No

[toxic maple]

I tried to boot Kinoite, but after entering my disk password, the screen just shows a blinking underscore.
@bold dragon

[bold dragon]

can you drop to a tty

(ctrl + alt + fN)

and then journalctl -b0

[toxic maple]

I did

Now?

[bold dragon]

journalctl -b0 | fpaste

then send the link

[toxic maple]

https://paste.centos.org/view/328fce68

[bold dragon]

wat

this says that the boot was up for hours

oh wait nvm i see

Ağu 28 23:02:13 secureblue systemd[1]: Mounting sysroot.mount - /sysroot...
Ağu 28 23:02:13 secureblue kernel: BTRFS: device label fedora_fedora devid 1 transid 25825 /dev/mapper/luks-37d6f1ad-84de-4cc2-9fbf-dea192425c2f (252:0) scanned by mount (650)
Ağu 28 23:02:13 secureblue kernel: BTRFS info (device dm-0): first mount of filesystem 06446260-f457-4a58-847f-1d4b673aad3b
Ağu 28 23:02:13 secureblue kernel: BTRFS info (device dm-0): using crc32c (crc32c-x86_64) checksum algorithm
Ağu 28 23:02:13 secureblue kernel: BTRFS info (device dm-0): using free-space-tree
Ağu 28 23:02:13 secureblue systemd[1]: Mounted sysroot.mount - /sysroot.
Ağu 28 23:02:13 secureblue systemd[1]: Starting ostree-prepare-root.service - OSTree Prepare OS/...
Ağu 28 23:02:13 secureblue ostree-prepare-root[667]: Resolved OSTree target to: /sysroot/ostree/deploy/fedora/deploy/c5bb6cf79392514fc97a1424108169e2d0bf73262fb3efe48898c3fa77a7473c.0
Ağu 28 23:02:13 secureblue ostree-prepare-root[667]: sysroot.readonly configuration value: 1 (fs writable: 1)
Ağu 28 23:02:13 secureblue kernel: netfs: FS-Cache loaded
Ağu 28 23:02:13 secureblue kernel: erofs (device erofs): mounted with root inode @ nid 36.
Ağu 28 23:02:13 secureblue ostree-prepare-root[667]: composefs: mounted successfully (verity=false)
Ağu 28 23:02:13 secureblue systemd[1]: run-ostree-.private-cfsroot\x2dlower.mount: Deactivated successfully.
Ağu 28 23:02:13 secureblue systemd[1]: sysroot-ostree-deploy-fedora-var.mount: Deactivated successfully.
Ağu 28 23:02:13 secureblue systemd[1]: sysroot.tmp-etc.mount: Deactivated successfully.

it's loading and then just shutting down

wtf

this is very odd

[toxic maple]

¯_(ツ)_/¯

I have no idea

[toxic maple]

Any updates?

[toxic maple]

I'm stuck here

@bold dragon

[toxic maple]

I just tried sericea, but it still crashes. Probably not related to gnome...

[bold dragon]

strange..

[toxic maple]

The laptop became useless...

[bold dragon]

what do you mean

i thought this only affected trivalent

[bold dragon]

which backend do you have selected in the appearance section

[toxic maple]

I mean, without trivalent, it's pointless for me

[bold dragon]

oh i see

@toxic maple yeah try changing the backend if you can

also have you already tried clearing out config and cache?

[toxic maple]

Actually, I even reinstalled :/

[bold dragon]

oh wow.

did you try removing hardening kargs?

it could be some weird amd iommu bug

[toxic maple]

Yeah

[bold dragon]

also you were unable to repro on chrome right?

layered chrome

[toxic maple]

What is it? How?

[bold dragon]

settings > appearance > "use classic" vs "use gtk"

[toxic maple]

I'm on gnome now

[toxic maple]

If I understand correctly, no, haven't tried it

[bold dragon]

ah please do layer chrome and test

that will help narrow it down

[toxic maple]

I downloaded the rpm file and installed it locally, then reboot, but it doesn't launch

Did I do something wrong

[bold dragon]

you need ujust toggle-unconfined-domain-userns-creation

and then chrome --ozone-platform=wayland

[toxic maple]

And it crashes too..

[bold dragon]

with what error

oh the same issue you mean

[toxic maple]

Yeah, exactly the same

[bold dragon]

damn. okay

what if you ujust toggle-xwayland

then restart and check both trivalent and chrome

[toxic maple]

Nothing changed...

[bold dragon]

damn

even if you trivalent --ozone-platform=x11 ?

[toxic maple]

They work with this

So, the problem is about Wayland?

[bold dragon]

apparently so

something with wayland and your gpu setup

but if you're only using one app in x11, should be fine

i would recommend using that persistently and removing chrome

and you should be good to go 🙂

[toxic maple]

What I don't understand is why this problem didn't exist before

[bold dragon]

given that it's affecting chrome as well, it must be an upstream issue

it might be worth checking https://issues.chromium.org/ for similar issues relating to your hardware

[toxic maple]

The GPU is too old

[bold dragon]

even still though, they should support it

[toxic maple]

Will xwayland always be active?

And how to make it default for trivalent when opening from GUI?

[bold dragon]

run0 sed -i '$ a CHROMIUM_FLAGS+=" --ozone-platform=x11"' /etc/trivalent/trivalent.conf

@toxic maple did it work

[toxic maple]

Yes