#Performance degradation over other distros

1 messages · Page 1 of 1 (latest)

weary rivet
#

hi folks. i really want to get to the bottom of this as i feel like it will be insightful to others too. moving from debian to gentoo and setting very reasonable performance flags (only -O2 -march=native) i suffered 10% performance degradation both for cpu and graphics (tested on several benchmarks), on the same machine of course
i'm trying to change something here and there to see if it would make a difference, but nothing seems to matter. what could be some root causes of this? i use this machine for intensive computing, and i would like every drop of perf i can get. every finding will be relayed to the repo maintainers of course

woeful yarrow
#

and o2 and o3 are real comparable, you can also try to tune stuff for your processor arch

#

sometimes optimizations can make code slower

weary rivet
#

i'm using o2, which is the default build flag for debian packages tho

woeful yarrow
#

this is a massive rabbit hole, but you would probably want to see if a debian like kernel will compare better or worse, then you know it's kernel related

#

and what benchmarks say they are slower?

weary rivet
woeful yarrow
#

try the gentoo binpkgs

#

yeah so many things affect system perf

#

it could be a difference in scheduler choice

#

so many things

weary rivet
woeful yarrow
weary rivet
marsh shell
#

I wonder if it's the lackof lto

woeful yarrow
weary rivet
#

i have the lto and pgo flags enabled globally

marsh shell
#

that's not how lto works

#

~wiki LTO

modest pollenBOT
weary rivet
#

does it not mean that every package that could be emerged with lto will be?

woeful yarrow
#

if the package has a use flag for that sure

#

but setting it in your compiler flags makes gcc or whatever always try to use that

weary rivet
#

better keep that off and use it as last resort

#

if i really can't find it

#

ok i'll try a kernel first and i'll keep you posted

#

am i the only one suffering from this problem?

marsh shell
#

debian has lto on by default is what we are trying to say

weary rivet
#

i didn't know that, thank

marsh shell
#

check the kernel fitst though

weary rivet
#

that could very well be it

woeful yarrow
#

lto will make a larger difference than o2 vs o3 i think

#

and in gcc o3 is mostly useful for finding bugs lol

weary rivet
#

but how does a packaged structure even work with lto i wonder

marsh shell
#

it could do, it might not

marsh shell
weary rivet
#

nvm off to try this

#

but i will have to rebuild everything in the tree right?

marsh shell
#

try the kernel first

weary rivet
#

alright, off to do that

marsh shell
#

but yes if you want to see if lto speeds up everything then you wil need to rebuild everything

weary rivet
#

unrelated, but emerge --depclean does not remove old kernel entries, nor the kernel cleaner utility (even after grub update)

marsh shell
#

not really recommended

weary rivet
#

i needed to do that eventually

#

i have a really barebone system

#

so it will take a couple hours tops

marsh shell
#

installkernel handles what you want I beilve

woeful yarrow
marsh shell
woeful yarrow
#

they don't work at all?

marsh shell
#

it won't be built with lto

woeful yarrow
#

if you make your own binpkg?

marsh shell
#

oh in that case it will but how does that help

woeful yarrow
#

if you're testing optimizations

#

makes it easier to a/b

marsh shell
#

I see where you are going

woeful yarrow
#

yeah, if only the gentoo premade packages had lto

marsh shell
#

good idea, just needed a better explanation 🙂

woeful yarrow
#

yes 🙂

marsh shell
#

we need more testers for lto to get that

weary rivet
#

i can provide

#

how unmanageable is lto on gentoo? i don't consider gentoo that high maintainance

#

but i've heard this flag can be

#

or was it graphite?

marsh shell
#

LTO is fine it's the GentooLTO overlay which is broken

woeful yarrow
#

deviating from standard gcc with default emerge settings is going to cause more trouble as a rule of thumb, but i've had no issues using lto

marsh shell
#

I alredy showed you how to do LTO correctly

weary rivet
#

yes, thanks

woeful yarrow
#

and i believe lto breaking like that wold be a compiler issue, not a gentoo issue in particular right

weary rivet
#

when first approaching this i was sucked into the lto overlay rabbit hole

#

i thought it was the only way

marsh shell
#

I wrote the docs for LTO so you are in safe hands

woeful yarrow
#

far more people around to report bugs for "standard" installs

#

but that being said, doing weird stuff is fine, and it's really appreciated if you report the bugs

weary rivet
#

alright, i'll set it to build overnight

marsh shell
#

do the kernel first though

weary rivet
#

in the meanwhile i can't find a built binary for zen-kernel

#

i really don't want to build it

marsh shell
#

I don't think one exists

weary rivet
#

there is for debian/ubuntu/arch

#

i think

woeful yarrow
#

you could steal it lol

#

a kernel is a kernel

marsh shell
#

you could steal their config

woeful yarrow
#

note that you amy have issues with that, just build it and make food or something lol

weary rivet
#

can u confirm this is the proper way of rebuilding? emerge --emptytree @installed -a

marsh shell
#

emerge -e @world

woeful yarrow
#

^ that is what i used

weary rivet
#

you
wouldn't
steal
a kernel config

woeful yarrow
#

also you prob want the "buildpkg" feature added so itll make packags for that stuff

weary rivet
woeful yarrow
#

so if you ever do rebuilds and go back, you don't have to recompile

weary rivet
#

alright 👍

woeful yarrow
#

this is how i have it added

weary rivet
#

but what does it do?

#

this?

woeful yarrow
#

yea 🙂

#

itll make a binpkg the first time i builds something

#

and that --usepkg thing will try to use that over building

weary rivet
#

thank yuo

woeful yarrow
#

yw, good luck with that stuff

weary rivet
#

it takes very little to build the full system tho, if i exclude llvm and firefox

woeful yarrow
#

im sure it's still not going to be a quick process

marsh shell
#

it won't 🙂

woeful yarrow
#

because it's also reinstalling everything, lots of bits of overhead associated with totally rebuilding tour system

marsh shell
#

maybe an over nighter though

weary rivet
#

it will be yea

woeful yarrow
#

remaking the kernel is trivial compared to that

weary rivet
#

i'll set it up before going to sleep 😴

woeful yarrow
#

solid plan

weary rivet
#

i know but so far i have used the distribution kernel

marsh shell
#

even my 5950x takes a while

weary rivet
#

with a savedconfig

weary rivet
marsh shell
#

I do a lot of compiling so was worth the money

woeful yarrow
#

yeah i started rebuiulding my world a while back on my 7950x

image.png
#

i set those compiler flags a bit back and never rebuilt everything

#

lockfile errors because it's over a squashed nfs share lol

#

i still wish i could get this thing to load my cpu harder on most builds

slate scaffold
#

so what conclusion did we come to ?

#

if you want my input I'd ask you for your CPU and compiler

weary rivet
#

build expectedly(?) failed overnight, with libxcb to be precise. if i disable lto for this package only can i resume the emerge with the new flags?

slate scaffold
#

can you post the build log

#

also what did you end up setting your COMMON_FLAGS to

weary rivet
slate scaffold
#

curious to see how much of a difference just adding LTO makes but GCC O3 is a bit too unstable that immolo would recommend it

weary rivet
weary rivet
#

well not massively, but one of them has function unroll

slate scaffold
#

I want benchmarks

#

because LLVM O3 is not only safer but also faster

#

than GCC

weary rivet
#

cause i'm re-emerging everything and there is no way to keep track

slate scaffold
#

lmao

weary rivet
#

number of first place finishes is meaningless, but mike does a weighted average which is abit more telling

slate scaffold
#

you mean this?

#

My honest opinion is that people are too scared of O3 it can be enabled for a lot of packages

#

but

#

O2 is fine

#

anyway hope you see some nice benefits from enabling LTO

#

do you have the LTO and PGO use flags set globally?

weary rivet
#

yes

slate scaffold
#

PGO made an insane difference in python for me

#

I know it's a bad benchmark but

#

dependency resolution for an empty auDU world took 11 seconds for me

#

after enabling LTO and PGO it was 7-8

#

and those were consistent it was always the same amount of seconds

weary rivet
#

yea what's up with portage being made in python 😅

slate scaffold
#

it's easy to write in, someone decided it months ago

#

for me, as someone who develops a distro based on Gentoo

weary rivet
#

anyhow, this line tells me that:
The complete build log is located at '/var/tmp/portage/dev-perl/IO-Socket-SSL-2.83.0/temp/build.log'but if i go there it's just the same message

slate scaffold
#

it's actually very nice

#

because portage has an easy to use API

slate scaffold
weary rivet
#

yea it's a bit slow, but it does dependency resolution in a couple seconds instead of instantly, not really a problem

slate scaffold
#

someday it will take over as the package manager for gentoo

weary rivet
# 
sudo cat /var/tmp/portage/dev-perl/IO-Socket-SSL-2.83.0/temp/build.log
 * Package:    dev-perl/IO-Socket-SSL-2.83.0:0
 * Repository: gentoo
 * Maintainer: [email protected]
 * USE:        abi_x86_64 amd64 elibc_glibc kernel_linux
 * FEATURES:   network-sandbox preserve-libs sandbox userpriv usersandbox
 * Using ExtUtils::MakeMaker
 * perl Makefile.PL PREFIX=/usr INSTALLDIRS=vendor INSTALLMAN3DIR=none DESTDIR=/var/tmp/portage/dev-perl/IO-Socket-SSL-2.83.0/image
API-different OpenSSL versions compiled in (0x30000090) vs linked (0x30100030) at Makefile.PL line 68.
 * ERROR: dev-perl/IO-Socket-SSL-2.83.0::gentoo failed (configure phase):
 *   Unable to build!
 * 
 * Call stack:
 *     ebuild.sh, line  136:  Called src_configure
 *   environment, line 1762:  Called perl-module_src_configure
 *   environment, line 1262:  Called die
 * The specific snippet of code:
 *               perl Makefile.PL "$@" <<< "${pm_echovar}" || die "Unable to build!";
 * 
 * If you need support, post the output of `emerge --info '=dev-perl/IO-Socket-SSL-2.83.0::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=dev-perl/IO-Socket-SSL-2.83.0::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/dev-perl/IO-Socket-SSL-2.83.0/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/dev-perl/IO-Socket-SSL-2.83.0/temp/environment'.
 * Working directory: '/var/tmp/portage/dev-perl/IO-Socket-SSL-2.83.0/work/IO-Socket-SSL-2.083'
 * S: '/var/tmp/portage/dev-perl/IO-Socket-SSL-2.83.0/work/IO-Socket-SSL-2.083'```
slate scaffold
#

is there a config.log

#

configure phase error

weary rivet
#

in that same dir?

slate scaffold
#

it should link to one somewhere

#

it's very fast

#

and also python

#

ohhh

# 
API-different OpenSSL versions compiled in (0x30000090) vs linked (0x30100030) at Makefile.PL line 68.
weary rivet
#

dayum

slate scaffold
#

can you try to rebuild and update openssl and then rebuild perl

weary rivet
#

if i emerge anything i will lose my list

#

and will not be able to do emerge --resume

slate scaffold
#

I know there was some way to

#

reset the emerge backlog

#

it keeps your last two emerges

#

but I forgot

#

if you just emerge --resume it should skip that package for a little bit

weary rivet
#

i'll look for it

slate scaffold
#

maybe you could also just

#

emerge --resume --keep-going

weary rivet
#

i tried, it doesnt 😅

slate scaffold
#

and deal with all the stuff that failed to build later

#

if you just wanna resume and ignore this package

#

let's just do

#

emerge --resume --skipfirst

#

and deal with it later

#

you should rebuild openssl and dev-perl over time

#

keep in mind that any package under dev-perl/* will inherit the same LDFLAGS and CFLAGS as dev-lang/perl and also linked dependencies

#

it's a pain

weary rivet
#

thank you, skipfirst seems to have worked

#

i just have to remember which packages did that

slate scaffold
#

yes please keep a list

#

of packages that failed

#

I'll help you get them to build later

weary rivet
#

thank you

#

i'll keep you posted, this will take a long while

slate scaffold
#

took 8 hours on my pc

#

with 1000 packages

weary rivet
#

cpu?

slate scaffold
#

Ryzen 7 5800X3D

weary rivet
#

dayum

#

it's taking a bit less on my side, but i've yet to build gcc this time

#

500 packages to go still

slate scaffold
#

GCC with LTO and PGO takes an hour and 20 minutes

#

:)

weary rivet
#

i know 😪

weary rivet
#

scary stuff

weary rivet
#

full lto worked, but that wasn't it unfortunately

#

nor the kernel

slate scaffold
#

did it improve your benchmark scores at all

weary rivet
#

it skyrocketed firefox's

slate scaffold
#

very nice

weary rivet
#

irqbalance didn't yield the desired effect catsad

marsh shell
#

what did you benchmark?

weary rivet
#

cpu performance with passmark, graphical performance with unigine superposition benchmark and glxgears (for rough reference, it has no value) and xonotic benchmark

marsh shell
#

isn't passmark a binary?

weary rivet
#

yes

#

i tried sysproffing it but it only calls the kernel and libc

#

unigine benchmark calls radeonsi_dri.so and glibc

#

maybe something is wrong with libc?

#

how can i see what functions are called from my system? i'm not interested in what the binary does because i can't build it myself, i want to see how it interacts with my system

#

sysprof is nice but i'm having a hard time understanding the output

weary rivet
#

i used perf and i found the same results: libc6.so and radeonsi_dri.so

weary rivet
#

@marsh shell sorry to ping you, do you know of a fast way to check which compilation flags are used in the ubuntu kernel package?

woeful yarrow
#

im not sure if that is something you can expect to obtain, maybe they have info on that, but im not entirely sure there is a straightforward way to look at something and be like "it was compiled with these specific options"

#

id like to be wrong tho, sounds cool

weary rivet
#

it's a deb package, the compilation options are there

#

i really believe deb packages keep track of that

#

since we're here, is there a fast way to enable lto on kernel too?

woeful yarrow
weary rivet
#

yea, i know i could look there, but everytime i tried i got lost, as debian packaging is a little hard

#

my question was only whether there was a blatantly obvious option that fedora/ubuntu/debian were using

woeful yarrow
#

maybe debian/rules in the package metadata

marsh shell
#

I'm sure debian will just use standard kflags

cedar mauve
#

Honestly, do your benchmarks, gather data points, plot 'em, and make a Gentoo Forums post.

#

Pretty graphs will help

weary rivet
#

i will sometime thanks simd

real phoenix
#

but yes I'm very very interested

#

i thought everyone used lto btw ._.

#

also to make changes apply do emerge -e @world

#

annd uhhh yeah try the exact same kernel

cedar mauve
slate scaffold
#

I am a big LTO enjoyer

#

if I get around to it I will do some benchmark suite

#

with clang o2/o2 + lto/o3/o3 + lto

#

I guess now that I can build on a server I can also benchmark Os on my X220

#

and finally probe why I think more ppl should use it on lower spec hardware

weary rivet
#

some follow up to more findings in a bit