#SIMD perf between SSE and AVX

45 messages · Page 1 of 1 (latest)

rapid wren
#

you should vzeroupper or vzeroall before returning so the SIMD unit can avoid aliasing

#

if you use AVX & EVEX encoded instructions the SIMD unit becomes tainted because the upper half of the registers might have data in them.
This is fine, however if you then use a non AVX or EVEX instruction touching the vector registers the CPU is very confused and has to handle the upper and lower half indepdently which throttle down the SIMD core a lot, this never comes back unless you use vzeroall or vzeroupper.

#

using vzeroupper or vzeroall reset the upper halfs to something clean

#

the reason AVX and EVEX instructions do not pause a problem is because writting to the lower half of a register with thoses implicitely zero the upper half (or any smaller denomination for that matters) this avoid "false sharing" issues.

#

this does not happens with non AVX or EVEX instructions, so using vzeroupper always make it safe and reset to a known good state

#

just so you know on your particular CPU, vzeroall executes in up to 12 cycles while vzeroupper in up to 10 (from memory)
there is no performance cost to having garbage data in the lower half so might as well not bother with clearing the lower half

#

modern CPUs are marvelous machines they can rewrite your code and they are pipelined and speculative

#

the number of uops hint us that vzeroall is actually implement as a vzeroupper (to reset the state correctly) and a whole bunch of vpxor x0 x0 xN

#

this is a guess but that would make sense

#

I learned assembly by reverse enginering compiler output, I played a lot with godbolt and was learning how do the compiler implement various things

#

if you are stuck on memory by interacting with the stack, control flow or something else, the extra cost of vzeroall which is an arithmetic operation can be hidden, basically the CPU will do multiple things together at once

#

oh and last thing, in case this wasn't clear there is no cost to mixing AVX and EVEX instructions because they both have the same sementics.

#

so the very simple explaination of pipelining.
Let's assume I show you this code:

ADDQ $23, AX
NEGQ AX

how long do you think this takes to execute ? (*assuming no instruction merging which I don't belive happens here)

#

@modern spindle (ping, idk if you want some or not) 🙂

#

correct

#

what about:

ADDQ $23, AX
NEGQ BX

?

#

so negq and addq both have a latency of 1 cycle

#

here indeed the cpu is smart, it understand ax and bx are two different registers

#

the cpu have "ports", they do various things (read memory, write memory, do math, do fp math, ...)

#

there are many many maths ports on the same cpu

#

so addq and negq both gets dispatched to different ports in the same cycle

#

there are two intresting metrics:

latency

this is how long does the instruction takes to execute.
Here it is 1 cycle for both (I belive nop is the only instruction that can execute faster than 1 cycle)
That means you take some input, and someone else waits on this input, how long does the other instruction has to wait

throughput

This is the number of this instructions you can execute if they do not depends on each other

#

ports is an intel thing btw

#

amd cpus also have bigger than one throughput but it's not called ports

#

so the CPU will try very hard to avoid aliasing
for example take this:

MOVQ (AX), BX
ADDQ $42, BX
NEGQ BX
MOVQ BX, (AX)
MOVQ 8(AX), BX
ADDQ $42, BX
NEGQ BX
MOVQ BX, 8(AX)

which corresponds to this go code:

ax[0] = -(ax[0] + 42)
ax[1] = -(ax[1] + 42)

the cpu will do magic, so called register renaming

#

here it looks like we have to execute instructions one by one

#

but the CPU actually execute them two by two*

#

this is because even tho BX is reused twice, the second time it does not depends on the first result

#

it's a whole new bx value from scratch

#

so the CPU will "rename" the register to use an other one

#

this also means CPU aren't limited to the number of registers in the ISA, we might only have 16 general purpose registers, but the CPU internally has hundreds

#

so this code could execute like this:

MOVQ (AX), BX_0 | MOVQ 8(AX), BX_1
MOVQ (AX), BX_0 | MOVQ 8(AX), BX_1
MOVQ (AX), BX_0 | MOVQ 8(AX), BX_1
MOVQ (AX), BX_0 | MOVQ 8(AX), BX_1
; mov hits memory this might take many cycles, here it's cached in a not so slow cache and is here in 4 cycles
ADDQ $42, BX_0  | ADDQ $42, BX_1
NEGQ BX_0       | NEGQ BX_1
MOVQ BX_0, (AX)
; here the two memory writes has to wait on each other because my cpu has 2 memory loaders, but only 1 memory writer unit
MOVQ BX_1, 8(AX)
; the memory writes are faster because we assume my l1 isn't backed up, since the cache line is very fresh
#

here the ports are the fact multiple operations execute in parallel

#

however you can't modify and read from the register on two different ports at the same time (since the second port can't compute without the result from the first one)

#

so register renaming the optimisation where the CPU detects registers that use the same register in the assembly but don't actually depend on each other

#

The last intresting this is speculation, the CPU just made up shit if it is stuck (instead of waiting).
They can make up the value of registers, the result of conditionals, ... and are able to revert to a previous correct state if they get it wrong so called "pipeline flushes" reverting is a very expensive process tho
Not all CPUs do all of this, modern amd64 do that but smaller cheaper and more power efficient ones might not

#

if you want a deeper view you can try the perf tool on linux

#

this mesure performance counters in the cpu and can tell you various thing

#

you can build your benchmark with go test -c then run the bench binary with perf

#

for example an average go binary has a branch missprediction rate of ~0.5% to ~2% in my anecdotal experience.
That means out of the conditional jumps result that the CPU made up almost all of them are correct

#

amd cpus use multiple hardware implement perceptrons to do this, not really usefull info but cool

#

ok last thing I promise.
This is not a very good pattern (it's fine for testing)
In production you want your assembly functions to take a pointer to the slice's array and length as an argument and work on many elements in a loop.
This is because the overhead of calling assembly is non trivial.
Even more when you use the stack ABI like here.

#

so better do 1 call and loop inside assembly then many small calls

#

Usually I do padding in go or C so like if you have like 17 elements, pass 16 (2*8) to your avx2 impl, and do the reminder using go even if it's not vectorised.
Just because having more than 1 loop in assembly, when you have ifs and stuff it gets messy