powerinfer

[old musk]

Does anyone know about the PowerInfer project? It's based on llama.cpp. I want to improve it

[inland leaf]

PowerInfer looks promising, but they are not really being up front about everything. A quick glance would indicate;

• they have a fork of llama.cpp
• there are caveats eg "it is important to note that Powerinfer is great for certain hardware, but right not lacking for others."
• PI needs special, 'ReLU' models
• their 11x maybe be more marketing than truth (IMO)

llama.cpp is already considering merging some of the PI magic back into main llama.cpp - https://github.com/ggerganov/llama.cpp/pull/4543