keen tulip
I tried git pulling and then a complete fresh install and I keep getting these errors for exllamav2 and exllamav2hf.
I've followed the installation steps from the repo page for linux instead of using the one click installer.
I notice my cuda is different, I'm currently Build cuda_11.5.r11.5/compiler.30672275_0
Did I miss a step?
novel onyx
Not sure, but I would guess that it is an incompatible version of gcc.
Does this do any better:
python -m pip install git+https://github.com/jllllll/exllamav2 --force-reinstall --no-deps -v
It's having trouble with the wheel.
novel onyx
The error is likely due to the code not being compatible with CUDA 11.5 as it is the same error as before.
Additionally, looking at it closer, the error is occurring while running NVCC.
This will install CUDA 11.7 into the env:
conda install cuda -c nvidia/label/cuda-11.7.0
You will need to set CUDA_HOME to the env directory for it to be used.
You may prefer installing 11.7 globally or using the one-click-installer to automate it.
The webui's default packages are built for CUDA 11.7, so anything older is unlikely to work very well.
At the very least, exllamav2 isn't going to work.
keen tulip
That appears to have been the major issue! Thanks for the assist!
novel onyx
You need to set your CUDA_HOME environment variable to the location that you have the CUDA Toolkit installed.
If you install the Toolkit as described above, then you can launch the webui with this and have the variable set correctly:
CUDA_HOME=$CONDA_PREFIX python server.py