I'm trying to get GPT-for-Llama installed in my virtual environment in Linux (archlinux to be precise), mostly to get monkeypatch running, but CUDA is of course mismatched with pyTorch. The instructions on how to downgrade gxx are only for conda and won't work with pip. Any ideas how to get this thing installed?!
#How to install GPTQ-for-Llama with venv on Linux?
urban trout
shut trellis
arch is a beayatch to get anything working. but if you have docker that might be a good idea,. if not podman. pull nvidia/cuda 11.8.0-devel-ubuntu22.04
nvidia/cuda 11.8.0-runtime-ubi8
nvidia/cuda 11.8.0-base-ubi8 then just a simple docker compose up --build with the proper models in the right place and the correct .env to match and depending on your gpu's you'll need the right arch_type generally 30 series and up is arch version 8.6
