#Issues using OpenAi api exstension

7 messages · Page 1 of 1 (latest)

hallow hatch
#

I'm trying to use the OpenAI API extension for various projects. I've tried many of the tested and verified to work apps but still can't seem to get them to work.

  • Aider (set bash export env variables still does not work)
  • shell_gpt (set bash export env variables still does not work)
  • gpt-shell (set bash export env variables still does not work)
  • Auto-GPT (using plugin still does not recognize the API key)
  • vscode-openai (request fails with status code 404, freezes when it requests an embeddings model)

This is what I have set as export in my bashrc file:

export OPENAI_API_KEY=sk-111111111111111111111111111111111111111111111111
export OPENAI_API_BASE=http://127.0.0.1:5001/v1

This is the output of this command:

python server.py --listen --extensions openai --api

Output:

[2023-07-16 08:22:20,390] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
bin /home/zack/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so
/home/zack/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/cextension.py:33: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
INFO:Loading the extension "openai"...

Failed to load embedding model: all-mpnet-base-v2
Starting OpenAI compatible api:
OPENAI_API_BASE=http://0.0.0.0:5001/v1
Starting streaming server at ws://0.0.0.0:5005/api/v1/stream
Starting API at http://0.0.0.0:5000/api
image.png
hallow hatch
#

I'm still having issues with this can somebody help me I've tried testing by calling from my terminal and I'm getting this response curl http://127.0.0.1:5001/v1/completions -H "Content-Type: application/json" -d '{ "model": "TheBloke_Llama-2-7B-Chat-GGML", "prompt": "A long time ago in a galaxy far, far away", "temperature": 0.7 }' {"id": "conv-1690445347713927424", "object": "text_completion", "created": 1690445347, "model": "TheBloke_stable-vicuna-13B-GPTQ", "choices": [{"index": 0, "finish_reason": "length", "text": "...\n\n### Human: What is the name of the planet where Luke", "logprobs": null}], "usage": {"prompt_tokens": 13, "completion_tokens": 17, "total_tokens": 30}}
So it is generating a repsonse however its defaulting to the model
This still however is not working in any of the before mentioned projects it doesnt seam to be detecting the models at all

compact lantern
#

it does not change the model this way

#

you need to change the model with the models API, or start the server with --model .....

#

did you fix the CUDA error?

#

I am away from my computer, but did you make sure the environment variables are in the app environment?

#

ie. confirm with : env | grep OPENAI