🎉 Access to Claude-2/Llama-70b/etc. [community-supported api]

[main edge]

Hi,

I noticed a few people trying to access new / run larger LLMs than their computers could support:

• claude-2
• Llama-2-70b
• CodeLlama
  etc.

So I hosted a proxy API to give people access to these models (~20 hosted models) through my keys.

Tutorial for how to add to openinterpreter: https://docs.litellm.ai/docs/proxy_api

Note: You will need to clone and modify the Github repo, until this PR is merged.

git clone https://github.com/krrishdholakia/open-interpreter-litellm-fork

To run it do:

poetry build 

# call gpt-4 - always add 'litellm_proxy/' in front of the model name
poetry run interpreter --model litellm_proxy/gpt-4

# call llama-70b - always add 'litellm_proxy/' in front of the model name
poetry run interpreter --model litellm_proxy/togethercomputer/llama-2-70b-chat

# call claude-2 - always add 'litellm_proxy/' in front of the model name
poetry run interpreter --model litellm_proxy/claude-2

🎉 Access to Claude-2/Llama-70b/etc. [community-supported api]

[long abyss]

Will this work with together api?? @main edge

[main edge]

Yup @long abyss - we spun up a proxy server that calls anthropic, together ai, hf inference endpoints, openai, etc. using litellm (https://github.com/BerriAI/liteLLM-proxy)

[snow pebble]

I get an error trying gpt4

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/kyle/src/github/litellm/open-interpreter/interpreter/interpreter.py", line 137, in cli
    cli(self)
  File "/home/kyle/src/github/litellm/open-interpreter/interpreter/cli.py", line 210, in cli
    interpreter.chat()
  File "/home/kyle/src/github/litellm/open-interpreter/interpreter/interpreter.py", line 300, in chat
    self.respond()
  File "/home/kyle/src/github/litellm/open-interpreter/interpreter/interpreter.py", line 462, in respond
    info = self.get_info_for_system_message()
  File "/home/kyle/src/github/litellm/open-interpreter/interpreter/interpreter.py", line 161, in get_info_for_system_message
    message_for_semantic_search = {"role": message["role"]}
KeyError: 'role'

It lists the steps, the cursor goes off to the right then blank prompt

My bad, there are 2 occurences, so I picked the wrong one

spoke too soon. The second prompt blew up the same way, trying another model

[main edge]

what version of litellm is this?

i just pushed a fix

try 0.1.652

[snow pebble]

Successfully installed litellm-0.1.652

Excellent, now to do the same thing I do every night, try to take over the world!

Thank you again so much

[snow pebble]

Tried claude-2, same thing. gpt-4 works fine. Trying others

> write a program to list all the python functions and their docstrings in a directory
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/kyle/src/github/litellm/open-interpreter/interpreter/interpreter.py", line 139, in cli
    cli(self)
  File "/home/kyle/src/github/litellm/open-interpreter/interpreter/cli.py", line 210, in cli
    interpreter.chat()
  File "/home/kyle/src/github/litellm/open-interpreter/interpreter/interpreter.py", line 306, in chat
    self.respond()
  File "/home/kyle/src/github/litellm/open-interpreter/interpreter/interpreter.py", line 468, in respond
    info = self.get_info_for_system_message()
  File "/home/kyle/src/github/litellm/open-interpreter/interpreter/interpreter.py", line 163, in get_info_for_system_message
    message_for_semantic_search = {"role": message["role"]}
KeyError: 'role'

[main edge]

I think i found the bug, we have some streaming issues we need to fix for non-openai models. I'll have a patch out shortly. Thanks for helping here @snow pebble

[bitter shore]

Dude, you're too nice.

@main edge

[snow pebble]

Ya I was getting that on a lot of models. Was tired, Morpheus was tapping his foot at me and giving me "the look"

No, thank you

comma, a much underappreciated punctiation mark

[snow pebble]

if you do this:

self.model = os.environ['HACKMOD']

It will put an env hook in, then I'm using this to print a command. For some reason I get a very odd include error when trying to shell this, so I just print then cut paste, or you could redirect to .sh file to build up a command for each thing. This will print an export command then a launch. thusly

export HACKMOD='gpt-3.5-turbo'; poetry run interpreter

I got tired of typeing.

#!env python3

import curses

def select_model(stdscr, models):
    curses.curs_set(0)
    current_row = 0

    while 1:
       stdscr.clear()
       h, w = stdscr.getmaxyx()

       for idx, row in enumerate(models):
           x = w//2 - len(row)//2
           y = h//2 - len(models)//2 + idx
           if idx == current_row:
               stdscr.attron(curses.color_pair(1))
               stdscr.addstr(y, x, row)
               stdscr.attroff(curses.color_pair(1))
           else:
               stdscr.addstr(y, x, row)

       key = stdscr.getch()

       if key == curses.KEY_UP and current_row > 0:
           current_row -= 1
       elif key == curses.KEY_DOWN and current_row < len(models)-1:
           current_row += 1
       elif key == curses.KEY_ENTER or key in [10, 13]:
           return models[current_row]

       stdscr.refresh()

def main():
    models = [
        "gpt-4",
        "gpt-3.5-turbo",
        "gpt-3.5-turbo-16k",
        "togethercomputer/llama-2-70b-chat",
        "togethercomputer/llama-2-70b",
        "togethercomputer/LLaMA-2-7B-32K",
        "togethercomputer/Llama-2-7B-32K-Instruct",
        "togethercomputer/llama-2-7b",
        "togethercomputer/CodeLlama-34b",
        "WizardLM/WizardCoder-Python-34B-V1.0",
        "NousResearch/Nous-Hermes-Llama2-13b",
        "togethercomputer/falcon-40b-instruct",
        "togethercomputer/falcon-7b-instruct",
        "j2-ultra",
        "j2-mid",
        "j2-light",
        "dolpin",
        "chatdolphin",
        "claude-2",
        "claude-instant-v1",
    ]

    selected_model = curses.wrapper(select_model, models)
    print(f"export HACKMOD='{selected_model}'; poetry run interpreter")

if __name__ == "__main__":
    main()

I suppose you could | /bin/sh haven't tried

oops need openai/

print(f"export HACKMOD='openai/{selected_model}'; poetry run interpreter")

It also is a list since, well that's a few items

Successfully installed litellm-0.1.674

[long abyss]

@snow pebble Did you get it to work with the together api?

[snow pebble]

Now it just errors out when I try to launch.

poetry run interpreter
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/kyle/anaconda3/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/kyle/src/github/litellm/open-interpreter/interpreter/__init__.py", line 1, in <module>
    from .interpreter import Interpreter

--snip--

  File "/home/kyle/src/github/litellm/open-interpreter/.venv/lib/python3.10/site-packages/urllib3/exceptions.py", line 3, in <module>
    import socket
  File "/home/kyle/anaconda3/lib/python3.10/socket.py", line 54, in <module>
    import os, sys, io, selectors
  File "/home/kyle/anaconda3/lib/python3.10/selectors.py", line 291, in <module>
    class SelectSelector(_BaseSelectorImpl):
  File "/home/kyle/anaconda3/lib/python3.10/selectors.py", line 318, in SelectSelector
    _select = select.select
AttributeError: module 'select' has no attribute 'select'

I'll repull everything, might have picked up a gremlin hitchhiker

[snow pebble]

gh repo clone KillianLucas/open-interpreter
cd open-interpreter
gh pr checkout 288

poetry run interpreter
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/kyle/anaconda3/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/kyle/src/github/litellm2/open-interpreter/interpreter/__init__.py", line 1, in <module>
    from .interpreter import Interpreter
  File "/home/kyle/src/github/litellm2/open-interpreter/interpreter/interpreter.py", line 387
    if slitellm.api_base = "https://proxy.litellm.ai"
       ^^^^^^^^^^^^^^^^^
SyntaxError: cannot assign to attribute here. Maybe you meant '==' instead of '='?

  385       litellm.api_key = self.api_key
  386     else:
= 387       if slitellm.api_base = "https://proxy.litellm.ai"
= 388 elf.api_key == None:
  389         if 'OPENAI_API_KEY' in os.environ:
  390           self.api_key = os.environ['OPENAI_API_KEY']

This IS still the way , yes?

Some helper aliases:

mkvenv='python3 -m venv .venv; echo "source `pwd`/.venv/bin/activate" > .env; echo deactivate > .env.leave; source .venv/bin/activate; pip install --upgrade pip'
mcoy='pip install --upgrade pip; pip install poetry; poetry install'
oi='poetry run interpreter'
oil='poetry run interpreter --local'

[slow ginkgo]

I made this GIF with the help of Open Interpreter and your API to say thank you. I heavily appreciate the fact that you're hosting these models for free, it lets me experiment more freely without having to go into more debt than I'm already in XP So again, thank you so much @main edge

[main edge]

omg thanks @slow ginkgo

[slow ginkgo]

I will say there's a slight problem though, no matter what model I put in that line, whether it's "openai/gpt-4" or "openai/claude-2", it seems to use gpt-4 no matter what. Not that I'm complaining too much, it could just be user error

[main edge]

that's odd

[slow ginkgo]

No problem, again, you are awesome for doing this.

[main edge]

thanks for letting me know

i got rate limited on openai today morning and was wondering why

how are you setting it?

[snow pebble]

Is this still pr 288 or whatever number for this?

[slow ginkgo]

I'm setting it on the line that's given in the litellm docs, let me check the actual line number rq

[slow ginkgo]

Oh dear, I hope I didn't do that, I have been experimenting a lot. Not like crazy amounts but.. still XD

Line 342

https://github.com/KillianLucas/open-interpreter/blob/f803d0d7a545edabd541943145a2a60beaf604e4/interpreter/interpreter.py#L342C10-L342C10

[snow pebble]

If this is going to be held a bit, I was already looking at how to issue a patch so people could pull the pr and patch

[slow ginkgo]

Also, if I ever pull myself together financially eventually, I would love to donate to you to help cover costs of the API calls. Probably won't be for quite some time, but in the future I def owe ya one XD

[slow ginkgo]

It's fairly simple to patch it just by copy pasting in some code, but I can assume it is hard for some people to know exactly how to navigate the code and figure out where to put everything (by everything I mean the one block of code, and the line you need to change XD). So if you could come up with a simpler process for patching that would probably help a lot of people.

[snow pebble]

It took me a couple tries. I also patched in a hook for changing models.

[slow ginkgo]

Oh I see. Also nice.

I myself only know the bare minimum of Python, if that. I guess I got it first try cuz I have pretty decent intuition with shtuff like this.

[snow pebble]

I'm a pro. It's difficult for me to tell what people consider hard

I'll see about repulling and making it a little easier

[slow ginkgo]

I mean, everyone has different levels of knowledge and intuition which can make certain things seem easier or harder. It's all based on the individual. So yeah.

[snow pebble]

Actually, is there any reason I can't just fork and patch my copy and people can just pull that maybe

We have the Treknology

[slow ginkgo]

That would probably work for now, yeah.

[snow pebble]

I'm overthinking, you need 1 file

I'll prolly have it soonish.

[snow pebble]

openai.error.APIError: Invalid response object from API: 'Internal Server Error' (HTTP response code was 500)

Looks like there's an issue with the endpoint

[main edge]

Working on it - give me till EOD, just fixing cross-provider streaming

[snow pebble]

Ok. I'll post up a new interpreter.py and the ncurses helper. Gives me an excuse to make a bootcupboard repo

[main edge]

@slow ginkgo @snow pebble just pushed updates to the PR https://github.com/KillianLucas/open-interpreter/pull/288

it should simplify calling

example call:

poetry run interpreter --model litellm_proxy/togethercomputer/llama-2-70b-chat  

[snow pebble]

Thx

[main edge]

let me know if it works / if you face any issues

i'm also going to make the proxy server code public, so it's easy for people to spin up their own llm proxy's

[snow pebble]

I'll give it a spin now, between matches

[main edge]

sg

[snow pebble]

gh repo clone KillianLucas/open-interpreter
cd open-interpreter
gh pr checkout 288
mkvenv
mcoy

[slow ginkgo]

Ooooh, that'll be good if I want to run a local model on my PC and then connect from my Macbook Pro.

My Macbook Pro was originally bought for machine learning, but then I got the 8GB RAM model and didn't realize it... so.. that sucks lmao

[snow pebble]

It askes for a key

expected?

litellm-api-key

This looks wrong, somehow:

  },
    'litellm_params': {
        'return_async': False,
        'api_key': None,
        'force_timeout': 600,
        'logger_fn': <function Interpreter.respond.<locals>.logger_fn at 0x7f3f6908b1c0>,
        'verbose': False,
        'custom_llm_provider': None,
        'api_base': None,
        'litellm_call_id': '985a275b-136a-45b8-a08b-72bb72dc7252',
        'model_alias_map': {},
        'completion_call_id': None,
        'stream_response': {}
    },
    'input': [
        {
            'role': 'system',
            'content': "You are Open Interpreter, a world-class programmer that can complete any goal by executing code.\nFirst, write a plan. **Always recap the plan between
each code block** (you have extreme short-term memory loss, so you need to recap the plan between each message block to retain it).\nWhen you send a message containing code
to run_code, it will be executed **on the user's machine**. The user has given you **full and complete permission** to execute any code necessary to complete the task. You
have full access to control their computer to help them. Code entered into run_code will be executed **in the users local environment**.\nNever use (!) when running
commands.\nOnly use the function you have been provided with, run_code.\nIf you want to send data between programming languages, save the data to a txt or json.\nYou can
access the interne

[main edge]

what's the error you're seeing? @snow pebble

that's just the log output

just comment this out

[snow pebble]

oh, ok

openai.error.APIError: Invalid response object from API: 'Internal Server Error' (HTTP response code was 500)

[snow pebble]

Voodoo from friendly witchdoctor to make use litellm:

sed -i '/None/{s//"litellm-api-key"/;:p;n;bp}' interpreter/interpreter.py
sed -i '/None/{s//"https:\/\/proxy\.litellm\.ai"/;:p;n;bp}' interpreter/interpreter.py

[long abyss]

@main edge Getting this error:

  File "/Users/.../PycharmProjects/Open-Interpreter_OtherModels/open-interpreter/interpreter/interpreter.py", line 204, in chat
    self.llama_instance = get_hf_llm(self.model, self.debug_mode, self.context_window)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/.../PycharmProjects/Open-Interpreter_OtherModels/open-interpreter/interpreter/get_hf_llm.py", line 41, in get_hf_llm
    raw_models = list_gguf_files(repo_id)
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/.../PycharmProjects/Open-Interpreter_OtherModels/open-interpreter/interpreter/get_hf_llm.py", line 266, in list_gguf_files
    gguf_files = [file for file in files_info if "gguf" in file.rfilename]
                                   ^^^^^^^^^^
UnboundLocalError: cannot access local variable 'files_info' where it is not associated with a value

▌ Failed to install litellm_proxy/togethercomputer/llama-2-70b-chat. ```

[ionic flower]

In this required on the main branch or 288? For some reason no model is working on that branch.

[long abyss]

Fixed the problem that I had earlier but I am still encountering other problems...

[long abyss]

Looking at the problem more closely it seems that litellm.completion function is only sending the first delta.

Anyone of the litellm people know why this is??

[main edge]

yep i saw this - had to fix by changing OI code to stop thinking anything with a / was a local model

git clone https://github.com/krrishdholakia/open-interpreter-litellm-fork

To run it do:

poetry build 

# call gpt-4 - always add 'litellm_proxy/' in front of the model name
poetry run interpreter --model litellm_proxy/gpt-4

# call llama-70b - always add 'litellm_proxy/' in front of the model name
poetry run interpreter --model litellm_proxy/togethercomputer/llama-2-70b-chat

# call claude-2 - always add 'litellm_proxy/' in front of the model name
poetry run interpreter --model litellm_proxy/claude-2

let me know if this works @ionic flower @long abyss

[snow pebble]

I don't think you need bujild, that's for pip iirc

I don't do build and it uses src

@main edge need me to run anything?

[main edge]

try the fork and let me know if it works for you

[main edge]

ah fair - i use poetry build to ensure it's running my local version

[snow pebble]

K samew repo I'm assuming

Killing feew pixels first

[snow pebble]

poetry run interpreter --model litellm_proxy/gpt-4

huggingface_hub.utils._errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-6507394d-1743350c3ec927c95d311ecc;232b9392-6c03-437d-b18b-d86a0f643c13)

Repository Not Found for url: https://huggingface.co/api/models/litellm_proxy/gpt-4/tree/main?recursive=True&expand=False.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid username or password.

▌ Failed to install litellm_proxy/gpt-4.

poetry run interpreter --model litellm_proxy/togethercomputer/llama-2-70b-chat

> write a program to list all the python functions and their docstrings in a directory
> write a program to list all the python functions and their docstrings in a directory
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/kyle/src/github/testclaude/open-interpreter/interpreter/interpreter.py", line 134, in cli
    cli(self)
  File "/home/kyle/src/github/testclaude/open-interpreter/interpreter/cli.py", line 210, in cli
    interpreter.chat()
  File "/home/kyle/src/github/testclaude/open-interpreter/interpreter/interpreter.py", line 297, in chat
    self.respond()
  File "/home/kyle/src/github/testclaude/open-interpreter/interpreter/interpreter.py", line 459, in respond
    info = self.get_info_for_system_message()
  File "/home/kyle/src/github/testclaude/open-interpreter/interpreter/interpreter.py", line 158, in get_info_for_system_message
    message_for_semantic_search = {"role": message["role"]}
KeyError: 'role'

huggingface_hub.utils._errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-65073a04-0ece65d2571390b60bd88c68;e39a2ba0-abb6-4592-b2a9-95e00271bed6)

Repository Not Found for url: https://huggingface.co/api/models/litellm_proxy/claude-2/tree/main?recursive=True&expand=False.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid username or password.

▌ Failed to install litellm_proxy/claude-2.

[long abyss]

@main edge It works now. But the markdowns are not correctly implemented. Like it runs the code with the markdowns:

> Create python code that prints out “Hello World!”
                        
  .\n\n### 2. Create a new file called hello.py\n\n### 3. Open the file and type the following code:\n\n

  print(\"Hello World!\")
                                                                
  Would you like to run this code? (y/n)

Cause when I run this on together api it doesn't do this error for such a simple task

Then when I run the code it has an error running the code because the code is wrong then the terminal outputs this:

  have added the code.\n\nComment: The code you added is not the code that is causing the error.\n\nComment: @mkrieger1 I have added the code that is causing the error.\n\nComment: The code you added is not the code that is
  causing the error.\n\nComment: @mkrieger1 I have added the code that is causing the error.\n\nComment: The code you added is not the code that is causing the error.\n\nComment: @mkrieger1 I have added the code that is causing
  the error.\n\nComment: The code you added is not the code that is causing the error.\n\nComment: @mkrieger1 I have added the code that is causing the error.\n\nComment: The code you added is not the code that is causing the
  error.\n\nComment: @mkrieger1 I have added the code that is causing the error.\n\nComment: The code you added is not the code that is causing the error.\n\nComment: @mkrieger1 I have added the code that is causing the
  error.\n\nComment: The code you added is not the code that is causing the error.\n\nComment: @mkrieger1 I have added the code that is causing the error.\n\nComment: The code you added is not the code that is causing the
  error.\n\nComment: @mkrieger1 I have added the code that is causing the error.\n\nComment: The code you added is not the code that is causing the error.\n\nComment: @mkrieger1 I have added the code that is causing the
  error.\n\nComment: The code you added is not the code that is causing the error.\n\nComment: ```

  error.\n\nComment: @mkrieger1 I have added the code that is causing the error.\n\nComment: The code you added is not the code that is causing the error.\n\nComment: @mkrieger1 I have added the code that is causing the             
  error.\n\nComment: The code you added is not the code that is causing the error.\n\nComment: @mkrieger1 I have added the code that is causing the error.\n\nComment: The code you added is not the code that is causing the
  error.\n\nComment: @mkrieger1 I have added the code that is causing the error.\n\nComment: The code you added is not the code that is causing the error.\n\nComment: @mkrieger1 I have added the code that is causing the
  error.\n\nComment: The code you added is not the code that is causing the error.\n\nComment: @mkrieger1 I have added the code that is causing the error.\n\nComment: The code you added is not the code that is causing the
  error.\n\nComment: @mkrieger1 I have added the code that is causing the error.\n\nComment: The code you added is not the code that is causing the error.\n\nComment: @mkrieger1 I have added the code that is causing the```

When I continue the convo it only outputs this.

[snow pebble]

Ok, seriously. All this time I thought the emoji on this channel title was a piece of pizza. I just now looked closely. I thought, oh pizza! That will attract attention. My mind works in mysterious ways.

[snow pebble]

This was working not that long ago
openai.error.APIError: Invalid response object from API: 'Internal Server Error' (HTTP response code was 500)

maybe catch the 500? Crashing looks so unprofessional

Not trying to be that way just sayin

refactor coming, note to self

[snow pebble]

In fact, now that i can depend on failure, I'll just catch it myself

so now it does

ValueError: Unable to map your input to a model. Check your input - {'model': 'custom_openai/claude-2', 'messages': [{'role': 'user', 'content': 'write a program to print hi

with claude-2
Perhaps I'm doing something wrong. Norman coordinate

It was like that when I got here. Nobody saw me do it.

odd, it should be caught

Magic, it's a bitch Godess

Magic try catch jiggery pokery because I found the 3 calls and they're caught

interpreter/interpreter.py
610:              response = litellm.completion(
620:                response = litellm.completion(
630:                response = litellm.completion(

[long abyss]

@main edge any updates??

[main edge]

providing this now - https://docs.litellm.ai/docs/proxy_server

[snow pebble]

Thanks @main edge

Excellent! This should allow me to use pyautogen easily. 🔥