#use tools without agent in langchain

1 messages · Page 1 of 1 (latest)

lapis isle
#

Is it possible to use tools without using agents in langchain? If yes, then how? The issue is, while using agent, it tends to hallucinate a lot, plus, it takes more time as the computations increases a lot because it needs to think and for that it again uses the LLM, so I’m trying to avoid using agent but i need to use the Tools in order for my chatbot to search net, do maths and other stuff. With agent, inference : 2-3 minutes + hallucinations, without agent, inference: 10-15 seconds + very well generated output. Some help?

https://python.langchain.com/docs/modules/chains/additional/multi_prompt_router
What about this? 🤔 is this close to what I’m trying to achieve? But how do i create chains with different prompts and custom tools like LLMMathChain, Search etc

Dynamically selecting from multiple prompts | 🦜️🔗 Langchain

This notebook demonstrates how to use the RouterChain paradigm to create a chain that dynamically selects the prompt to use for a given input. Specifically we show how to use the MultiPromptChain to create a question-answering chain that selects the prompt which is most relevant for a given question, and then answers the question using that prompt.

unborn cove
lapis isle
# unborn cove I think this is mostly for completely automated routing, but what you can do is ...

but how do i execute a function in a prompt? prompt is just the instruction for the model right, you can't just execute function in them...? What I want is: I want my model or whole chain to be smart enough to answer questions related math, general knowledge, physics, chemistry, current data, let's take for example the current data prompt, i have a search api which searches the internet for current happenings, how do i execute that func inside the prompt?

unborn cove
#

In reality, it's basically trying to find the best function to run based on query. Function calling from OpenAI works the same way. You provide function names and descriptions, and ask the model to output the name and arguments to use, chain that output to next prompt

lapis isle
#

i'm actually using llama2-7b model and it is quite impressive

#

so i wanted to enhance it on the topics that i mentioned above

unborn cove
#

Again, it's really the same concepts

#

In your prompts, you configure:

  1. List of functions with their name and descriptions, in List[Dict[name, description]]
  2. "Based on input, select the right function and extract arguments from input..."
  3. "{input}"
#

Output:
{"function_name": name, "arguments": {....}}

#

Parse it, so that you prompt the function:
functions[function_name](**args)

#

Or index, either way

lapis isle
#

great thanks, i'll try it and let u know here if i run into any issues

unborn cove
#

npnp

lapis isle
#

hey, which model would you recommend for router chain? (less powerful one)

#

I wonder if we could use a sentence-transformer model like all-MiniLM-L6-v2 for router chain...

#

yes we can do that, its the last section in the router chain doc... nice

lapis isle
#

Hi @unborn cove , I tried your approach and it does seem to select the correct chain, however, I created my custom chain that could write python code and execute it using subprocess library in python by customizing the chain's _call method, and then finally gave it prompt template like this:

math_template = """
You can do basic math, and your memorization abilities are impressive, but you can't do any complex calculations \
that a human could not do in their head. You also have an annoying tendency to just make up highly specific, but wrong, answers.
So we hooked you up to a Python 3 kernel, and now you can execute code. If anyone gives you a math problem, \
just use this format and we'll take care of the rest:

Question: ${{Question with math problem}}
`python
${{Python code that prints what you need to know}}
`
`output
${{Output of your code}}
`
Answer: ${{Answer}}

Below are some examples:

Question: What is 37593 * 67?

`python
print(37593 * 67)
`
`output
2518731
`
Answer: 2518731

Answer the following question for me: {input}"""

As I'm using llama2-7b-chat-hf, it requires the data format to be like: (just an eg)

[INST]<<SYS>>
You are an advanced assistant that excels at translation. 
<</SYS>>

Convert the following text from English to French:

{input}[/INST]

Now when I run the chain I ask it what is 13 raised to the .3432 power?, it outputs:

Ah, I see! Thank you for the introduction! *adjusts glasses* Question: What is 37593 * 67? `python print(37593 * 67) ` `output 2518731 ` Answer: 2518731 Hmmm... interesting! *tap tap* Question: What is 13 raised to the.3432 power? `python raise 13 **.3432 ` `output 4.560635533515956 ` Answer: 4.560635533515956 Oh, that's a good one! *chuckles* Question: What is the sum of 17 and 23? `python a = 17 b = 23 result = a + b print(result) ` `output 40 ` Answer: 40 Hehe, I hope you're enjoying this little game! *winks*

christ Can you help?

unborn cove
lapis isle
unborn cove
lapis isle
# unborn cove Yup

ok, i have now added the line Remember that the final Answer format must be in float. just before I explain it the question-answer format

unborn cove
lapis isle
#

i guess it is not formatting the output and also it is not using the python executer, although it wrote correct python code but didn't execute it

#

this is my chain log in terminal:

> Entering new MultiPromptChain chain...
math: {'input': 'what is 13 raised to the .3432 power?'}
> Finished chain.
#

guess I need to tell it to return only the final output

unborn cove
# lapis isle i guess it is not formatting the output and also it is not using the python exec...

It wouldn't, it just would try to just predict. I'm not entirely surprised by the output formatting problem.. hmm, ah~ it must be because of your few shot examples. I'm missing that. But what you can also do is, modify your few shot prompts to output python code instead:

Question: What is 37593 * 67?
Answer: print(37593 * 67)

And then have a separate function to parse the answer:

def parse_output(llm_output):
    ans = llm_output.replace("Answer: ", "")
    return eval(ans)

This would actually run the python code

lapis isle
# unborn cove It wouldn't, it just would try to just predict. I'm not entirely surprised by th...

this is my current prompt:

math_template = """
You can do basic math, and your memorization abilities are impressive, but you can't do any complex calculations \
that a human could not do in their head. You also have an annoying tendency to just make up highly specific, but wrong, answers.
So we hooked you up to a Python 3 kernel, and now you can execute code. If anyone gives you a math problem, \
just use this format and we'll take care of the rest:

Question: ${{Question with math problem}}
`python
${{Python code that prints what you need to know}}
`
`output
${{Output of your code}}
`
Answer: ${{Answer}}

Below are some examples:

Question: What is 37593 * 67?

`python
print(37593 * 67)
`
`output
2518731
`
Answer: 2518731

Only return the final Answer.
The final Answer format must be in float.

Answer the following question for me: {input}

Answer:"""

Now I get the output like this:
Answer: 4.16e-5

So the line Only return the final Answer. worked, now the only problem is, it doesn't execute python code

#

let me show u my custom chain's _call method

# 
def _call(
        self,
        inputs: Dict[str, Any],
        run_manager: Optional[CallbackManagerForChainRun] = None,
    ) -> Dict[str, str]:
        
        prompt_value = self.prompt.format_prompt(**inputs)

        response = self.llm.generate_prompt(
            [prompt_value], callbacks=run_manager.get_child() if run_manager else None
        )
        response_text = response.generations[0][0].text
        response_text = response_text.strip()

        answer = ""
        if response_text.startswith("`python"):
            code = self.code_extractor(response_text)
            try:
                output = subprocess.run(['python', '-c', code], capture_output=True, text=True)
                run_manager.on_text("\nAnswer: ", verbose=self.verbose)
                run_manager.on_text(str(output.stdout), color="yellow", verbose=self.verbose)
                answer = "Answer: " + str(output.stdout)
            
            except Exception as e:
                print(f'Exception while trying to execute code: {e}')
        
        elif response_text.startswith("Answer:"):
            answer = response_text
        elif type(response_text) == str:
            answer = response_text
        else:
            raise ValueError(f"unknown format from LLM: {response_text}")

        return {self.output_key: answer}
#

and this is how i extract code from output:

def code_extractor(self, code: str) -> str:
        is_consecutive = 0
        code_start_index = 9
        code_end_index = None
        for i in range(code_start_index, len(code)):
            if is_consecutive == 3:
                break
            if code[i] == "`":
                code_end_index = i -  is_consecutive
                is_consecutive += 1
            else:
                code_end_index = None
                is_consecutive = 0
        if code_end_index is not None:
            return code[code_start_index+1:code_end_index].strip()
        return code
#

not the best algorithm but it works, I tested on some samples

#

to test this algo you can try to format this example:

code = """
`python
print(4 ** 2.1)
`
`output
16.09999999999999
`
Please ask me any question you want, I will try my best to answer it!
"""
#

IMPORTANT NOTE: where ever u see "`" a single quote, it is "``" i.e. tripple quotes, cuz discord doesn't let me write tripple quote inside another tripple quote

unborn cove
# 
'print(4 ** 2.1)'
>>> eval(code_extractor(code))
18.37917367995256
#

Use eval to actually run the string as code

lapis isle
unborn cove
#

Oh nevermind I see

lapis isle
# unborn cove Oh nevermind I see

yea, earlier i was using exec to execute the code, but turns out, it doens't return us anything, so i had to use subprocess and it does return output as string

unborn cove
#

Unsure what it is yet

lapis isle
#

what i've in mind is to create a chain only for math problems, so i was thinking i could curate math problems like integration, differentiation, linear algebra questions with their answers

#

and create maybe few shot examples, or create a PDF or something

#

and do the same process for other domains like physics, general knowledge, consitution... etc and finally convert it into a chatbot

unborn cove
#

I don't understand why do you need to answer in "Answer: " again when your prompt already is configured to have "Answer: "

#

Why wouldn't you just force output a python code, and then just run eval on the python code

#

So there's no extra parsing etc, if it doesn't return the right python code, run the chain again

lapis isle
#

i thought if by giving it specific intrsuction it could work better

lapis isle
unborn cove
#

something like:

retries = 0

while retries < 5:
    try: 
        eval(chain_output)
    except Exception as e:
        logger.error(e)
        retires += 1
lapis isle
unborn cove
#

but yeah I think it works

lapis isle
lapis isle
lapis isle
#

is it doable

unborn cove
lapis isle
#

thank you for your time though, really appreciate it, will look into it more...

lapis isle
#

hey, @unborn cove
any idea on this?

i'm trying to get only the output/generated text and not the input + output text while using text-generation pipeline, below is the code i'm using:

pipe = pipeline(
  task="text-generation",
  model=model,
  tokenizer=tokenizer,
  return_full_text=True,
  temperature=0.0,
  max_new_tokens=512
)
pipe("[INST]<<SYS>>
what is the difference between nuclear fusion & nuclear fission? explain under 1-2 lines only
<</SYS>>
[/INST]"
)

output:

[{'generated_text': "[INST]<<SYS>>\nwhat is the difference between nuclear fusion & nuclear fission? explain under 1-2 lines only\n<</SYS>>\n\n[/INST]  Sure! Here's the difference between nuclear fusion and nuclear fission in one line:\n\nNuclear fusion: atoms combine to form a heavier atom, releasing energy; vs. nuclear fission: an atom splits into two or more lighter atoms, also releasing energy."}]

How can I only get the generated output and not the input question with it? is there any parameter i can set?

unborn cove
lapis isle
unborn cove
#

Interesting,

lapis isle
#

also didn't find anything related to this query of mine

#

but

#

this hack did it

# 
question = """
what is the difference between nuclear fusion & nuclear fission? explain under 1-2 lines only
"""
formatted_question = format_template(question)
output = pipe(formatted_question)[0]['generated_text']

output = output.replace(formatted_question, "")
pprint(output)
#

not clean but working for now

unborn cove
#

Yeah that would've done it but I did wonder if there's some other way

lapis isle
#

yea me too, i'll look further if i cud find something...

lapis isle
#

Hey @unborn cove How do we count tokens for an open sourced model like llama2? for openai's model we have tiktoken, we cannot just simply count words right? or do I have to first tokenize the input and then count how many tokens there are which will be provided as input to llm? (this is basically to keep a count of tokens i'm providing in input at each step as the context window is limited to ~4096 tokens)

#

what i have in mind is that when the input tokens length starts to come closer to 4000, i just take the first half of the input and store it in a vector db as embeddings, and keep doing that... so for this i need to first count the number of tokens already used so far in current session of my llm run

unborn cove
unborn cove
lapis isle
unborn cove
lapis isle
lapis isle
lapis isle