#thanks. we still do a bit of manual
1 messages · Page 1 of 1 (latest)
say more? i think if there is a tool call in the context it should still work
could you show us the test you would like to create
@edgy kindle is the wizard behind our testing framework and im sure hed be eager for your feedback
test spec
if theres a doc that shows how to do that, great, but i spent 15 mins poking around and didnt find anything.
(we also have a test to validate the right tool call, which works.. but - I dont care what the tool call does/replies with - need to test the LLMs response to teh user)
ah so you want to test the tool call happens and then the reply is correct?
Yes. Simply verify the tool was called isnt really valuable (and is testable in the tool test already).
The value is ensuring the response to the user is the right thing.
hey @teal sage any news on this?
If we cant use the builtin testing... then we're back to using our own external testing agent, which brings us back into our ecosystem of just building our own LLM API chatbot 🙁
Hi @agile ginkgo, I am aware of this limitation, work is underway to resolve it