#πŸ‘‹ I'm just looking to get this setup

1 messages Β· Page 1 of 1 (latest)

nimble fable
boreal pecan
#

Hi! It's probably not user error.. I just pushed a change to how we handle LLM endpoint configuration, and probably broke something... sorry!

  • Can you confirm that the .env is in the current directory? At the moment it doesn't work if the .env is in a parent dir (even though it should)
#

Also: try setting these as actual env variables, rather than .env, and see if it resolves the issue?

nimble fable
#

will do - do you have any advise on the format for the BASE_URL env var?

boreal pecan
nimble fable
#

πŸ€” is there a way to trigger a one off call to the LLM?

❯ ~/bin/dagger-llm shell -c "llm | with-prompt 'hello' | history "
βœ” connect 0.1s
βœ” looking for module 0.2s
βœ” loading type definitions 0.2s

βœ” llm: Llm! 0.0s
βœ” .withPrompt(prompt: "hello"): Llm! 0.0s
β”‚πŸ§‘ hello
βœ” .history: [String!]! 0.0s

πŸ§‘ πŸ’¬hello
boreal pecan
nimble fable
# 
❯ ~/bin/dagger-llm shell -c "llm | with-prompt 'hello' | loop | history "
βœ” connect 0.1s
βœ” looking for module 0.2s
βœ” loading type definitions 0.2s

βœ” llm: Llm! 0.0s
βœ” .withPrompt(prompt: "hello"): Llm! 0.0s
β”‚πŸ§‘ hello
βœ” .loop: Llm! 1.4s

I can't see the call to the LLM, or an error out

boreal pecan
#

thought it looked cool, because you can see where the "agentic loop" takes place. But adds unnecessary verbosity so I will change it back

boreal pecan
#

@nimble fable can you try running dagger shell then running the same dagger pipeline but interactively?

nimble fable
#

ok - there is this in the trace now with the env var set

Screenshot_2025-02-18_at_15.30.46.png
#

so that's picked up the correct vars, I think

#

ok - more of an error this time

# 
❯ ~/bin/dagger-llm shell                                                                  [15:30:22]
Dagger interactive shell. Type ".help" for more information. Press Ctrl+D to exit.

βœ” llm | model 0.0s
llama-3-70b(other)

✘ llm | with-prompt "can you say hello world back to me" | loop | history 1.3s
! input: llm.withPrompt.loop panic while resolving Llm.loop: runtime error: invalid memory address or nil pointer dereference
#

this code

func (r *LlmRouter) isOpenAIModel(model string) bool {
    return strings.HasPrefix(model, "gpt-") || strings.HasPrefix(model, "openai/")
}
# 
func (r *LlmRouter) isLlamaModel(model string) bool {
    return strings.HasPrefix(model, "llama-") || strings.HasPrefix(model, "meta/")
}
#

my env of Model = "llama-3-70b" will fall out to

func (r *LlmRouter) routeOtherModel() *LlmEndpoint {
    return &LlmEndpoint{
        BaseURL:  r.OPENAI_BASE_URL,
        Provider: Other,
    }
}

which now doesn't send the API key

# 
☸ staging (civo-system) in melvin on ξ‚  main [!] on ☁️  dinesh@civo.com
❯ env | grep OPENAI | grep -v KEY
OPENAI_BASE_URL=https://api.relax.ai
OPENAI_MODEL=openai/llama-3-70b
☸ staging (civo-system) in melvin on ξ‚  main [!] on ☁️  dinesh@civo.com
❯ ~/bin/dagger-llm shell
Dagger interactive shell. Type ".help" for more information. Press Ctrl+D to exit.

βœ” llm | model 0.0s
llama-3-70b(other)
#

I would expect the llm | model to output

lama-3-70b (openai)

boreal pecan
#

yes that's the very new code (12 hours old) and clearly it's broken... will try to fix now

#

ah that's it, I didn't think I would need to send the API key. silly in retrospect

#

but that shouldn't cause a panic so there's probably something else

#

can you see the stack trace anywhere?

ivory moss
fierce sail
#

I noticed with the former code that I would get a "couldn't find .env file" error some percentage of the time, even though it was there, when running dagger shell and connecting to the llm dev engine.

nimble fable
kind oriole
fierce sail
boreal pecan
#

reminder guys we're in a support thread 😁

fierce sail
# nimble fable

Your error looks very familiar to me. I think I was getting that previously. Happy to jump on with you to troubleshoot in discord.

boreal pecan
#

it's probably something silly that I introduced last night

nimble fable
#

@fierce sail - I can hop on a call within the next hour

boreal pecan
#

I'm pushing a first fix, to send the API key on your endpoint. That shouldn't be the cause for the panic, but it's needed either way

#

pushed

#

If you pull the latest version of melvin, and re-run dagger shell -c './dagger-llm | engine | up' (or re-run ./setup.sh) the fix should be applied

fierce sail
#

@boreal pecan the v1 is getting stripped off the baseURL somehow when chat/completions is appended

kind oriole
nimble fable
fierce sail
#

Going to add the trailing slash, restart engine, and try it

kind oriole
#

Oh yep trailing slash is required

#

a quirk of the openai client

nimble fable
#

restarted the engine, and this is now sending requests

A new shell only did not work

#

requests are now being sent to the LLM, I still get this error

Screenshot_2025-02-18_at_17.43.41.png
fierce sail
#

Anything else we could collect for diagnosis @kind oriole besides πŸ‘†

kind oriole
#

This is on relax? Which model?

fierce sail
#

If he drops loop the errors go away, but doesn't call the LLM

nimble fable
#

llama 3, 70b

#

I've shared an API key for relax with Jeremy

fierce sail
#

@kind oriole we can figure it out and follow up.

kind oriole
#

Maybe because it's streaming? Not sure

#

fwiw I'm running vanilla ollama and not seeing this error

fierce sail
kind oriole
boreal pecan
fierce sail
boreal pecan
#

@nimble fable can you let me know when you have a chance to try again? I'm curious if my fix helps

nimble fable
#

checking now

#

@boreal pecan, I can't see a recent change. The last one I have is

  • 23cf032 (HEAD -> main, origin/main, origin/HEAD) update dagger dependency

this now get's a response from the LLM correctly (so passes the key in the request) but it still shows the stacktrace above

boreal pecan
#

Oh sorry I missed that update

#

I feel like I'm missing something obvious... Do you spot anything @kind oriole @ivory moss in that stack trace?

#

It seems related to streaming

#

Maybe it crashes if an endpoint doesn't support streaming, or handles it in a non-standard way?

#

This is in the stack trace:

    acc := new(openai.ChatCompletionAccumulator)
    for stream.Next() {
        if stream.Err() != nil {
            return nil, stream.Err()
        }

        res := stream.Current()
        acc.AddChunk(res)

                // πŸ‘‡
        if content := res.Choices[0].Delta.Content; content != "" {
                // πŸ‘†
            if logsW == nil {
                // only show a message if we actually get a text response back
                // (as opposed to tool calls)
                ctx, span := Tracer(ctx).Start(ctx, "LLM response", telemetry.Reveal(), trace.WithAttributes(
                    attribute.String("dagger.io/ui.actor", "πŸ€–"),
                    attribute.String("dagger.io/ui.message", "received"),
                ))
                defer telemetry.End(span, func() error { return rerr })

                stdio := telemetry.SpanStdio(ctx, "",
                    log.String("dagger.io/content.type", "text/markdown"))

                logsW = stdio.Stdout
            }

            fmt.Fprint(logsW, content)
        }
    }
ivory moss
#

that's my guess yeah

kind oriole
#

Yeah can you check res.Choices first before [0]?

ivory moss
#

it's kind of weird, i'd expect stream.Err() != nil in that case

boreal pecan
#

@kind oriole @ivory moss can I let you handle this one? I'm close to a fix for "bug 7" (clean llm config without having to re-run llm after setup & each engine restart)

kind oriole
fierce sail
kind oriole
#

Got the repro, thanks @fierce sail , working on a fix

kind oriole
#

Pushed a fix. My repro is working now!