#How do I better handle output for debugging and running tools locally?

1 messages · Page 1 of 1 (latest)

forest sandal
#

I've been working on #1319757538471837796

I've been tempted to just flip to using a normal Go CLI instead, because the noisy output makes this tricky to debug and control the output noise. I'm sharing as maybe I'm just missing some features.

  • The actual CLI/api calls development was the least complicated, it was more working around file/output confusion with dagger that slowed me down.
  • For example PlainText from a secret is outputting a line on every paginated call instead of being at a debug level.
  • My fmt.Printlns only show up if I increase the verbosity level and at that point it's a ton to dig through.
  • I didn't see any simple way to use span/otel like you have done in your code and have it enter a dedicated line that I could put at information/higher level so I could have easy structured log output. The only way I think I could get what I want is maybe using a OTEL viewer locally?
span.SetAttributes(attribute.KeyValue{
    Key:   attribute.Key("url"),
    Value:  attribute.StringValue(url),
})

That's verbose but if it had shown up results locally on a line I could deal with that, but I couldn't find.

  • I'd be fine with not outputting spinners and other nice CLI stuff if I could better handle structured lines that would show up at -v level for me to see indented on the output.

The lack of output control is hard for me since I like to be make very clean log output with Mage, or use structured logging with zerolog and use the console writer with nice formatting tweaks to support clean local reading.
It's also hard if I want to wrap up some of this and share with folks as they'll get a "wall of noise" for any progress I want to output to summarize from what I can see.

Do you have any projects that give examples of controlling output for better debugging and working through issues?

CleanShot_2024-12-20_at_14.10.482x.png
verbal hemlock
#

i vibe with this pretty hard, been working on the engine itself and it's often an unreasonable amount of logs to dig through when you need to examine the emitted logs of a service or even the mainline call... maybe @hybrid igloo or @sullen vale have tips?

hybrid igloo
#

I usually babysit verbosity and bring it down to 0 to trim out the completed steps, and/or zoom in/out of spans to focus on them, and/or open individual spans in Cloud (esp after navigating to it in the TUI so it opens the UI directly to it)

fmt.Println shouldn't need increased verbosity, bit confused by that one. If that's happening could you open an issue? 🙏 (will try to keep it in mind and check when I'm in the area)

I do think there's more to do here for sure, still trying to figure it out

forest sandal
#

I'm at the point today been considering just flipping it to normal Go, instead of wrapping this automation up in dagger, so if anyone has some spare cycles later in maybe 430-5pm CST and wants to do a huddle to chat, maybe you can inspire me. I normally do async, but maybe if I just share my screen and talk I can iron this out better. No problem if busy though. it's not urgent, just would be great as someone continuing to try to push past some friction points to see what I might be missing.

Esp confused on the file creation/mapping/export #1319757538471837796 as I could have been done with this in normal Go a while ago, but trying to figure out the dagger nuances has slowed this down. (When I called dagger via mage (before functions redesign) it made sense, but now I'm not quiet there on the model.

can do a chat later if folks are still busy for holidays and not a good time for sure.

forest sandal
#

@hybrid igloo would it be better for me to post this in github? I don't mind, just have this one + other with no responses and not sure how to get some help/input.

I know I have compromises over controlling with something like Mage, but i definitely need to be able to debug a bit better and understand directory/path. Gonna go look at the docs today again in case I missed something.

hybrid igloo
#

@forest sandal sorry, you're in the right place but this must have fallen through the cracks with the holiday season + everyone ramping back up 🙏

#

also posted a reply in the other thread, hope that helps

forest sandal
#

@hybrid igloo i'm gonna be doing one of the first show and tells for a dagger style project tomorrow for my team.
Is there any recommendation for the current output options that would clean things up?

For example, every time I read the secret to plain text right now it just spams an entry and this makes a lot of noise. Right now even basic output without verbose isn't going to be as nice for demoing.

I can't really use dagger cloud and needed to be local.

If nothing right now that's fine. Just figured I'd ask. Maybe I can run it with the interactive TUI if that still exists so after it's collapsed, expand different span sections perhaps.

hybrid igloo
#

@forest sandal yeah sorry, things in the "make the UI less cluttered" category are still in progress. but you can run with -E to keep it from exiting, so you can browse around at your liesure

forest sandal
#
  • can i supress the plaintext secret output streaming?
  • what level do I use if i want something to show up as a level to expand or not.... like can I i specify a span or do I need a dagger pipeline name or something to "group" so it's easier to show?
forest sandal
#

well the web ui is beautiful so i'll do that for demo'ing. I just need to see some details on security policy for local trace viewing, and then that would be wait better, as the spans group and everything like i'd want

#

this is really annoying though. Is there a way to supress this? Perhaps i need to use OTEL filter exclusion and figure that part out if not built in. Makes it hard to use in web.

If I figure out the child spans to nest it then I can organize by individual orgs/groups

CleanShot_2025-01-14_at_19.54.452x.png
hybrid igloo
#

@forest sandal when is your demo? i just implemented a TUI feature for that (press 's' to sift a call's child spans revealing only custom spans, i.e. hiding calls). I can add the same in the v3 UI (https://v3.dagger.cloud) tomorrow

Dagger Cloud

Browse and visualize Dagger traces.

forest sandal
#

oh nice!

#

no pressure, just cool to see. That sounds neat. I am doing in morning 11am cst. Reducing noise is always a win for demos 🙂

#

keep up the good work!

hybrid igloo
forest sandal
#

@hybrid igloo ok looking now... if i should respond in PR/issue then point me to it.

  • what's the proper way to install your version to run? ( use aqua-proj binaries to grab release, but if I should go install it or such please advise and I'll use this and test/use for a bit!)
  • i have demo in 1-2 hours so if the web ui is updated let me know.
hybrid igloo
#

@forest sandal yeah - that's the branch to build for the CLI + engine changes. Do you need help getting that set up?

#

web UI is shipped

forest sandal
#
  1. to same url or in prod?
  2. if there is a go install or a dagger call to produce the binary for me from a target branch let me know. I'll check out the TUI improvements. Not sure if dagger call against your forked repo or a go install would be the go to for now
hybrid igloo
#
  1. this URL - it hasn't made it to prod yet: https://v2-v3-redirect-v3.preview.dagger.cloud/
  2. I would clone the repo, check out the branch, and run ./hack/dev - that will build + start the dev engine and export the dev CLI to ./bin/dagger.

Then, for your demo you can add a shim script like this to your $PATH:

#!/bin/bash

~/src/dagger/hack/with-dev dagger "$@"

I have this installed as ~/bin/dagger-dev but for your demo you might want to just call it dagger and make sure it takes precedence

verbal hemlock
#

you can also just path-shim all of dagger/hack/, like this: ```export PATH=$HOME/src/dagger/hack/:$PATH

#

then you dev to build the engine and with-dev dagger call ...

hybrid igloo
#

yeah true - i just think you probably wouldn't want all that typing for a demo

#

maybe you could do with-dev <shell> thinkspin

forest sandal
#

doing now.

Might be useful to have a dagger function that points to any target fork/repo and produces a platform binary and script ready to go. Might take a swing someday at that 🙂

Random side note, I think i helped advocate for mage way back in the begining as a big fan. I know it's "deprecated" but I love seeing it still holding on by it's teeth. 😆

hybrid igloo
forest sandal
#

i have a task file to run the args. I can just switch the binary I point to an problem solved :-p

#

i want to do terminal but the only reason i have to do this is if I can use TUI interactive elements for user to select options/etc and I wasn't clear if this was persisted in scope since i think file changes aren't

verbal hemlock
hybrid igloo
#

i think this works? (running locally, just takes a bit)

dagger call -m github.com/vito/dagger@frontend-api dev terminal
forest sandal
#

sorry what's this supposed to be for? I mean i know it's the terminal, but what does it relate to this issue?

#

And is it supposed to support me using interactive console prompts to then pass that back as variables or is that scope lost?

verbal hemlock
#

it'll drop you into a terminal where vito's develop version of the dagger engine and cli are installed, inside of which you can run whatever you were hoping to run without needing the ./hack/ scripts

forest sandal
#

That's nifty, i need to ensure I have a few things mapped.
How do I convert this type of call into a terminal one as I need a token mapped in.

    {{.DAGGER_BINARY}} call issues \
              -E \
              --org-id=all \
              --format=csv,xlsx \
              --dedup=true \
              --days=7 \
              --token=env:SNYK_TOKEN \
              export --path .artifacts/

And i need the files from the report that is running

verbal hemlock
#

think_spin wiring the env var in is awkward, I think... but you'd add dev --target=$path-to-module-source-where-issues-is-implemented terminal to get your module in there (also probably possible to load it from a git source)

forest sandal
#

if willing, please advise full command so i grok this. No pressure.
I'm wrapping up stuff and not sure what this would look like and demo in one hour so trying to get the TUI to run for me now

#

back online in 15. I'm not getting it to work yet but if you have a way for me to invoke this custom binary + the engine to be resolved from my other folder/module let me know. Right now it finds the dagger cli but the engine doesn't get loaded from the folder and my attempts to grab the with-dev and use didn't work yet.

verbal hemlock
# 
dagger call -m github.com/vito/dagger@frontend-api dev --target=~/src/cwlbraa-daggerverse/benchmarks terminal

replace src/cwlbraa-daggerverse-benchmarks with your own module (where issues is implemented), then call dagger call as normal in the terminal you're dropped into

hybrid igloo
#

eh yeah probably best to just skip the dev terminal approach tbh, if you have a demo prepared assuming you're on the host you'll have more things to figure out just to get started

verbal hemlock
# 
dagger call -m github.com/vito/dagger@frontend-api dev --target=$YOUR_REPO with-env-variable --name=SNYK_TOKEN --value=$SNYK_TOKEN terminal
#

is the basic idea tho

#

but yeah vito's right, if you've got other client filesystem interactions going on hack is gonna make things easier

forest sandal
#

The last one is useful. But in the terminal what am I expected to type to invoke , just the dagger call as I would normally do but now I'm invoking in this sandbox? And would export to host for report still work?

verbal hemlock
#

yes just the normal dagger call, export to host will "work", but in that case the "host" is the container running that terminal

forest sandal
#

Inception , exporting my export from a Container in a container

verbal hemlock
forest sandal
#

i didn't get to demo the reduced noise, some issues to get that working but did successfully demo the call.

The ssh failed to work for some reason with them despite them being able to clone this way, but the https worked out of the box the first time. NICE