Need help with go dependencies | Discord Gophers | Page 1

normal spindle Jun 18, 2023, 12:31 PM

#

I need help with go dependencies. I need to look at a repository and then look at all the immediate dependencies and store each of them inside a map of a custom struct type:

type Artifact struct {
  Name          string
  Version       string
 Dependencies []*Artifact
} ```
Within each immediate dependency, I want to then store its further dependencies in a similar way. Is there any way to do this? Please help if so

brisk dome Jun 18, 2023, 3:58 PM

#

normal spindle I need help with go dependencies. I need to look at a repository and then look a...

so you want an infinitely recursible tree of sorts?

#

if so, then you've already done it with the Dependencies []*Artifact bit

#

a struct contains a field that refers to either a pointer of itself or an array of itself is recursive

#

if i can give one piece of advice?
you might want to add another field called Dependent, that's just a pointer to the Artifact that it depends on. this way, you can go up and down the "tree"

normal spindle Jun 18, 2023, 4:09 PM

#

brisk dome if so, then you've already done it with the `Dependencies []*Artifact` bit

That really isnt my concern. My concern is more about where I can get the deps from

#

So far I've run go list -json ./... | jq -r '.Imports[]' | sort | uniq on my repo after cloning it locally

brisk dome Jun 18, 2023, 4:11 PM

#

normal spindle That really isnt my concern. My concern is more about where I can get the deps f...

so you're trying to add dependencies to your project?

#

what does your go.mod look like

normal spindle Jun 18, 2023, 4:11 PM

#

Not really, I'm trying to create a dependency tree, given a go github repo

#

I mean right now, I'm testing on the https://github.com/etcd-io/etcd/tree/main package

GitHub

GitHub - etcd-io/etcd: Distributed reliable key-value store for the...

Distributed reliable key-value store for the most critical data of a distributed system - GitHub - etcd-io/etcd: Distributed reliable key-value store for the most critical data of a distributed system

brisk dome Jun 18, 2023, 4:13 PM

#

normal spindle Not really, I'm trying to create a dependency tree, given a go github repo

have you tried go list <modulename>?

normal spindle Jun 18, 2023, 4:14 PM

#

brisk dome have you tried `go list <modulename>`?

Won't that just show the modulename?

brisk dome Jun 18, 2023, 4:15 PM

#

with the json flag it shows the dependencies

normal spindle Jun 18, 2023, 4:15 PM

#

Yep so I'm doing this here: go list -json ./... | jq -r '.Imports[]' | sort | uniq

lunar coral Jun 18, 2023, 4:16 PM

#

What about go mod graph?

normal spindle Jun 18, 2023, 4:16 PM

#

lunar coral What about `go mod graph`?

go mod graph gives me all the dependencies, transitive included

#

To create a tree, I need one layer after another

lunar coral Jun 18, 2023, 4:17 PM

#

How about go list -m -json all

#

That has an Indirect flag

normal spindle Jun 18, 2023, 4:18 PM

#

lunar coral How about `go list -m -json all`

Hmm, that seems to be an option

#

So, say I have to have my output like this (in json):

  {
     "name": "A",
     "version": "1.0",
     "dependencies": [
      {
         "name": "B",
         "version": "2.2",
         "dependencies": [
          {
             "name": "D",
             "version": "3.7",
             "dependencies": []
          }
        ]
      },
      {
         "name": "C",
         "version": "0.4",
         "dependencies": []
      }
    ]
  }
]```

lunar coral Jun 18, 2023, 4:20 PM

#

You can just parse the output of go list -json with a json.Decoder

#

You just need to decode multiple elements

#

Each call to json.Decoder.Decode() will decode a single value. So, if you've got a stream of values like what go list -json outputs, you just call Decode() until you get io.EOF

normal spindle Jun 18, 2023, 4:21 PM

#

I was using this: go list -json ./... | jq -r '.Imports[]' | sort | uniq. Does this jq work?

lunar coral Jun 18, 2023, 4:22 PM

#

That won't give you module dependencies

#

you need the -m flag for that

normal spindle Jun 18, 2023, 4:23 PM

#

Also, with go list -json, am I getting the deps for all the packages?

lunar coral Jun 18, 2023, 4:23 PM

#

No

normal spindle Jun 18, 2023, 4:23 PM

#

So, I'm a bit confused about package dep and module dep

lunar coral Jun 18, 2023, 4:24 PM

#

module-aware go list will look at the dependencies defined in the go.mod file

normal spindle Jun 18, 2023, 4:24 PM

#

So, if I look at a project's mode file, its direct deps are all the packages that have been under import() in diff files?

lunar coral Jun 18, 2023, 4:25 PM

#

Yeah, any require line that doesn't end in // indirect is a direct dependency

normal spindle Jun 18, 2023, 4:25 PM

#

So, won't I be getting the same output if I run go list -json ./...?

#

Also, what does all do in go list -json all?

lunar coral Jun 18, 2023, 4:26 PM

#

See go help list

normal spindle Jun 18, 2023, 4:27 PM

#

lunar coral See `go help list`

Oof, its huge

lunar coral Jun 18, 2023, 4:28 PM

#

Try this go list -m -json all | jq -r "select(.Indirect != true) | .Path"

normal spindle Jun 18, 2023, 4:34 PM

#

lunar coral Try this `go list -m -json all | jq -r "select(.Indirect != true) | .Path"`

This gives the required deps from the go.mod

#

Shouldnt this give the same output as well: go list -json ./... | jq -r '.Imports[]' | sort | uniq?

lunar coral Jun 18, 2023, 4:35 PM

#

normal spindle Shouldnt this give the same output as well: `go list -json ./... | jq -r '.Impor...

See for yourself

#

I don't think you can get version information from that though

normal spindle Jun 18, 2023, 4:36 PM

#

It isnt

lunar coral Jun 18, 2023, 4:36 PM

#

Also it includes stdlib packages, which I assume you don't care about

normal spindle Jun 18, 2023, 4:36 PM

#

lunar coral I don't think you can get version information from that though

Yea

normal spindle Jun 18, 2023, 4:36 PM

#

lunar coral Also it includes stdlib packages, which I assume you don't care about

Okay so require section in the mod file doesnt provide standard libs

lunar coral Jun 18, 2023, 4:39 PM

#

What you're doing is literally just looking at the import of all your packages. It's not module-aware. It doesn't have version information

normal spindle Jun 18, 2023, 4:39 PM

#

So I kind of get it. The command which I specified listed each imported pacakge, even if sub libraries of the same lib got used like here:
go.etcd.io/etcd/api/v3/authpb
go.etcd.io/etcd/api/v3/etcdserverpb
go.etcd.io/etcd/api/v3/mvccpb

normal spindle Jun 18, 2023, 4:40 PM

#

lunar coral What you're doing is literally just looking at the `import` of all your packages...

Yes, it makes sense. Just the import sentences.

lunar coral Jun 18, 2023, 4:40 PM

#

go list -m -json all | jq -r 'select(.Indirect != true)'

#

This lists all of the direct dependencies for your module

#

You can extract the base import path and version information from that json

normal spindle Jun 18, 2023, 4:41 PM

#

Yes it makes sense now

lunar coral Jun 18, 2023, 4:41 PM

#

If you need more granular information, such as the dependencies for sub-packages of a module, you can combine the two approaches. Looking at the imports and cross-referencing it with the modules

normal spindle Jun 18, 2023, 4:42 PM

#

Yep

#

Also, I'll also have to skip the main: true pair

#

when listing deps

lunar coral Jun 18, 2023, 4:42 PM

#

Yeah

#

jq -r 'select(.Indirect != true and .Main != true)

normal spindle Jun 18, 2023, 4:43 PM

#

So say I have my first layer ready. Now if I need to find a sub-dep's deps like say this:
{
"Path": "github.com/bgentry/speakeasy",
"Version": "v0.1.0",
"Time": "2017-04-17T20:07:03Z",
"Dir": "/home/abhisman/go/pkg/mod/github.com/bgentry/[email protected]",
"GoMod": "/home/abhisman/go/pkg/mod/cache/download/github.com/bgentry/speakeasy/@v/v0.1.0.mod"
}

lunar coral Jun 18, 2023, 4:46 PM

#

That I'm not too sure about. I'm not sure if there's a way to do it without cloning the repository. I imagine there is

normal spindle Jun 18, 2023, 4:46 PM

#

I think i'll need this package's mod file first

lunar coral Jun 18, 2023, 4:50 PM

#

With that command above, each entry has a Dir

#

You can cd into that dir, run go mod download and then re-run the go list command

#

That works, but it feels flimsy. There must be a better way

normal spindle Jun 18, 2023, 4:53 PM

#

Or maybe I can use the Path?

#

Like here: go list -json github.com/bgentry/speakeasy

lunar coral Jun 18, 2023, 4:55 PM

#

That doesn't appear to work. I was hoping that too

normal spindle Jun 18, 2023, 4:55 PM

#

lunar coral That doesn't appear to work. I was hoping that too

What output are you getting?

lunar coral Jun 18, 2023, 4:55 PM

#

It just prints out the one package, not any of its dependencies

#

Whereas if I cd into Dir and run the same go list command, I get all of the dependencies of that module

normal spindle Jun 18, 2023, 4:57 PM

#

I think I ran go mod download prior because I get all the info for that repo

lunar coral Jun 18, 2023, 4:57 PM

#

So, my project depends on go-redis

#

$ go list -json -m github.com/redis/go-redis/v9
{
        "Path": "github.com/redis/go-redis/v9",
        "Version": "v9.0.4",
        "Time": "2023-05-02T05:40:46Z",
        "Dir": "/home/user/Projects/go/pkg/mod/github.com/redis/go-redis/[email protected]",
        "GoMod": "/home/user/Projects/go/pkg/mod/cache/download/github.com/redis/go-redis/v9/@v/v9.0.4.mod",
        "GoVersion": "1.18"
}```

#

Whereas if I run cd /home/user/Projects/go/pkg/mod/github.com/redis/go-redis/[email protected] and then re-run go list

#

Huh, I get the same thing

#

wtf I swear this was different a minute ago 😆

normal spindle Jun 18, 2023, 4:58 PM

#

Lmao

#

Check if you used Path instead

lunar coral Jun 18, 2023, 5:00 PM

#

Oh duh

#

I was running go list -json -m github.com/redis/go-redis/v9 still, so it only showed my that one module. If I instead run go list -json -m all within the Dir of redis, I get all of the dependencies

normal spindle Jun 18, 2023, 5:01 PM

#

What does all do?

lunar coral Jun 18, 2023, 5:01 PM

#

$ go list -json -m all
{
        "Path": "github.com/redis/go-redis/v9",
        "Main": true,
        "Dir": "/home/user/Projects/go/pkg/mod/github.com/redis/go-redis/[email protected]",
        "GoMod": "/home/user/Projects/go/pkg/mod/github.com/redis/go-redis/[email protected]/go.mod",
        "GoVersion": "1.18"
}
{
        "Path": "github.com/bsm/ginkgo/v2",
        "Version": "v2.7.0",
        "Time": "2023-01-27T15:08:10Z",
        "Dir": "/home/user/Projects/go/pkg/mod/github.com/bsm/ginkgo/[email protected]",
        "GoMod": "/home/user/Projects/go/pkg/mod/cache/download/github.com/bsm/ginkgo/v2/@v/v2.7.0.mod",
        "GoVersion": "1.16"
}
{
        "Path": "github.com/bsm/gomega",
        "Version": "v1.26.0",
        "Time": "2023-01-27T15:21:26Z",
        "Dir": "/home/user/Projects/go/pkg/mod/github.com/bsm/[email protected]",
        "GoMod": "/home/user/Projects/go/pkg/mod/cache/download/github.com/bsm/gomega/@v/v1.26.0.mod",
        "GoVersion": "1.18"
}
{
        "Path": "github.com/cespare/xxhash/v2",
        "Version": "v2.2.0",
        "Time": "2022-12-04T02:06:23Z",
        "Dir": "/home/user/Projects/go/pkg/mod/github.com/cespare/xxhash/[email protected]",
        "GoMod": "/home/user/Projects/go/pkg/mod/cache/download/github.com/cespare/xxhash/v2/@v/v2.2.0.mod",
        "GoVersion": "1.11"
}
{
        "Path": "github.com/dgryski/go-rendezvous",
        "Version": "v0.0.0-20200823014737-9f7001d12a5f",
        "Time": "2020-08-23T01:47:37Z",
        "Dir": "/home/user/Projects/go/pkg/mod/github.com/dgryski/[email protected]",
        "GoMod": "/home/user/Projects/go/pkg/mod/cache/download/github.com/dgryski/go-rendezvous/@v/v0.0.0-20200823014737-9f7001d12a5f.mod"
}

normal spindle Jun 18, 2023, 5:02 PM

#

Okay so lemme just summarize till now

#

go list -m -json all | jq -r 'select(.Indirect != true and .Main != true)' to get all the deps for the project that I'm on

#

Then go mod download to get all the deps locally in cache

lunar coral Jun 18, 2023, 5:03 PM

#

Yah

normal spindle Jun 18, 2023, 5:04 PM

#

And then run go list -m -json all | jq -r 'select(.Indirect != true and .Main != true)' inside each dep's dir

lunar coral Jun 18, 2023, 5:04 PM

#

But I think you need to run go mod download in each dependency directory to get all of its dependencies in the module cache

normal spindle Jun 18, 2023, 5:04 PM

#

lunar coral But I think you need to run `go mod download` in each dependency directory to ge...

Yes, will have to

#

Ig another way without less resources would be by just reading the go.mod file

#

But I'm not sure if that'll work

#

Lemme check

lunar coral Jun 18, 2023, 5:05 PM

#

normal spindle Ig another way without less resources would be by just reading the `go.mod` file

Yeah, that might work too actually

#

You would just need to parse the go.mod file which is pretty straightforward

normal spindle Jun 18, 2023, 5:07 PM

#

lunar coral Yeah, that might work too actually

Okay so it won't it seems

#

I get: go: go.etcd.io/etcd/api/[email protected] (replaced by ./api): reading api/go.mod: open /home/abhisman/Documents/oss/test/api/go.mod: no such file or directory

lunar coral Jun 18, 2023, 5:07 PM

#

Yeah, the thing is, if you don't run go mod download, those modules won't be in your cache. The go.mod files will be, but that's it

#

You'd need to convert from the require lines in each go.mod file to the path in the go mod cache to get that dependency's go.mod file

normal spindle Jun 18, 2023, 5:09 PM

#

Ohh

#

So won't work

lunar coral Jun 18, 2023, 5:09 PM

#

If I run go clean -modcache and then re-run, none of the dependencies have a .Dir, only a GoMod

$ go list -m -json github.com/redis/go-redis/v9
{
        "Path": "github.com/redis/go-redis/v9",
        "Version": "v9.0.4",
        "Time": "2023-05-02T05:40:46Z",
        "GoMod": "/home/user/Projects/go/pkg/mod/cache/download/github.com/redis/go-redis/v9/@v/v9.0.4.mod",
        "GoVersion": "1.18"
}

#

And for redis dependencies, those don't even have a GoMod!

#

$ cat /home/user/Projects/go/pkg/mod/cache/download/github.com/redis/go-redis/v9/@v/v9.0.4.mod
module github.com/redis/go-redis/v9

go 1.18

require (
        github.com/bsm/ginkgo/v2 v2.7.0
        github.com/bsm/gomega v1.26.0
        github.com/cespare/xxhash/v2 v2.2.0
        github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f
)

#

$ go list -m -json github.com/bsm/ginkgo/v2
{
        "Path": "github.com/bsm/ginkgo/v2",
        "Version": "v2.7.0",
        "Time": "2023-01-27T15:08:10Z",
        "Indirect": true
}

normal spindle Jun 18, 2023, 5:13 PM

#

Ig you'll have to cd into the local redis repo and then run go mod download?

lunar coral Jun 18, 2023, 5:13 PM

#

If I run go mod download inside my project, that gives all of the direct dependencies a Dir which I can cd into. When I cd into one of those dirs, and re-run the go list all command, I get that dependency's dependencies, but not all of those have a Dir, only some of them

#

So I'd need to run go mod download for every dependency directory, to ensure all of it's dependencies are present in the local module cache

#

Fairly straightforward

normal spindle Jun 18, 2023, 5:17 PM

#

Yep so cd into each local dir, run go mod download and then reiterate that process recursively until the tree's formed

lunar coral Jun 18, 2023, 5:17 PM

#

Yeah that sounds about right

normal spindle Jun 20, 2023, 5:52 AM

#

This is the code that I have so far

📎 message.txt

#

@fading lichen

#

Still have to make changes but a lot of the work's done

fading lichen Jun 20, 2023, 5:55 AM

#

alright, i took a quick cusrory look

#

there's a quickcomment i have

normal spindle Jun 20, 2023, 5:56 AM

#

Sure

fading lichen Jun 20, 2023, 5:56 AM

#

instead of ```go
func main() {
// Get the GitHub repo URL from the command line.
if len(os.Args) != 3 {
fmt.Println("Usage: go run main.go <repo URL> <branch or tag> ")
os.Exit(1)
}
repoLink := os.Args[1]
branchOrTag := os.Args[2]

// Clone repository
dir, err := filepath.Abs(filepath.Base(repoLink))
if err != nil {
    fmt.Println("Error: Failed to get absolute path for directory")
    os.Exit(1)
}

#

func main() {
    if err := run(); err != nil {
        fmt.Fprintf(os.Stderr, "error: %v", err)
        os.Exit(1)
    }
}

func run() error {
    // Get the GitHub repo URL from the command line.
    if len(os.Args) != 3 {
        return fmt.Errorf("not enough arguments: go run main.go <repo> <branch|tag>")
    }
    repoLink := os.Args[1]
    branchOrTag := os.Args[2]

    // Clone repository
    dir, err := filepath.Abs(filepath.Base(repoLink))
    if err != nil {
        return fmt.Errorf("could not get repo: %w", err)
    }

#

will make your life much easier

normal spindle Jun 20, 2023, 5:59 AM

#

Why so?

fading lichen Jun 20, 2023, 5:59 AM

#

instead of having to print the error and exiting

#

you just return errors

normal spindle Jun 20, 2023, 6:00 AM

#

Hmm, okk

fading lichen Jun 20, 2023, 6:00 AM

#

which are already much better handled and more usable in go

normal spindle Jun 20, 2023, 6:00 AM

#

Will do that

#

Also, I have another question. How do I perform go list -m -json all for a seperate package, lets say: github.com/bradfitz/slice

fading lichen Jun 20, 2023, 6:01 AM

#

go list -m -json github.com/bradfitz/slice

#

no -m tho

#

imo you should work on the package level isntead of on the module level

normal spindle Jun 20, 2023, 6:02 AM

#

fading lichen imo you should work on the package level isntead of on the module level

But I need the version number as well and package level doesn't give me that info

fading lichen Jun 20, 2023, 6:03 AM

#

ah

normal spindle Jun 20, 2023, 6:03 AM

#

So I think I'll need to navigate inside this package locally and then run go list -m -json all

fading lichen Jun 20, 2023, 6:05 AM

#

maybe, i dont have the time to look into the otuput of -m atm unfortunately

#

but if you want to do that, shold be pretty straightforward with os.Chdir

normal spindle Jun 20, 2023, 6:06 AM

#

Yep, will do that

fading lichen Jun 20, 2023, 6:06 AM

#

"GoMod": "/home/abhisman/go/pkg/mod/cache/download/github.com/bradfitz/slice/@v/v0.0.0-20180809154707-2b758aa73013.mod" you have the info you need here, using filepath

normal spindle Jun 20, 2023, 6:06 AM

#

Or I can use the "Dir" key from the json data instead

fading lichen Jun 20, 2023, 6:07 AM

#

yeah

#

you are right

normal spindle Jun 20, 2023, 6:09 AM

#

Ah but it seems like some deps like aws-sdk-go don't have Dir key:

  {
    "Path": "github.com/aws/aws-sdk-go",
    "Version": "v1.44.244",
    "Time": "2023-04-14T18:34:58Z",
    "GoMod": "/home/abhisman/go/pkg/mod/cache/download/github.com/aws/aws-sdk-go/@v/v1.44.244.mod",
    "GoVersion": "1.11"
  }

#

GoMod it is

lunar coral Jun 20, 2023, 8:08 AM

#

Not all of them will have a GoMod either, from my testing

fading lichen Jun 20, 2023, 8:09 AM

#

does does go mod download -json show for those deps without a go.mod

lunar coral Jun 20, 2023, 8:11 AM

#

fading lichen does does `go mod download -json` show for those deps without a go.mod

It doesn't

#

I get weird stuff like the module name being "command-line-arguments"

#

go mod anything just fails

fading lichen Jun 20, 2023, 8:12 AM

#

i mean

#

if you import something without a go.mod

#

go mod download works

lunar coral Jun 20, 2023, 8:12 AM

#

Oh yeah it does

fading lichen Jun 20, 2023, 8:12 AM

#

but if you go.mod download with -json what does it show

lunar coral Jun 20, 2023, 8:30 AM

#

Looks very similar to go list -m -json all, only it doesn't show some transitive deps. Specifically those without a GoMod or Dir

#

Presumably they're not downloaded because they're not actually used? They're just indirect dependencies

#

This is the (partial) output of go list -m -json all on my project. I've just done a go clean -modcache followed by go mod download for the project. The second two dependencies aren't shown when I run go mod download -json

{
  "Path": "github.com/deitrix/playground/redisratelimiter",
  "Main": true,
  "Dir": "/home/user/Projects/go/src/github.com/deitrix/playground/redisratelimiter",
  "GoMod": "/home/user/Projects/go/src/github.com/deitrix/playground/redisratelimiter/go.mod",
  "GoVersion": "1.20"
}
{
  "Path": "github.com/bsm/ginkgo/v2",
  "Version": "v2.7.0",
  "Time": "2023-01-27T15:08:10Z",
  "Indirect": true
}
{
  "Path": "github.com/bsm/gomega",
  "Version": "v1.26.0",
  "Time": "2023-01-27T15:21:26Z",
  "Indirect": true
}
{
  "Path": "github.com/cespare/xxhash/v2",
  "Version": "v2.2.0",
  "Time": "2022-12-04T02:06:23Z",
  "Indirect": true,
  "Dir": "/home/user/Projects/go/pkg/mod/github.com/cespare/xxhash/[email protected]",
  "GoMod": "/home/user/Projects/go/pkg/mod/cache/download/github.com/cespare/xxhash/v2/@v/v2.2.0.mod",
  "GoVersion": "1.11"
}
{
  "Path": "github.com/dgryski/go-rendezvous",
  "Version": "v0.0.0-20200823014737-9f7001d12a5f",
  "Time": "2020-08-23T01:47:37Z",
  "Indirect": true,
  "Dir": "/home/user/Projects/go/pkg/mod/github.com/dgryski/[email protected]",
  "GoMod": "/home/user/Projects/go/pkg/mod/cache/download/github.com/dgryski/go-rendezvous/@v/v0.0.0-20200823014737-9f7001d12a5f.mod"
}```

fading lichen Jun 20, 2023, 8:43 AM

#

fun

lunar coral Jun 20, 2023, 8:45 AM

#

I made my own deps walker program and it frequently gets caught in a loop because a lot of package depend on older versions of themselves transitively 😄

#

particularly the golang.org/x packages

#

There's also the whole MVS thing, whether you want to account for that or not

normal spindle Jun 20, 2023, 9:25 AM

#

Okay so to stay safe, go mod init

lunar coral Jun 20, 2023, 10:08 AM

#

Well, you won't get correct dependency versions if you do that. The question is, do you need them?

normal spindle Jun 20, 2023, 10:30 AM

#

lunar coral Well, you won't get correct dependency versions if you do that. The question is,...

Why not? And yes, I need the versions

normal spindle Jun 20, 2023, 1:36 PM

#

@fading lichen If you're done with work then can I please show you the recursion bit of my program. I haven't been able to solve it yet

fading lichen Jun 20, 2023, 1:37 PM

#

just need to post the code, either myself or someone else who's available can take a look

normal spindle Jun 20, 2023, 1:38 PM

#

Sure

#

I've uploaded it here. The recursion bit starts from the populateDependencyTree() function

📎 message.txt

normal spindle Jun 20, 2023, 2:24 PM

#

gophercrying

normal spindle Jun 20, 2023, 7:45 PM

#

📎 message.txt

#Need help with go dependencies