#python-codegen :: Daggerverse
lucid dragon
starting a thread for this!
@wintry marten if you run dagger install github.com/jpadams/daggerverse/trivy@v0.4.0 that will do all the generation for you
after that in your python code, you can call dag.trivy().scanImage(imageRef) to scan your image
wintry marten
this is not good way:
dagger call scan-image --image-ref alpine:latest
✘ initialize 7.2s
! query module objects: returned error 400 Bad Request: failed to get schema for module "trivy": type "Trivy" is already defined by module "trivy"
Error: query module objects: returned error 400 Bad Request: failed to get schema for module "trivy": type "Trivy" is already defined by module "trivy"
before it:
dagger call scan-image --image-ref alpine:latest
alpine:latest (alpine 3.19.1)
=============================
Total: 2 (UNKNOWN: 0, LOW: 2, MEDIUM: 0, HIGH: 0, CRITICAL: 0)
┌────────────┬───────────────┬──────────┬────────┬───────────────────┬───────────────┬───────────────────────────────────────────────────────────┐
│ Library │ Vulnerability │ Severity │ Status │ Installed Version │ Fixed Version │ Title │
├────────────┼───────────────┼──────────┼────────┼───────────────────┼───────────────┼───────────────────────────────────────────────────────────┤
│ libcrypto3 │ CVE-2024-2511 │ LOW │ fixed │ 3.1.4-r5 │ 3.1.4-r6 │ openssl: Unbounded memory growth with session handling in │
│ │ │ │ │ │ │ TLSv1.3 │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-2511 │
├────────────┤ │ │ │ │ │ │
│ libssl3 │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
└────────────┴───────────────┴──────────┴────────┴───────────────────┴───────────────┴───────────────────────────────────────────────────────────┘
code, install is same like from this site step 5:
steps:
normal Ubuntu 22 install
dagger init --sdk=python trivy
cd trivy
dagger call scan-image --image-ref alpine:latest
jagged sleet
@wintry marten the python-codegen module you linked to is not meant to be used. It's part of a POC.
As for the "Trivy" is already defined by module "trivy" error it seems like you've created a "trivy" module based on the docs, but then tried installing another Trivy module on top. Those are two different implementations. You can use the one in Daggerverse directly. What are you trying to do? Are you just trying to learn or do you need to integrate a Trivy scan with a larger pipeline?
wintry marten
As for the `"Trivy" is already defined by module "trivy"` error it seems like yo...
i need to integrate trivy to scan my docker images from gitlab pipeline, i making POC on it...
wintry marten
we want to try on Czech TV if its free, usefull and we dont want to use so much of files, but rather program it than settuping it on yamls
jagged sleet
Ok, seems like you need your own module, not named "trivy", and with the external trivy module installed (#1234865198431469621 message) for the scanning part.
lucid dragon
Ok, seems like you need your own module, not named "trivy", and with the externa...
@misty spire were you able to make progress with Milan today? 🙏
misty spire
<@933501536624054272> were you able to make progress with Milan today? 🙏
Yes, we made some good progress untangling the combo of things they use.
wintry marten
How to setup it to use this flag:
--ignore-unfixed
its in this file in normal trivy:
https://github.com/aquasecurity/trivy/blob/main/pkg/flag/vulnerability_flags.go
If you're using the external module you'll need to ask the author (issue) or send a PR. In the meantime, or if you're using a copy of the function you see in the docs, you can add it manually there.
misty spire
@wintry marten we could add it to my module easily, wdyt?
wintry marten
super,
lets try, it will be realy nice 😉
wintry marten
Hi all,
I trying to call my github repo.
i have GITPW and GITUSER on .env and this code
but i have problem with:
load_dotenv()
if i have it there, the dagger may not find main.py which is not there...
i have only init.py
I tryed:
@function
async def scan(self, gitpw: str) -> str:
"""Functions for scanning images for vulnerabilities using Trivy"""
if not gitpw:
raise ValueError("GITPW is not provided")
gitpw_encoded = base64.b64encode(gitpw.encode()).decode()
scanned_dir = dagger.Gitrepositary("https://gitlab.ofourcompany.cz/microservice/name-api.git").branch("main").with_auth_token(gitpw_encoded).tree()
return await (
dag.trivy()
.scan_container(self.test(scanned_dir), severity="MEDIUM,HIGH,CRITICAL"))
but it says me that dagger have no Gitrepositary
i try it cause of:
https://dagger-io.readthedocs.io/en/sdk-python-v0.11.2/client.html#dagger.GitRepository.with_auth_token
jagged sleet
Hi all,
I trying to call my github repo.
i have GITPW and GITUSER on .env and t...
First you have two typos in Gitrepositary, but you can't initialize types directly, they need to be created through dag (cc @pallid jackal, this is a basic concept we need to make clear in the docs). Secondly, with_auth_token requires a secret, there's no need to encode a string into base64:
@function
async def scan(self, gitpw: dagger.Secret) -> str:
"""Functions for scanning images for vulnerabilities using Trivy"""
scanned_dir = (
dag.git("https://gitlab.ofourcompany.cz/microservice/name-api.git")
.with_auth_token(gitpw)
.head()
.tree()
)
return await (
dag.trivy()
.scan_container(self.test(scanned_dir), severity="MEDIUM,HIGH,CRITICAL"))
it says: GitRef have no .with_auth_token()
this is my try now, before i will implementing secrets
@function
def get_secret(self) -> dagger.Container:
"""Get the secret value from the environment"""
return dag.container().with_exec(["printf 'my_dev:gnot-say-to-anyone;)' | base64" ])
@function
async def scan(self) -> str:
"""Functions for scanning images for vulnerabilities using Trivy"""
gitpw = self.get_secret()
scanned_dir = (
dag.git("https://gitlab.ofourcompany.cz/microservice/name-api.git")
.head()
.with_auth_token(gitpw)
.tree()
)
return await (
dag.trivy()
.scan_container(self.test(scanned_dir), severity="MEDIUM,HIGH,CRITICAL"))
i found something here on git by:
ssh_known_hosts:
Set SSH known hosts
ssh_auth_socket:
Set SSH auth socket
here i found something on GitRepository on DockerImageScan/dagger/sdk/src/dagger/client/gen.py:
on Class:
class GitRepository(Type):
"""A git repository."""
function:
def with_auth_token(self, token: "Secret") -> Self:
"""Token to authenticate the remote with.
Parameters
----------
token:
Secret used to populate the password during basic HTTP
Authorization
"""
_args = [
Arg("token", token),
]
_ctx = self._select("withAuthToken", _args)
return GitRepository(_ctx)
but dont know if it is right one
how to access there
and if it will be work
I generated Personal Access Tokens with all read rights here:
https://gitlab.ofourcompany.cz/-/user_settings/personal_access_tokens
, because i mean that feed token will not be helpfull
adress have anonymized root
jagged sleet
it says: GitRef have no .with_auth_token()
this is my try now, before i will imp...
Sorry, just change the order here:
dag.git("https://gitlab.ofourcompany.cz/microservice/name-api.git")
+ .with_auth_token(gitpw)
.head()
- .with_auth_token(gitpw)
.tree()
which token i must use? base64 encripted Feed token?
or only feed token?
or nane:password Personal Access Token on this private repo?
jagged sleet
which token i must use? base64 encripted Feed token?
or only feed token?
or nane...
Just the feed token.
wintry marten
still not working on my gitlab...
i have vpn open,
. env is in:
module/dagger/.env
running it by:
dagger call scan --token=env:GITPW
this is my code now:
@function
async def scan(self, token: dagger.Secret) -> str:
"""Functions for scanning images for vulnerabilities using Trivy"""
# gitpw = self.get_secret()
scanned_dir = (
dag.git("https://gitlab.my_company.cz/microservice/name-api.git")
.with_auth_token(token)
.head()
.tree()
)
return await (
dag.trivy()
.scan_container(self.test(scanned_dir), severity="MEDIUM,HIGH,CRITICAL"))
i run it with:
dagger call scan --token=env:GITPW
jagged sleet
with_auth_token uses the username x-access-token by default. If you need to use a different user, then try with_header_token. In this case your secret should be the value for the Authorization header, e.g., basic <base64 encoded username:password>. However, I've tested a private repo in GitLab using with_auth_token with no problem. You say you're using a VPN, so I wonder if everything is accessible network wise. Do you have a public repo on that instance that you can try?
What error do you get?
Also, it's better to test getting the repo's files in isolation instead of also test and trivy scan, so that you have an easier time debugging. Just create a similar function that returns the await scanned_dir.entries() (return type list[str]). This should return the list of files in that directory.
wintry marten
like:
@function
async def scan(self, token: dagger.Secret) -> list[str]:
"""Functions for scanning images for vulnerabilities using Trivy"""
scanned_dir = (
dag.git("https://gitlab.devct.cz/microservice/taxonomy-api.git")
.with_auth_token(token)
.head()
.branch("main")
.tree()
)
return await (
scanned_dir.entries())
but:
dagger call scan --token=env:GITPW
✘ initialize 4.7s
! failed to get value for argument "token": secret env var not found: "GIT..."
✔ Module.initialize: Module! 1.2s
Error: failed to get value for argument "token": secret env var not found: "GIT..."
Run 'dagger call scan --help' for usage.
The error suggests you don't have that env variable exported in your shell.
wintry marten
token is normal feed
jagged sleet
See if this gets a different error:
GITPW=wat dagger call scan --token=env:GITPW
I'd expect that to return a failure to authenticate since wat isn't your token.
You mentioned you have a .env file. Should that be loaded by direnv or something?
wintry marten
this is nice:
first part:
GITPW=wat dagger call scan --token=env:GITPW
✘ DockerImageScan.scan(
token: ✔ setSecret(name: "dontsay"): Secret! 0.0s
): [String!]! 1.3s
! call function "scan": process "/runtime" did not complete successfully: exit code: 1
┃ │ stderr: │
┃ │ fatal: unable to access │
┃ │ 'https://gitlab.dontsay.cz/microservice/dontsay-api.git/': SSL: no │
┃ │ alternative certificate subject name matches target host name │
┃ │ 'gitlab.dontsay.cz' │
┃ │ │
┃ ╰──────────────────────────────────────────────────────────────────────────────╯
second part:
✘ Directory.entries: [String!]! 0.1s
! failed to load cache key: error fetching default branch for repository https://gitlab.dontsay.cz/microservice/dontsay-api.git: git error: exit status 128
stderr:
fatal: unable to access 'https://gitlab.dontsay.cz/microservice/dontsay-api.git/': SSL: no alternative certificate subject name matches target host name 'gitlab.dontsay.cz'
✘ cache request: git://gitlab.dontsay.cz/microservice/dontsay-api.git 0.1s
! error fetching default branch for repository https://gitlab.dontsay.cz/microservice/dontsay-api.git: git error: exit status 128
stderr:
fatal: unable to access 'https://gitlab.dontsay.cz/microservice/dontsay-api.git/': SSL: no alternative certificate subject name matches target host name 'gitlab.dontsay.cz'
Error: response from query: input: dockerImageScan.scan resolve: call function "scan": process "/runtime" did not complete successfully: exit code: 1
Stderr:
╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ Function execution error: resolve: failed to load cache key: error fetching │
│ default branch for repository │
│ https://gitlab.dontsay.cz/microservice/dontsay-api.git: git error: exit │
│ status 128 │
│ stderr: │
│ fatal: unable to access │
│ 'https://gitlab.dontsay.cz/microservice/dontsay-api.git/': SSL: no │
│ alternative certificate subject name matches target host name │
│ 'gitlab.dontsay.cz' │
│ │
it will be nice to load it, but i still try it from src/main/init.py
with dotenv and dotenv load
and it may not find src/main
jagged sleet
it will be nice to load it, but i still try it from src/main/__init__.py
with do...
Ah, the problem is it's the CLI that needs the env var so when it reaches python it's too late.
wintry marten
i used on this only feed token not name:pass personal access token base64
ou.... i have idea if it not needed "Bearer " before it?
jagged sleet
Depends on your setup.
wintry marten
cause the secrets like gitlab adress and token, i may not show you it... grrr baad
jagged sleet
You may have a problem with that SSL certificate. What you're doing with git http auth won't fix that.
It looks like it's not finding a certificate to match the domain you're using to connect. Not sure if you're using a different domain that just resolves to the same IP, possibly having the web server not know about it. Do you know if you have manually installed certificates in your computer to make it work with normal git clone or something like that?
Since dagger runs in containers, you need to replicate the environment in that container, do you know if you need something to make it work? Does it work in a clean docker run -it --rm debian sh?
wintry marten
Hi all,
how to setup dagger to have socket like in this command (-v /var/run/docker.sock:/var/run/docker.sock):
docker run --network=host -v /var/run/docker.sock:/var/run/docker.sock aquasec/trivy image gitlabregistry.devct.cz/microservice/taxonomy-api:1.6.6
how to setup --network=host
jagged sleet
Unfortunately it's currently not possible with functions. See:
What's the use case? Do you need to make a trivy scan on images in your local docker engine? Were they built by an external system to Dagger, or is it possible to build them with dagger? If external, are they accessible in a registry so that you can pull from dagger with dag.container().from_()?
wintry marten
Unfortunately it's currently not possible with functions. See:
- https://github....
we have vpn to company where running gitlab and i need to connect to there and scanning images there on our repo
jagged sleet
Yeah, but are the images available in a registry? Are you building them externally and just need dagger to scan (not build)?
wintry marten
Yeah, but are the images available in a registry? Are you building them external...
but i may not connect cause the vpn has some certificates...
connection to vpn is through the oathtool and we use mobile app Authy
so we will want to build, deploy, scan on merge request with trivy, ruff...
with dagger after this POC on trivy
all of this we have on lots of config files like yaml, yml...
and we want to have it on python in one simple file