Managing pipeline steps and partial execution

[fading oracle]

Hi, I'm new to dagger, currently trying to evaluate if it could fit us.

Our current pipelines are gitlab based.

I've read the documentation and examples, and couldn't figure out if there is a way to:

1. re-run failed jobs.
2. run a subset of the pipeline.

I also wish to understand how will a multi stage (+multi job) gitlab pipeline will look visually(today i can clearly see the execution status of each job)?
is there a way I can figure out exactly what stage failed (using the web gui...) or I must have a single job containing all the logic 🤔

Thank you!

#python-sdk
#gitlab

[daring hound]

👋 Hi @fading oracle! Welcome 🙂

Re-running whole pipelines or subsets of pipelines (smaller pipelines) is really easy to do locally and interactively with Dagger.

Is there a good way to re-run individual jobs in GitLab CI?

[still sky]

@fading oracle I'm using Dagger in Gitlab as well. https://docs.dagger.io/454108/python-ci (on the gitlab tab) is a good starting point. To answer your question about multi stage/job pipelines, each job is simply a call to run a python script just like the build job in that link. You can have one long python script but then you lose the visual pipeline so I've kept to separate jobs.

As for re-running failed or otherwise jobs, just use the GL UI as usual. To handle subsets of the pipeline you're presumably using a variety of on_success and manual jobs? I.e three jobs run in sequence on success, the next job is a manual step to kick off a production deployment.

In this use-case Dagger (through the python sdk) is simply replacing what often tends to be 1000+ lines of bash + yaml

And since I'm on the topic, you'll probably quickly find (like I did) that writing your pipeline jobs in Python and using Dagger where necessary means you can use other Python libraries which in my case removed a lot of AWS CLI commands (replaced by boto3), excessive jq usage, awful curl commands with template files to send notifications etc

[fading oracle]

first of all thank you for the warm welcome and the amazingly short response time!!! this is truly awesome

@daring hound it's easy - I click a button and all failed jobs are rerun...

[still sky]

And of course, the biggest win of them all, you won't ever see a "fixing gitlab ci file" commit message again, because you can run it all locally until it works and then know it'll work on your GL runner

[fading oracle]

@still sky if I get it right:
you're suggesting that every job will execute a single script (job), so I can continue enjoying giltlabs gui.

if so, what do I benefit from dagger?
*I will have to continue maintaining monster yaml files with dependencies and rules between jobs(or can I pass job state between dagger jobs, potentially executing on different runners?)

• I will not be able to just "run pipeline" locally

not trying to be negative, but to truly understand the power of the sdk and if it can solve our issues and make us happier 🙏😊
it sounds very promising

[still sky]

I think you have both of those wrong

My yaml files are about 1/10th the size they used to be, and each job calls the same thing - run X python script.

In fact 1/10th might be an underestimate, I think we've gone from about 4500ish lines of bash + yaml across two services to about 400ish

We (mostly me 😂 ) now fully develop our pipelines locally. Once it runs locally and does what I need I'll push it and merge that branch into master

I can't answer anything about the shared job states/cache, I don't have any need for that so I've not looked into it

[fading oracle]

so let me make I hard on you because I really feel I'm missing it...
stages:
pre, build, test, deploy

pre jobs:
-static tests

build jobs:
build A, trigger - path a
build B, trigger - path b

Tests jobs:
test A needs build A
test B needs build B
test unified, needs A and B

deploy on success

how can you replace rules->paths, needs, etc.

[still sky]

I'm not sure I fully understand the pipeline - the pipelines I've been working on are linear; stages with jobs are basically:

1. unit tests, sonarqube
2. build image, tag as build, push to dev/prod image repos
3. (manual job) pull build image, retag for qa release, deploy to qa env (aws ecs)
4. (on success) run qa test suite (front-end, automated test pack)
5. (on success/failure) notify qa release success/failure
6. (manual) pull build image, retag for preprod release, aws codedeploy blue/green
7. same as 4
8. same as 5
9. (manual) prod env, repeat 3/4/5 for prod, obviously reduced automated test suite here

Some other stuff at certain stages but that's the core build/test/deploy and as you can imagine it's basically one long horizontal line in GL pipeline editor

[fading oracle]

moat stages in my pipeline have multiple leaves. it's a pretty long and complex DAG.
Can you for example execute 1+2 in a single command locally?

[still sky]

Yeah, sure, it's basically poetry run python3 <script>.py for any given job

As long as I've assumed the right aws role and have creds for a couple of other services in my environment it works just fine

I'm not sure how well GL CI + Dagger will do at a more complex pipeline with a lot of conditionals and leaves, I started on this simpler pipeline because I was pretty certain it'd work

[fading oracle]

imaging I have few isolated services I can test in parallel, and a compose of all those services to a container, needed to be tested as a whole.

[still sky]

Sounds do-able