#Jenkins Integration and CentOS 8

1 messages · Page 1 of 1 (latest)

tough hornet
#

Hey, I was looking at migrating our tests over to Dagger, and I ran into an issue that I wasn't sure how to solve. We run our tests on a local Jenkins server running in Docker. We've manually installed Python3.10, and mounted docker and docker socket into the contianer. We then try to run our Dagger based CI using the following Jenkin's pipeline

pipeline {
    agent any

    stages {
        stage('build image') {
            steps {
                git branch: 'master', url: 'https://github.com/robertu94/libpressio'

                sh '''#!/bin/bash
                python3.10 -m venv workenv;
                . ./workenv/bin/activate;
                pip3.10 install "dagger-io";
                python3.10 ./ci/test.py
                '''
            }
        }
    }
}

However, I get a permissions error on /.cache/dagger. Is there some setup step that I missed?

ornate reef
#

@tough hornet On my machine, that .cache/dagger directory is under the user's homedir. Here's mine under my homedir ~:

~ ➤ ls ~/.cache/dagger
dagger-engine-session-9ad18aca71b1b1ba
#

So it looks like maybe your shell user has homedir set to / aka root of the filesystem, but they don't have permission to write there: /.cache/dagger

tough hornet
#

Is there a way to put this in a specified directory (e.g. WORKSPACE/.cache or /tmp)

rocky vector
ornate reef
tough hornet
#

I think that worked!

tough hornet
#

Ok this got me past my permissions issues, but now I have a different one.

#

The dagger engine is in an infinite crash loop. I am binding both docker and the docker socket into the container running Jenkins, but I get the same error if I try to run the docker engine using the same command on the host. I've also confirmed that I get the same error running outside of the container.

#

I've confirmed that fuse-overlayfs is installed in both places as well as modprobe

#

I'm running on a CentOS 8 host with SELinux in permissive mode.

#

I get the same error on the host running the same pipeline.

#

I've uninstalled podman and am using actual Docker 23.0.1

hollow spade
#

@slate rampart, does it look like something related to the network changes for the services API?

tough hornet
#

Jenkins Integration and CentOS 8

rocky vector
#

can you run iptables -t nat -S --wait on the host? Additionally, can you run lsmod | grep tables and share the result please?. cc @tough hornet

tough hornet
#

This looks normal to me.

#

@rocky vector

rocky vector
#

@tough hornet can you add --security-opt seccomp=unconfined to your docker run and check if that helps?

tough hornet
#

@rocky vector same exact error as before. (modulo the timestamps).

rocky vector
#

ok, one last try. How about disabling apparmor? --security-opt apparmor=unconfined

tough hornet
#

No apparmor on CentOS.

#

And just to confirm, I get the same error message.

rocky vector
#

👍 can you try running this on the host and giving it another shot pleasea?

sudo modprobe iptable_filter
sudo modprobe iptable_nat
tough hornet
#

That seems to have fixed it. I'm currious as to why these kernel modules didn't load on-demand.

#

Let me try in the container next.

#

As a work around, I can add a directive on the host to load these modules on boot.,

rocky vector
#

yeah... CentOS has some strange behaviors about this. I recall having related issues in the past

tough hornet
#

Is there somewhere in the Docs, I could submit a PR on what we've found here?

#

I looked in the docs, but I didn't find anything/anywhere that seemed appropriate.

rocky vector
#

docker docs, you mean?

tough hornet
#

dagger.

#

Docker works fine for other containers.

#

Even networking things like nginx/plex.

rocky vector
#

that's because nginx doesn't use iptables. If you try to run anything that uses iptables inside the containers you'll get the same issue

#

this is not a dagger issue

tough hornet
#

Got it.

#

Could we perhaps then document for dagger what kernel modules it requires?

#

Then the user could at least investigate if they were loaded?

rocky vector
tough hornet
# tough hornet Let me try in the container next.

Ok. I ran into a few more permissions issues in the container. I needed to create /.docker and give the jenkins user access to it, but my CI is getting way farther than it has previously running in Jenkins. It takes about 1/2 an hour to bootstrap this so I 'll know then.

regal lotus
tough hornet
#

Ok. Everything seems to be working now including weekly cache invalidation. thanks everyone!