Postgres & Redis in Dagger | Dagger | Page 1

dense cipher Jul 14, 2023, 7:05 PM

#

Hi and thank you in advance for help!

I am trying to use postgres and redis for my Dagger pipeline. I grab the environ variables from GitLab at the top of my code, then pull them into my postgres container using .with_env_variable(). I have the following code:

        # ======================== POSTGRES ========================
        postgres = (
            # use a postgres container
            client.container().from_("postgres:12")
            # set the environment variables
            .with_env_variable("POSTGRES_USER", POSTGRES_USER)
            .with_env_variable("POSTGRES_PASSWORD", POSTGRES_PASSWORD)
            .with_env_variable("POSTGRES_DB", POSTGRES_DB)
            .with_env_variable("POSTGRES_HOST_AUTH_METHOD", POSTGRES_HOST_AUTH_METHOD)
            .with_env_variable("PORTALD_DATABASE_URL", PORTALD_DATABASE_URL)
            .with_exec(["postgres"])
        )

        # ======================== REDIS ========================
        redis = (
            # use a redis container
            client.container().from_("redis")
            # set the environment variables
            .with_env_variable("PORTALD_REDIS_URL", PORTALD_REDIS_URL)
        )

then binding the service to test container:

python = (
            # use a python 3.10 container
            client.container().from_("python:3.11-slim-buster")
            # bind redis service
            .with_service_binding("redis", redis)

I got this error when doing so:

resolve image config for docker.io/library/redis:latest ERROR: failed to copy: httpReadSeeker: failed open: failed to do request: Get "[link]": remote error: tls: handshake failure

In my configuration, am I doing something?

lavish cargo Jul 14, 2023, 8:49 PM

#

@dense cipher Could you try to put a docker login in your GitLab CI yaml before running your Dagger python pipeline.
Perhaps you're hitting an unauthenticated rate limit.

slender gyro Jul 15, 2023, 2:33 PM

#

@dense cipher were you able to resolve this?

dense cipher Jul 17, 2023, 12:02 PM

#

lavish cargo <@358807393515208704> Could you try to put a `docker login` in your GitLab CI ya...

I'll try this now and let you know if it fixes it!

dense cipher Jul 17, 2023, 12:02 PM

#

slender gyro <@358807393515208704> were you able to resolve this?

not yet, trying again today!

slender gyro Jul 17, 2023, 12:23 PM

#

👍 let us know in case you need more help

dense cipher Jul 31, 2023, 1:26 PM

#

Hi @slender gyro @lavish cargo I hope you both are doing well! I resolved that error, but now I have a different problem that you two might be able to help with. The redis service finishes set up fine, but then when the postgres service is starting, here is what happens:

45: > in from postgres:12
45: > in service GMPK37FADP1EK
45: extracting sha256:30eea71c5b9f3722001a1479eaad8fecbd7369e44a3094e4393fe5ca15077d50 
45: extracting sha256:30eea71c5b9f3722001a1479eaad8fecbd7369e44a3094e4393fe5ca15077d50 [0.01s]
45: pull docker.io/library/postgres:12 DONE
43: exec docker-entrypoint.sh postgres
43: > in service GMPK37FADP1EK
43: [1.67s] Success. You can now start the database server using:
43: [1.67s] 
43: [1.67s]     pg_ctl -D /var/lib/postgresql/data -l logfile start
43: [1.67s] 
43: [1.71s] waiting for server to start....2023-07-31 13:00:53.261 UTC [58] LOG:  starting PostgreSQL 12.15 (Debian 12.15-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
43: [1.73s] 2023-07-31 13:00:53.262 UTC [58] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
43: [1.74s] 2023-07-31 13:00:53.276 UTC [59] LOG:  database system was shut down at 2023-07-31 13:00:52 UTC
43: [1.75s] 2023-07-31 13:00:53.282 UTC [58] LOG:  database system is ready to accept connections
43: [1.81s]  done
43: [1.81s] server started
43: [2.13s] CREATE DATABASE
43: [2.13s] 
43: [2.13s] 
43: [2.13s] /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
43: [2.13s] 
43: [2.13s] 2023-07-31 13:00:53.662 UTC [58] LOG:  received fast shutdown request
43: [2.13s] waiting for server to shut down....2023-07-31 13:00:53.662 UTC [58] LOG:  aborting any active transactions
43: [2.13s] 2023-07-31 13:00:53.665 UTC [58] LOG:  background worker "logical replication launcher" (PID 65) exited with exit code 1
43: [2.13s] 2023-07-31 13:00:53.666 UTC [60] LOG:  shutting down
43: [2.14s] 2023-07-31 13:00:53.677 UTC [58] LOG:  database system is shut down
43: [2.23s]  done
43: [2.23s] server stopped

then it continues on to:

43: [2.23s] PostgreSQL init process complete; ready for start up.
43: [2.23s] 
43: [2.25s] 2023-07-31 13:00:53.785 UTC [15] LOG:  starting PostgreSQL 12.15 (Debian 12.15-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
43: [2.25s] 2023-07-31 13:00:53.785 UTC [15] LOG:  listening on IPv4 address "0.0.0.0", port 5432
43: [2.25s] 2023-07-31 13:00:53.785 UTC [15] LOG:  listening on IPv6 address "::", port 5432
43: [2.25s] 2023-07-31 13:00:53.786 UTC [15] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
43: [2.27s] 2023-07-31 13:00:53.802 UTC [87] LOG:  database system was shut down at 2023-07-31 13:00:53 UTC
43: [2.28s] 2023-07-31 13:00:53.808 UTC [15] LOG:  database system is ready to accept connections

and hangs on this message until I cancel the job

#

here is my updated dagger file:

        postgres = (
            # use a postgres container
            client.container().from_("postgres:12")
            # set the environment variables
            .with_env_variable("POSTGRES_USER", POSTGRES_USER)
            .with_env_variable("POSTGRES_PASSWORD", POSTGRES_PASSWORD)
            .with_env_variable("POSTGRES_DB", POSTGRES_DB)
            .with_env_variable("POSTGRES_HOST_AUTH_METHOD", POSTGRES_HOST_AUTH_METHOD)
            .with_exec(["postgres"])
            .with_exposed_port(5432)
        )
        # ======================== PYTHON ========================
        python = (
            # use a python 3.10 container
            client.container().from_("python:3.11-slim-buster")
            # bind postgres service
            .with_service_binding("pg", postgres)
            # bind redis service
            .with_service_binding("redis", redis_srv)
            # postgres env variables
            .with_env_variable("PG_USER", POSTGRES_USER)
            .with_env_variable("PG_PASSWORD", POSTGRES_PASSWORD)
            .with_env_variable("PG_DB", POSTGRES_DB)
            .with_env_variable("PG_HOST_AUTH_METHOD", POSTGRES_HOST_AUTH_METHOD)
            .with_env_variable("PG_DATABASE_URL", f"pgsql://{POSTGRES_USER}:{POSTGRES_PASSWORD}@postgres/{POSTGRES_DB}")
            .with_env_variable("PORTALD_SECRET_KEY", PORTALD_SECRET_KEY)

cyan sphinx Jul 31, 2023, 6:19 PM

#

I'm not sure if this code is just a partial excerpt, but is the Python container actually executing anything? See https://docs.dagger.io/api/975146/concepts/#lazy-evaluation

dense cipher Jul 31, 2023, 6:49 PM

#

cyan sphinx I'm not sure if this code is just a partial excerpt, but is the Python container...

the python container is, i was just including a snippet. sorry for the confusion!

dense cipher Jul 31, 2023, 6:55 PM

#

cyan sphinx I'm not sure if this code is just a partial excerpt, but is the Python container...

here is the entire container:

        # ======================== PYTHON ========================
        python = (
            # use a python container
            client.container().from_("python:3.11-slim-buster")
            # bind postgres service
            .with_service_binding("db", postgres)
            .with_env_variable("DB_HOST", "db")
            .with_env_variable("DB_PASSWORD", "")
            .with_env_variable("DB_USER", "test")
            .with_env_variable("DB_NAME", "test")
            .with_env_variable("PORTALD_DATABASE_URL", f"pgsql://{POSTGRES_USER}:{POSTGRES_PASSWORD}@postgres/{POSTGRES_DB}")
            # bind redis service
            .with_service_binding("redis", redis_srv)
            .with_env_variable("PORTALD_SECRET_KEY", PORTALD_SECRET_KEY)
            # load in dependencies cache
            .with_mounted_cache("/root/.cache/pip", pip_cache)
            .with_mounted_cache("/root/.cache/pypoetry", poetry_cache)
            # mount cloned repository into image
            .with_directory("/src", src)
            # set current working directory for next commands
            .with_workdir("/src")
            .with_exec(["apt-get", "update"])
            .with_exec(["apt-get", "install", "-y", "build-essential", "libssl-dev", "libffi-dev", "python3-dev", "cargo", "libvirt-dev", "npm", "binutils", "libproj-dev", "gdal-bin"])

void gale Jul 31, 2023, 7:14 PM

#

Hey @dense cipher 👋 I used this to try to reproduce the behavior (changed to use postgres:12 too) using this code: https://github.com/dagger/examples/blob/main/python/db-service/pipeline.py and wasn't able to see the issue. Can you let me know which sdk version you're seeing this on?

dense cipher Jul 31, 2023, 7:39 PM

#

void gale Hey <@358807393515208704> 👋 I used this to try to reproduce the behavior (chang...

hi Kyle! I am using v0.6.4. I simplified my code to just be running:

import sys
from urllib.parse import urljoin
import anyio
import dagger
import os


async def test():
    async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:

        # ======================== POSTGRES ========================
        database = (
            # use a postgres container
            client.container().from_("postgres:12")
            .with_env_variable("POSTGRES_PASSWORD", "")
            .with_env_variable("POSTGRES_HOST_AUTH_METHOD", "trust")
            .with_env_variable("POSTGRES_DB", "test")
            .with_exec(["postgres"])
            .with_exposed_port(5432)
        )
        
        # get reference to the local project
        src = client.host().directory(".")

        # ======================== PYTHON ========================
        python = (
            # use a python 3.10 container
            client.container().from_("python:3.10-slim-buster")
            # bind postgres service
            .with_service_binding("db", database)
            .with_env_variable("DB_HOST", "db")
            .with_env_variable("DB_PASSWORD", "")
            .with_env_variable("DB_USER", "test")
            .with_env_variable("DB_NAME", "test")
            # mount cloned repository into image
            .with_directory("/src", src)
            # set current working directory for next commands
            .with_workdir("/src")
            .with_exec(["apt-get", "update"])
            .with_exec(["apt-get", "install", "-y", "build-essential", "libssl-dev", "libffi-dev", "python3-dev", "cargo", "libvirt-dev", "npm", "binutils", "libproj-dev", "gdal-bin"])
            .with_exec(["pip", "install", "poetry"])
            .with_exec(["poetry", "install", "--with", "test"])
            # run tests
            .with_exec(["poetry", "run", "pytest", "--junitxml=\"test-result.xml\"", "--cov-report", "xml", "--cov-report", "term", "--cov=portald"])
        )
        
        # execute
        await python.directory("/src").export(".")
        print("Succeeded!")
        
if __name__ == "__main__":
    anyio.run(test)

which is pretty similar to your code, and it's still hanging

void gale Aug 1, 2023, 12:38 AM

#

dense cipher hi Kyle! I am using v0.6.4. I simplified my code to just be running: ```yaml imp...

Thanks for doing this! Let me see if I see the same issue locally

slender gyro Aug 1, 2023, 1:41 AM

#

👋 this seems to work ok @dense cipher

import sys
from urllib.parse import urljoin
import anyio
import dagger
import os


async def test():
    async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:

        # ======================== POSTGRES ========================
        database = (
            # use a postgres container
            client.container().from_("postgres:12")
            .with_env_variable("POSTGRES_PASSWORD", "")
            .with_env_variable("POSTGRES_HOST_AUTH_METHOD", "trust")
            .with_env_variable("POSTGRES_DB", "test")
            .with_exec(["postgres"])
            .with_exposed_port(5432)
        )

        # ======================== PYTHON ========================
        python = (
            # use a python 3.10 container
            client.container().from_("python:3.10-slim-buster")
            # bind postgres service
            .with_service_binding("db", database)
            .with_env_variable("DB_HOST", "db")
            .with_env_variable("DB_PASSWORD", "")
            .with_env_variable("DB_USER", "test")
            .with_env_variable("DB_NAME", "test")
            .with_exec(["apt", "update"])
            .with_exec(["apt", "install", "-y", "postgresql-client"])
            .with_exec(["pg_isready", "-d", "test", "-h", "db", "-U", "postgres"])

        )

        # execute
        await python.sync()

if __name__ == "__main__":
    anyio.run(test)

#

^ if that works for you then postgres is running correctly and I'll double check your poetry tests to make sure they have the correct config to connect to your DB

dense cipher Aug 1, 2023, 3:11 PM

#

Hi @slender gyro @void gale thank you both for your help! I tested out your configuration, Marcos, and it works locally on my end, but not on GitLab. It keeps hanging on the same spot. Could this perhaps be an issue with the .gitlab-ci.yml file?

slender gyro Aug 1, 2023, 7:39 PM

#

dense cipher Hi <@336241811179962368> <@135620352201064448> thank you both for your help! I t...

Interesting.. are you using Gitlab's public runners? or self-hosted?

dense cipher Aug 1, 2023, 7:58 PM

#

slender gyro Interesting.. are you using Gitlab's public runners? or self-hosted?

Self-hosted runners

slender gyro Aug 1, 2023, 8:00 PM

#

dense cipher Self-hosted runners

are you ok with jumping into #911305510882513037 and share your screen about the pipeline that I posted above to check what might be happening?

dense cipher Aug 1, 2023, 8:01 PM

#

slender gyro are you ok with jumping into <#911305510882513037> and share your screen about t...

Yeah, for sure! I'll be there in a minute, need to move to a quieter location

astral pewter Aug 7, 2023, 4:43 PM

#

Hi all, did this get resolved in the audio session? If so, what was the issue? I'm curious 🙂

slender gyro Aug 7, 2023, 7:59 PM

#

Hey Solomon! Yes, we solved this with a 1:1. Guilly found that there was some misconfiguration in their Gitlab CI job which was causing this issue. I'm unsure what the specifics were but they were able to get unblocked.

#Postgres & Redis in Dagger