#getting psycopg_pool.PoolTimeout while running multiple threaded tasks

3 messages · Page 1 of 1 (latest)

pseudo carbon
#

hey!

lets say i have this database config with postgres + psycopg3 with pools

    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql',
            'NAME': DATABASE_URL.path.replace('/', ''),
            'USER': DATABASE_URL.username,
            'PASSWORD': DATABASE_URL.password,
            'HOST': DATABASE_URL.hostname,
            'PORT': DATABASE_URL.port,
            'OPTIONS': {
                'sslmode': 'require',
                'pool': {
                    'timeout': 120,
                    'min_size': 4
                    'max_size': 4,
                },
            },
        }
    }

forcing multiple threaded tasks to force the error (> pool max_size):

def task():
    user = User.objects.get(email='[email protected]')
    obj = Obj.objects.get(user=user)
    obj.name = ''.join(random.choice(string.ascii_letters + string.digits) for _ in range(10))
    obj.save()

for _ in range(6):
    threading.Thread(target=task).start()

as expected i get this error:
File ".venv\Lib\site-packages\psycopg_pool\pool.py", line 203, in getconn
raise PoolTimeout(
psycopg_pool.PoolTimeout: couldn't get a connection after 120.00 sec

basically i run out of connection because they are 'stuck' in the task threads

i fix this by removing the pool config and using normal postgres connections with CONN_MAX_AGE etc
or by writing connection.close() at the end of every threaded task

def task():
...
connection.close()

but i wonder, whats the best way to handle this problem?

edit: idk why but code style is bugging

tender monolith
#

@pseudo carbon is this your correct code?

pseudo carbon
#

this is the example:

def task():
    user = User.objects.get(email='[email protected]')
    obj = Obj.objects.get(user=user)
    obj.name = ''.join(random.choice(string.ascii_letters + string.digits) for  in range(10))
    obj.save()

for _ in range(6):
    threading.Thread(target=task).start()