Extremely slow media and socket issues since updating to +3.4X.x | Payload | Page 1

ancient seal Jul 17, 2025, 2:42 PM

#

I bumped my payload version to 3.44, 3.45, and now 3.46, and since then I've been having a lot of issues with media. (It more or less was at the same time of having more people adding data to our application).
I'm using the s3 adapter.

The main issue was that when retrieving data, the socket limit (default config) would hit everytime, it wasn't even able to load like 10 images without being hit... I tried upping the socket limit, enabling keepAlive for connections and other stuff, which didn't work (and I think this should NOT be the solution for a problem like this, specially when there's not even that many people using the app, maybe 5-10 people simultaneously using it).

I ended up using the generateFileURL method to retrieve the s3 object url to server the images directly from S3, which has been workin fine so far.

The problem is that for the admin panel it still uses the API, and it still gets extremely slow, I'm talking about taking 5-10 seconds to load the images, and I can't even upload images anymore, production takes maybe 1-2 minutes to load a single image, and staging, which has slightly less resources gets a 524 (Timeout) every single time.

I honestly don't know what else to do here... could someone help me pls

deep pawnBOT Jul 17, 2025, 2:42 PM

#

New Community-Help Thread Created!

Help is on the way! To mark it as solved, use the /solve command. In the meantime, here are some existing threads that may help you:

Documentation:

ancient seal Jul 17, 2025, 3:41 PM

#

I just disabled the storage-s3 plugin and tried uploading an image (locally) and it's still taking a huge amount of time to upload... meaning that the issue is probably not on the s3 plugin...

ancient seal Jul 17, 2025, 5:49 PM

#

s3 plugin config:

#

import { s3Storage as buildS3Plugin } from '@payloadcms/storage-s3';
import { Plugin } from 'payload';

export const s3Plugin: Plugin = buildS3Plugin({
  collections: {
    media: {
      prefix: 'media',
      generateFileURL: ({ filename, prefix }) => {
        return `https://${process.env.S3_BUCKET}.s3.${process.env.S3_REGION}.amazonaws.com/${prefix}/${filename}`;
      },
    },
  },
  bucket: process.env.S3_BUCKET ?? '',
  config: {
    requestHandler: {
      httpsAgent: {
        maxSockets: 200,
        timeout: 60000,
      },
    },
    region: process.env.S3_REGION ?? 'eu-central-1',
    credentials: {
      accessKeyId: process.env.AWS_ACCESS_KEY_ID ?? '',
      secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY ?? '',
    },
    logger: {
      debug: (msg) => console.log('📥 S3 Debug:', msg),
      info: (msg) => console.log('📥 S3 Info:', msg),
      warn: (msg) => console.log('⚠️ S3 Warn:', msg),
      error: (msg) => console.log('🚨 S3 Error:', msg),
    },
  },
});

#

payload config:

export default buildConfig({
  admin: {
    components: {
      afterNavLinks: [
        '@/components/admin/CurrentWebsiteSelector/CurrentWebsiteSelector#CurrentWebsiteSelector',
      ],
    },
    importMap: {
      baseDir: path.resolve(dirname),
    },
    user: Users.slug,
    livePreview: {
      breakpoints: [
        {
          label: 'Mobile',
          name: 'mobile',
          width: 375,
          height: 667,
        },
        {
          label: 'Tablet',
          name: 'tablet',
          width: 768,
          height: 1024,
        },
        {
          label: 'Desktop',
          name: 'desktop',
          width: 1440,
          height: 900,
        },
      ],
    },
  },
  editor: defaultLexical,
  db: postgresAdapter({
    idType: 'uuid',
    pool: {
      user: process.env.DB_USER || '',
      password: process.env.DB_PASSWORD || '',
      database: process.env.DB_NAME || '',
      port: Number(process.env.DB_PORT as string) || 5432,
      host: process.env.DB_HOST || '127.0.0.1',
      ssl: process.env.NODE_ENV === 'production' ? { rejectUnauthorized: false } : false,
    },
    push: false,
  }),
  logger,
  loggingLevels: {
    APIError: 'error',
    AuthenticationError: 'warn',
    ErrorDeletingFile: 'error',
    FileRetrievalError: 'warn',
    FileUploadError: 'error',
    Forbidden: 'warn',
    Locked: 'info',
    LockedAuth: 'warn',
    MissingFile: 'warn',
    NotFound: 'debug',
    QueryError: 'error',
    ValidationError: 'info',
  },
  collections: [Websites, Articles, Products, Categories, Users, Media],
  localization: {
    locales: locales as unknown as Locale[],
    defaultLocale: 'en-US',
    defaultLocalePublishOption: 'active',
  },
  cors: [getServerSideURL()].filter(Boolean),
  plugins: [s3Plugin, redirectsPlugin],
  secret: process.env.PAYLOAD_SECRET,
  sharp,
  typescript: {
    outputFile: path.resolve(dirname, 'payload-types.ts'),
  },
  upload: {
    safeFileNames: true,
  },
});

stiff isle Jul 17, 2025, 5:56 PM

#

Hmm, have you guys tried setting signedDownloads: true?

#

You guys removed the S3 plugin but the symptoms persist even then?

ancient seal Jul 17, 2025, 6:04 PM

#

yes, same symptoms, specially noticed when loading media. I can't test the download with real data as all of our images are stored in s3

stiff isle Jul 17, 2025, 6:06 PM

#

I'd suggest trying with this config:

requestHandler: {
  httpAgent: {
    maxSockets: 300,
    keepAlive: true,
  },
  httpsAgent: {
    maxSockets: 300,
    keepAlive: true,
  },
  connectionTimeout: 5 * 1000,
  requestTimeout: 5 * 1000,
},

As perscribed in that issue, maybe on staging/test env

#

And the issue is only on large uploads?

#

Sounds like you have two issues here, one on uploading large files, the other on reading them

#

Are there any hooks that are firing on upload?

ancient seal Jul 17, 2025, 6:17 PM

#

no hooks for media collection whatsoever.

reading files there was the socket limit issue. I solved that just by serving images from s3 directly.

The issue now is that, the more images we download (admin panel) the slower it gets, until we try to upload an image, that's where it stops working (long waits & timeouts)

stiff isle Jul 17, 2025, 6:19 PM

#

Can you inspect the downloads, look for the request that initiates it and try to see if it's being called with a limit property set on it to something that makes sense

#

You can do this locally

ancient seal Jul 17, 2025, 6:54 PM

#

im getting home, but are you talking about the request from the media collection list view? or which one?

stiff isle Jul 17, 2025, 7:41 PM

#

Yeah, media collection in list view

#

Does the limit in the request for these images align with the limit you have on the list view? IE with 100 on list view, the request also shows 100 and only loads 100 docs

ancient seal Jul 18, 2025, 7:05 AM

#

I'm looking at the metrics on the RDS instance (attached image) and it seems like it usually hits 16 connections.

I want to guess that these connections are reusable, that these are not 16 parallel active connections although I'm not 100% sure how it works.

#

Also, CPUUtilization metric doesn't go over 7-10% tops

ancient seal Jul 18, 2025, 7:09 AM

#

stiff isle Does the limit in the request for these images align with the limit you have on ...

I'm going to do what you said @stiff isle in a freshly deployed environment right now

#

It loaded pretty fast now, I have to say.
This is the url: https://www.mywebsite.es/admin/collections/media/?limit=100&sort=filename (it's what's generated when opening the media collection)

Images loaded almost instantly this time to my surprise.

#

These are the Fetch/XHR requests loaded:

#

There were exactly 100 Img requests made, for the 100 images that were downloaded in the list view

#

These are the requests when trying to upload an 8MB image

#

it took kinda long

#

I'll upload a couple more and see what happens

#

it's taking a normal amount of time, and I uploaded it like 4 times already... that's crazy hahaa

#

this is so werid

snow stream Jul 18, 2025, 12:29 PM

#

I am able to repro this on atlas

ancient seal Jul 19, 2025, 8:58 AM

#

Any updates guys?

snow stream Jul 20, 2025, 11:09 AM

#

I would definitely use the generateFileURL for now. This issue has to do with how serverless platforms and mongodb connections function. Vercel is spinning up lambdas for these requests and for some reason the warm runners don’t get reused. Now if you were on a server(full) instance mongodb connections pool properly.

ancient seal Jul 21, 2025, 5:13 PM

#

But, usgin the generateFileURL works for serving the media, but not for uploading/serving on the admin panel firhg?

snow stream Jul 22, 2025, 1:31 AM

#

It will work in both

ancient seal Jul 23, 2025, 4:53 PM

#

I used the generateFileURL method but in the Media list view it's still making requests to /api/media/.... How exactly can I load these thumbnails from S3 instead?

Additional question: is there any way to lazy load the list views? specially for the media collection?

ancient seal Jul 23, 2025, 5:50 PM

#

I saw the clientUploads config, which is supposed to be used with vercel, but I'm testing it locally and it seems to work as well....
This would essentially upload the images directly to S3 from the client right? @stiff isle @snow stream meaning that it would bypass making the request through the server -> meaning less requests and leaving more stuff to be handled by s3...

stiff isle Jul 23, 2025, 5:51 PM

#

Correct, the upload is handled directly from the client to s3 bypassing Vercel upload limits

ancient seal Jul 23, 2025, 5:52 PM

#

I've applied this locally and I've noticed that uploads aren't any faster whatsoever, which makes me wonder if the guilt is not even the server or the database for the slowness 🤔

dreamy hill Jul 23, 2025, 9:21 PM

#

stiff isle Correct, the upload is handled directly from the client to s3 bypassing Vercel u...

I am also on Vercel + S3 (R2 Cloudflare) and I cannot get clientUploads working

#

it still says file size too large although i have clientUploads set to true

#

I have set CORS policy for the bucket on R2 as well

#

localhost works well ofc, since not using vercel, but as soon as pushed to preview or production, the upload doesnt work anymore and getting "filesize too large"

#

is there anything else to setup besides clientUplaods true

#

#

on a 6.3MB image file

#

CORS policy on the r2 bucket set for the production domain

#Extremely slow media and socket issues since updating to +3.4X.x

Documentation: