#Long build times for image-heavy site

13 messages · Page 1 of 1 (latest)

rapid spruce
#

I just rebuilt my personal photography website with Astro. It has around 500 unique photos in JPG format. I generate AVIF and WebP in multiple sizes for responsive image sizing. The build generates close to 10,000 image variants in total, which is a lot, but not unreasonable for the amount of images I have. I've already cut down the number of variants I generate in the srcset, reducing the number of variants further is not viable.

The problem is that the build takes a long time. The site is fully static is hosted on Cloudflare Pages. The build on Pages timed out after around 35 minutes (there's a soft cap of 20 minutes, so that's to be expected). Now I build and deploy the site through GitHub actions. The build takes a bit over one hour, 99 % of that are the image variants being generated. While this works, I'd like to optimize the build time.

I've already implemented caching in the GH Action via the actions/cache action. This caches and restores the .astro folder (see the GitHub Action below). However, that doesn't really move the needle – it's still generating all the images in every build. I don't think the generated image variants are cached in the .astro folder at all, because the cache size indicated by the action is tiny, just a couple of kilobytes. Can someone confirm if that is expected?

The main problem is that every build regenerates all variants every time, even though 99 % of them are identical between builds. But I haven't really found a solution for that.

I know I could deploy the site on workers instead and have the images generate on request. But the site is completely static and often doesn't change for months on end. Running workers and dynamically generating everything is completely unnecessary and a waste of energy. Same with outsourcing the image hosting to an external service like S3 or whatever.

Is there any way to implement incremental builds and reuse image variants between builds? Or a different approach altogether?

#

This is the deployment GitHub Action:

name: Build and Deploy

on:
  push:
    branches: [main]
  workflow_dispatch:

concurrency:
  group: deploy
  cancel-in-progress: false

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v6

      - uses: actions/setup-node@v6
        with:
          node-version-file: package.json
          cache: npm

      - name: Install dependencies
        run: npm ci

      - name: Restore Astro image cache
        uses: actions/cache/restore@v5
        with:
          path: .astro/
          key: astro-${{ hashFiles('src/content/**/*.{jpg,jpeg,png,webp,avif,gif,tiff}', 'src/utils/responsive-images.ts') }}
          restore-keys: astro-

      - name: Build
        run: npm run build

      - name: Save Astro image cache
        uses: actions/cache/save@v5
        with:
          path: .astro/
          key: astro-${{ hashFiles('src/content/**/*.{jpg,jpeg,png,webp,avif,gif,tiff}', 'src/utils/responsive-images.ts') }}

      - name: Deploy to Cloudflare Pages
        run: npx wrangler pages deploy dist/ --project-name=${{ vars.CLOUDFLARE_PAGES_PROJECT }}
        env:
          CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
          CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
#

(The path .astro for the cache action is correct, I have cacheDir: '.astro' in my config)

hollow vector
rapid spruce
hollow vector
#

Yikes, my bad. Completely missed that. Do the images show up in that folder locally?

rapid spruce
hollow vector
#

Yeah, I believe the files should show up there

#

When you unset the option, are they visible in the node_modules/.astro dir?

#

I think this might be a case of the top level .astro dir not working for caching

rapid spruce
hollow vector
#

Huh, alright. If you find anything, let me know!

rapid spruce
# hollow vector Huh, alright. If you find anything, let me know!

Ok, there really were just a couple of issues in the GitHub Action that caused the cache to fail 🤦‍♂️
The hashFiles() function doesn't support brace extension, so *.{jpg,jpeg,png,webp,avif,gif,tiff} didn't match anything. That caused the action to always re-use the very first cache from the first run (which didn't have any photos), instead of using the newer caches from previous runs. A couple other confounding factors, but that was the main issue.

Thanks for the help!

Leaving the final version of the deployment action here in case someone has a similar problem. Still testing a bit with freshly cleared caches, but looks like this is working now:

name: Build and Deploy

on:
  push:
    branches: [main]
  workflow_dispatch:

concurrency:
  group: deploy
  cancel-in-progress: false

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v6

      - uses: actions/setup-node@v6
        with:
          node-version-file: package.json
          cache: npm

      - name: Install dependencies
        run: npm ci

      - name: Restore Astro cache
        uses: actions/cache/restore@v5
        with:
          path: .astro/
          key: astro-${{ hashFiles('src/content/photos/**', 'src/utils/responsive-images.ts') }}
          restore-keys: astro-

      - name: Build
        run: npm run build

      - name: Save Astro cache
        uses: actions/cache/save@v5
        with:
          path: .astro/
          key: astro-${{ hashFiles('src/content/photos/**', 'src/utils/responsive-images.ts') }}

      - name: Deploy to Cloudflare Pages
        run: npx wrangler pages deploy dist/ --project-name=${{ vars.CLOUDFLARE_PAGES_PROJECT }}
        env:
          CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
          CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}