Astro SSR function use spike

[storm comet]

The project is using the Netlify adapter.

Output is set to static.

There are 3 pages that have prerender = false in them.

Those pages are added to robots.txt with "disallow", and they also have noindex nofollow meta tag in the DOM.

Despite this, there are a ton of requests (looking at ~500k) in like 2 days for Netlify functions.

Opening the Astro SSR log I can see that it's constantly being used, so it seems to be the culprit here.

Any idea what's going on? Does the Astro SSR function get called in other use cases, even if output is static and only 3 pages have prerender = false (and they're excluded anyway from being indexed and such)?

[covert timber]

Hi! I'll need more info here. Do you use Actions or API routes that are called from somewhere?

[storm comet]

Hello! Thanks for looking into it.

No actions at least, no.

Not sure about API routes. Yes, there are some fetch requests for Sanity (this is our CMS).

If you're talking specifically about:

import type { APIRoute } from "astro";

We only have that for our sitemaps that we generate.

[covert timber]

As for the logs, do they tell you the specific routes that are being hit?

[storm comet]

No, just memory usage and duration.

[covert timber]

Hm. That's a bit difficult to debug then

I'm assuming that maybe something is getting hit by a prefetch link? That's the only idea I have

[storm comet]

Wait a sec, I believe we use the Netlify Image CDN.

That would count towards function usage, no?

As that's something that requires the adapter.

We use the Astro <Image /> component quite a bit it looks like.

[covert timber]

Do the requests go to the /_image/ route? (you can check on the site in the source/srcset)

[storm comet]

They are in the form of .netlify/images?url=_astro

That's the src

And we don't use that at all in the src fields, so it must be Astro SSR adapter doing the transformations using Netlify Image CDN, and causing hundreds of thousands of requests per day. That is my guess atm 😄

[covert timber]

Yeah, sounds like an idea

@jaunty flare do you have experience with the Netlify CDN? Since you use Netlify for Dawnforge

[storm comet]

I guess I need to follow this, to disable it? #1404817807354499092 message

I need to do both the adapter setting and the domains one I imagine.

[covert timber]

https://docs.astro.build/en/guides/integrations-guide/netlify/#netlify-image-cdn-support

Yep

Astro docs strike again (and they're usually faster than AI)

[jaunty flare]

Never used the cdn sorry!

[storm comet]

Sounds good, ty. If I'm reading this correctly, compared to the AI response, it's not necessary to setup the image domains with an empty array at all, as default means they are all unauthorized?

Sorry I forgot to turn off ping this tjme

[covert timber]

Guess so yeah
(all good about the ping btw)

[storm comet]

Great, ty! I'll check it out when I can and mark the thread as solved if this was it.

[storm comet]

Unfortunately, this wasn't it.

Still constant function use.

[storm comet]

OK, I have a weird "fix".

Not only did I disable the Netlify Image CDN, but I also literally removed the .astro files that were the pages which had prerender = false.

Now, the Asto SSR function does not exist anymore in Netlify.

Because all pages are prerendered I guess (unsure if image CDN had anything to do with it or not, but going to keep it disabled for now).

Weird thing is that those 3 pages were blocked by robots.txt and also have noindex meta tags on them...

Still, we shouldn't be getting hundreds of thousands of function requests per day... I wonder if something else gets counted as "function use" just because the Astro SSR function is added, then?

Like 404s for example? We still don't get hundreds of thousands of 404s though...

[covert timber]

I'll call in the rest of support patrol to see if anyone else knows about this. Deffo a strange issue.

[vernal juniper]

Houston, we have a problem... and <@&1129102257422610512>, you’re our mission control! 🚀

[covert timber]

TL;DR: High Netlify function use on prerendered project with few SSR pages, Image CDN is not the cause, removing the SSR pages fixes the high function use

[storm comet]

Thanks!

I mean it's fine, we don't need those 3 pages for now necessarily, but we do want to add them back eventually.

I only have 2 theories at the moment:

• Those 3 SSR pages were being hammered, despite them being blocked in robots, the noindex meta tag on them, and also they are not in the sitemap
• The 3 pages existing added the Astro SSR function to Netlify, and it is possible that some other functionality gets added along with those 3 pages as well, that triggers function use. I'm thinking maybe 404s could do this when Astro SSR Netlify function exists, for example (though we shouldn't have that many 404s either).

[echo nest]

Why not re-enable Netlify Image CDN and see what happens? Then you'll know for sure if it's part of the problem or not.

As for the 3 SSR pages, check your routing and make sure traffic is not going there in error with some inadvertent catch-all route.

[storm comet]

Thanks! I'll re-enable the Netlify Image CDN at some point and check it out (at least to satisfy my inner curiosity), but for now the internal decision is to keep it off.

[echo nest]

FYI - This might help make sure your routes are correct, just in case.

[storm comet]

Ah nice, thank you 😄 Will take a look when I have time.

[storm comet]

Can confirm that enabling Netlify Image CDN did not re-add the Astro SSR function in Netlify. Weird, I was convinced this would use up functions, not sure what it uses then 😄

I've updated the dependencies to check out the new route conflicts warning, though in the blog post itself the flag is different from the one in the docs (different name). Just a heads-up. Mentioned this in #docs

[storm comet]

I have another update for this.

There were some conflicting routes, but nothing out of the ordinary I think. I have static routes and dynamic routes in the same folder, I thought that was fine enough and I wouldn't have to exclude the static routes from the dynamic routes, as static routes were prioritized by default.

Well, builds were failing (I guess it's normal with that flag enabled) because of conflicts like those. I fixed the conflicts by manually excluding the static routes from the dynamic ones, and builds started working again.

However, there are still tons of Netlify function requests, so I'm not sure if this had much impact at all. I suppose I'll get rid of the SSR pages again, so the Astro SSR function disappears.

[plain moon]

can you share your astro config by chance? and is it that there are too many functions being created in the dist/ or that they are getting used way too much?

[storm comet]

Sorry for the very long delay!

Here's astron.config.mjs:

import mdx from '@astrojs/mdx';
import react from '@astrojs/react';
import tailwind from '@astrojs/tailwind';
import AutoImport from 'astro-auto-import';
import { defineConfig } from 'astro/config';
import remarkCollapse from 'remark-collapse';
import remarkToc from 'remark-toc';
import config from './src/config/config.json';
import sanity from '@sanity/astro';
import { createClient } from '@sanity/client';

import icon from 'astro-icon';
import netlify from '@astrojs/netlify';

// Create Sanity client for sitemap filtering
const sanityClient = createClient({
  projectId: process.env.VITE_SANITY_PROJECT_ID,
  dataset: process.env.VITE_SANITY_PROJECT_DATASET,
  useCdn: false,
  apiVersion: '2023-01-01',
});

// Cache for published content to avoid repeated queries
let publishedSlugsCache = null;
let cacheTimestamp = 0;
const CACHE_TTL = 60000; // 1 minute cache

async function getPublishedSlugs() {
  const now = Date.now();
  if (publishedSlugsCache && (now - cacheTimestamp) < CACHE_TTL) {
    return publishedSlugsCache;
  }

  try {
    const [blogPosts, glossaryTerms] = await Promise.all([
      sanityClient.fetch(`*[_type == "post" && defined(slug.current) && !(_id in path("drafts.**"))] { "slug": slug.current }`),
      sanityClient.fetch(`*[_type == "glossary" && defined(slug.current) && !(_id in path("drafts.**"))] { "slug": slug.current }`)
    ]);

    publishedSlugsCache = {
      blogSlugs: new Set(blogPosts.map(post => post.slug)),
      glossarySlugs: new Set(glossaryTerms.map(term => term.slug))
    };
    cacheTimestamp = now;
    return publishedSlugsCache;
  } catch (error) {
    console.warn('Failed to fetch published slugs for sitemap filtering:', error);
    // Return empty sets to exclude everything if query fails
    return { blogSlugs: new Set(), glossarySlugs: new Set() };
  }
}

// https://astro.build/config
export default defineConfig({
  site: config.site.base_url ? config.site.base_url : '',
  base: config.site.base_path ? config.site.base_path : '/',
  trailingSlash: config.site.trailing_slash ? 'always' : 'never',
  output: 'static', // Static site generation
  adapter: netlify(),
  experimental: {
    preserveScriptOrder: true,
    failOnPrerenderConflict: true,
  },
  // Use modern-compiler for SCSS to fix deprecation warning
  vite: {
    css: {
      preprocessorOptions: {
        scss: {
          api: 'modern-compiler',
        },
      },
      // Add CSS optimization
      build: {
        cssCodeSplit: false,
        cssMinify: true,
      }
    },
  },
  integrations: [
    react(),
    sanity({
      projectId: import.meta.env.VITE_SANITY_PROJECT_ID,
      dataset: import.meta.env.VITE_SANITY_PROJECT_DATASET,
      useCdn: false, // Real-time data for previews
      apiVersion: '2023-01-01',
      token: import.meta.env.VITE_SANITY_AUTH_TOKEN, //needs to be used for draft versions (in preview)
    }),
    tailwind({
      applyBaseStyles: false,
    }),
    AutoImport({
      imports: [
        '@/shortcodes/Button',
        '@/shortcodes/Accordion',
        '@/shortcodes/Notice',
        '@/shortcodes/Video',
        '@/shortcodes/Youtube',
        '@/shortcodes/Tabs',
        '@/shortcodes/Tab',
      ],
    }),
    mdx(),
    icon(),
  ],
  markdown: {
    remarkPlugins: [
      remarkToc,
      [
        remarkCollapse,
        {
          test: 'Table of contents',
        },
      ],
    ],
    shikiConfig: {
      theme: 'one-dark-pro',
      wrap: true,
    },
    extendDefaultPlugins: true,
  },
});

Split up into 2 because I couldn't fit it in a single block (unsure if I should link to stuff like pastebin here).

I don't know if we use everything that is here, I tend to clean it up at some point... when I have time 😅

experimental: {
    preserveScriptOrder: true,
    failOnPrerenderConflict: true,
  },

To reiterate the issue, the issue is the Astro SSR function that is being used too much (I imagine it should have very few uses, if at all, as the site is almost purely SSG).

When I add the 3 SSR pages (which we use for previews for blogs, glossary, etc), Astro SSR function gets automatically added to the deploy in Netlify and its usage skyrockets immediately.

When I remove those 3 pages, the Astro SSR function is not present at all in Netlify, as expected.

These pages are excluded from robots.txt anyway and they also have noindex/nofollow on them just to be 100% sure.

I don't know exactly what I should be looking for in the dist folder in regards to the Astro SSR function.

It looks normal to me but it's not like I know what to look for 😄

[storm comet]

Bump because the thread disappeared from my list, guess this should keep it there for a while again.

[storm comet]

I'm gonna go ahead and guess it's this: #netlify-archived message

[plain moon]

Hmm so it’s 404s causing the problem